Hive Client Install Fail - hive

Using Ubuntu 18.04, Installing Hortonworks Data Platform using Ambari 2.7.3.0 gets stuck at Hive Client Install then fail after almost 30 minutes with the following error
2019-02-18 13:20:20,369 - File['/tmp/tmpFVEJd6'] {'content': StaticFile('/etc/apt/sources.list.d/ambari-hdp-1.list')}
2019-02-18 13:20:20,370 - Writing File['/tmp/tmpFVEJd6'] because contents don't match
2019-02-18 13:20:20,383 - Package['unzip'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2019-02-18 13:20:20,517 - Skipping installation of existing package unzip
2019-02-18 13:20:20,517 - Package['curl'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2019-02-18 13:20:20,638 - Skipping installation of existing package curl
2019-02-18 13:20:20,639 - Package['hdp-select'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2019-02-18 13:20:20,782 - Skipping installation of existing package hdp-select
2019-02-18 13:20:20,788 - The repository with version 3.1.0.0-78 for this command has been marked as resolved. It will be used to report the version of the component which was installed
2019-02-18 13:20:21,675 - Using hadoop conf dir: /usr/hdp/3.1.0.0-78/hadoop/conf
2019-02-18 13:20:21,765 - call['ambari-python-wrap /usr/bin/hdp-select status hive-server2'] {'timeout': 20}
2019-02-18 13:20:21,866 - call returned (0, 'hive-server2 - 3.1.0.0-78')
2019-02-18 13:20:21,866 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.0.0-78 -> 3.1.0.0-78
2019-02-18 13:20:21,938 - File['/var/lib/ambari-agent/cred/lib/CredentialUtil.jar'] {'content': DownloadSource('http://ip-172-31-16-181:8080/resources/CredentialUtil.jar'), 'mode': 0755}
2019-02-18 13:20:21,939 - Not downloading the file from http://ip-172-31-16-181:8080/resources/CredentialUtil.jar, because /var/lib/ambari-agent/tmp/CredentialUtil.jar already exists
2019-02-18 13:20:45,289 - Package['hive-3-1-0-0-78'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2019-02-18 13:20:46,021 - Installing package hive-3-1-0-0-78 ('/usr/bin/apt-get -o Dpkg::Options::=--force-confdef --allow-unauthenticated --assume-yes install hive-3-1-0-0-78')
Command failed after 1 tries

Related

Webpack error when deploying Rails 6 app on elastic beanstalk with npm, yarn and webpack

I tried to deploy my Rails 6 app that's running stable on production (!) into a second new environment on elastic beanstalk (eb) and I cannot get it to run despite just copying the configuration from the first setup.
After researching all resources I could find for 2 days, I'm currently stuck with the following error compilation failed: webpack not installed:
-------------------------------------
/var/log/eb-activity.log
-------------------------------------
+++ export RUBY_VERSION=2.6.5
+++ RUBY_VERSION=2.6.5
+++ export GEM_ROOT=/opt/rubies/ruby-2.6.5/lib/ruby/gems/2.6.0
+++ GEM_ROOT=/opt/rubies/ruby-2.6.5/lib/ruby/gems/2.6.0
++ (( 0 != 0 ))
+ /opt/elasticbeanstalk/support/scripts/check-for-gem.rb puma
+ echo true
[2020-01-09T20:22:02.966Z] INFO [1508] - [Application update app-f4fb-200109_211455#11/AppDeployStage0/AppDeployPreHook] : Completed activity. Result:
Successfully execute hooks in directory /opt/elasticbeanstalk/hooks/appdeploy/pre.
[2020-01-09T20:22:02.966Z] INFO [1508] - [Application update app-f4fb-200109_211455#11/AppDeployStage0/EbExtensionPostBuild] : Starting activity...
[2020-01-09T20:22:10.381Z] INFO [1508] - [Application update app-f4fb-200109_211455#11/AppDeployStage0/EbExtensionPostBuild/Infra-EmbeddedPostBuild] : Starting activity...
[2020-01-09T20:22:10.462Z] INFO [1508] - [Application update app-f4fb-200109_211455#11/AppDeployStage0/EbExtensionPostBuild/Infra-EmbeddedPostBuild/postbuild_0_XXX] : Starting activity...
[2020-01-09T20:22:23.059Z] INFO [1508] - [Application update app-f4fb-200109_211455#11/AppDeployStage0/EbExtensionPostBuild/Infra-EmbeddedPostBuild/postbuild_0_XXX/Command 01_restart_nginx] : Starting activity...
[2020-01-09T20:22:24.293Z] INFO [1508] - [Application update app-f4fb-200109_211455#11/AppDeployStage0/EbExtensionPostBuild/Infra-EmbeddedPostBuild/postbuild_0_XXX/Command 01_restart_nginx] : Completed activity. Result:
nginx: [warn] conflicting server name "_" on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "localhost" on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "localhost" on 0.0.0.0:80, ignored
Stopping nginx: [ OK ]
Starting nginx: nginx: [warn] conflicting server name "_" on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "localhost" on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "localhost" on 0.0.0.0:80, ignored
[ OK ]
[2020-01-09T20:22:24.293Z] INFO [1508] - [Application update app-f4fb-200109_211455#11/AppDeployStage0/EbExtensionPostBuild/Infra-EmbeddedPostBuild/postbuild_0_XXX] : Completed activity.
[2020-01-09T20:22:24.375Z] INFO [1508] - [Application update app-f4fb-200109_211455#11/AppDeployStage0/EbExtensionPostBuild/Infra-EmbeddedPostBuild/postbuild_1_XXX] : Starting activity...
[2020-01-09T20:22:37.045Z] INFO [1508] - [Application update app-f4fb-200109_211455#11/AppDeployStage0/EbExtensionPostBuild/Infra-EmbeddedPostBuild/postbuild_1_XXX/Command 10_install_webpack] : Starting activity...
[2020-01-09T20:25:12.155Z] INFO [1508] - [Application update app-f4fb-200109_211455#11/AppDeployStage0/EbExtensionPostBuild/Infra-EmbeddedPostBuild/postbuild_1_XXX/Command 10_install_webpack] : Completed activity. Result:
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents#1.2.11 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents#1.2.11: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
+ webpack#4.41.5
added 322 packages from 197 contributors and audited 4227 packages in 144.209s
3 packages are looking for funding
run `npm fund` for details
found 0 vulnerabilities
[2020-01-09T20:25:13.256Z] INFO [1508] - [Application update app-f4fb-200109_211455#11/AppDeployStage0/EbExtensionPostBuild/Infra-EmbeddedPostBuild/postbuild_1_XXX/Command 11_precompile] : Starting activity...
[2020-01-09T20:30:24.349Z] INFO [1508] - [Application update app-f4fb-200109_211455#11/AppDeployStage0/EbExtensionPostBuild/Infra-EmbeddedPostBuild/postbuild_1_XXX/Command 11_precompile] : Activity execution failed, because: yarn install v1.21.1
warning package-lock.json found. Your project contains lock files generated by tools other than Yarn. It is advised not to mix package managers in order to avoid resolution inconsistencies caused by unsynchronized lock files. To clear this warning, remove package-lock.json.
[1/4] Resolving packages...
[2/4] Fetching packages...
info fsevents#1.2.9: The platform "linux" is incompatible with this module.
info "fsevents#1.2.9" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
Done in 162.47s.
yarn install v1.21.1
warning package-lock.json found. Your project contains lock files generated by tools other than Yarn. It is advised not to mix package managers in order to avoid resolution inconsistencies caused by unsynchronized lock files. To clear this warning, remove package-lock.json.
[1/4] Resolving packages...
[2/4] Fetching packages...
info fsevents#1.2.9: The platform "linux" is incompatible with this module.
info "fsevents#1.2.9" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
[4/4] Building fresh packages...
Done in 83.51s.
I, [2020-01-09T20:29:36.608192 #2970] INFO -- : Writing /var/app/ondeck/public/assets/manifest-cadda289ef9c70eaa0879a36e6263cb33f7523a16b3ef862e0b8609cdc2bdab1.js
I, [2020-01-09T20:29:36.609243 #2970] INFO -- : Writing /var/app/ondeck/public/assets/manifest-cadda289ef9c70eaa0879a36e6263cb33f7523a16b3ef862e0b8609cdc2bdab1.js.gz
I, [2020-01-09T20:29:36.609422 #2970] INFO -- : Writing /var/app/ondeck/public/assets/XXX-c0be3de839559053bb0a9486d5645ccba7a7452f6ef0370ee498e1fa59e364b2.png
I, [2020-01-09T20:29:36.609592 #2970] INFO -- : Writing /var/app/ondeck/public/assets/XXX-469a46bb9645a42d499c7f74ee69ffad4176e08c4373b6fe67a418e8289f3d83.png
I, [2020-01-09T20:29:36.609775 #2970] INFO -- : Writing /var/app/ondeck/public/assets/XXX-e57001fd85a8e1f4da2bb4bbd309a7d880a9d18e3447d743190ff9befb86413f.jpg
I, [2020-01-09T20:29:36.610062 #2970] INFO -- : Writing /var/app/ondeck/public/assets/application-e18be23bdc9236e71700193c31376705b918eab0738fdd68ef83e572da76c13d.css
I, [2020-01-09T20:29:36.610155 #2970] INFO -- : Writing /var/app/ondeck/public/assets/application-e18be23bdc9236e71700193c31376705b918eab0738fdd68ef83e572da76c13d.css.gz
I, [2020-01-09T20:29:36.610249 #2970] INFO -- : Writing /var/app/ondeck/public/assets/wall_street-e57001fd85a8e1f4da2bb4bbd309a7d880a9d18e3447d743190ff9befb86413f.jpg
I, [2020-01-09T20:29:36.610500 #2970] INFO -- : Writing /var/app/ondeck/public/assets/asset-d0ff5974b6aa52cf562bea5921840c032a860a91a3512f7fe8f768f6bbe005f6.css
I, [2020-01-09T20:29:36.610582 #2970] INFO -- : Writing /var/app/ondeck/public/assets/asset-d0ff5974b6aa52cf562bea5921840c032a860a91a3512f7fe8f768f6bbe005f6.css.gz
I, [2020-01-09T20:29:36.690087 #2970] INFO -- : Writing /var/app/ondeck/public/assets/XXX-d0ff5974b6aa52cf562bea5921840c032a860a91a3512f7fe8f768f6bbe005f6.css
I, [2020-01-09T20:29:36.690356 #2970] INFO -- : Writing /var/app/ondeck/public/assets/XXX-d0ff5974b6aa52cf562bea5921840c032a860a91a3512f7fe8f768f6bbe005f6.css.gz
I, [2020-01-09T20:29:36.690743 #2970] INFO -- : Writing /var/app/ondeck/public/assets/main-e18be23bdc9236e71700193c31376705b918eab0738fdd68ef83e572da76c13d.css
I, [2020-01-09T20:29:36.690957 #2970] INFO -- : Writing /var/app/ondeck/public/assets/main-e18be23bdc9236e71700193c31376705b918eab0738fdd68ef83e572da76c13d.css.gz
I, [2020-01-09T20:29:36.691548 #2970] INFO -- : Writing /var/app/ondeck/public/assets/round-d0ff5974b6aa52cf562bea5921840c032a860a91a3512f7fe8f768f6bbe005f6.css
I, [2020-01-09T20:29:36.772944 #2970] INFO -- : Writing /var/app/ondeck/public/assets/round-d0ff5974b6aa52cf562bea5921840c032a860a91a3512f7fe8f768f6bbe005f6.css.gz
Compiling...
Compilation failed:
webpack not installed
Install webpack to start bundling: [32m
$ npm install --save-dev webpack
(ElasticBeanstalk::ExternalInvocationError)
[2020-01-09T20:30:24.349Z] INFO [1508] - [Application update app-f4fb-200109_211455#11/AppDeployStage0/EbExtensionPostBuild/Infra-EmbeddedPostBuild/postbuild_1_XXX/Command 11_precompile] : Activity failed.
[2020-01-09T20:30:24.349Z] INFO [1508] - [Application update app-f4fb-200109_211455#11/AppDeployStage0/EbExtensionPostBuild/Infra-EmbeddedPostBuild/postbuild_1_XXX] : Activity failed.
[2020-01-09T20:30:24.349Z] INFO [1508] - [Application update app-f4fb-200109_211455#11/AppDeployStage0/EbExtensionPostBuild/Infra-EmbeddedPostBuild] : Activity failed.
[2020-01-09T20:30:25.008Z] INFO [1508] - [Application update app-f4fb-200109_211455#11/AppDeployStage0/EbExtensionPostBuild] : Activity failed.
[2020-01-09T20:30:25.008Z] INFO [1508] - [Application update app-f4fb-200109_211455#11/AppDeployStage0] : Activity failed.
[2020-01-09T20:30:25.009Z] INFO [1508] - [Application update app-f4fb-200109_211455#11] : Completed activity. Result:
Application update - Command CMD-AppDeploy failed
[2020-01-09T20:36:59.175Z] INFO [5013] - [CMD-TailLogs] : Starting activity...
[2020-01-09T20:36:59.175Z] INFO [5013] - [CMD-TailLogs/AddonsBefore] : Starting activity...
[2020-01-09T20:36:59.175Z] INFO [5013] - [CMD-TailLogs/AddonsBefore] : Completed activity.
[2020-01-09T20:36:59.175Z] INFO [5013] - [CMD-TailLogs/TailLogs] : Starting activity...
[2020-01-09T20:36:59.175Z] INFO [5013] - [CMD-TailLogs/TailLogs/TailLogs] : Starting activity...
-------------------------------------
/var/log/eb-commandprocessor.log
-------------------------------------
[2020-01-09T20:15:53.363Z] DEBUG [1508] : Retrieving metadata for key: AWS::ElasticBeanstalk::Ext||_API||_Commands..
[2020-01-09T20:15:53.364Z] INFO [1508] : Found enabled addons: ["logpublish", "logstreaming"].
[2020-01-09T20:15:53.444Z] INFO [1508] : Updating Command definition of addon logpublish.
[2020-01-09T20:15:53.444Z] INFO [1508] : Updating Command definition of addon logstreaming.
[2020-01-09T20:15:53.444Z] DEBUG [1508] : Retrieving metadata for key: AWS::CloudFormation::Init||Infra-WriteApplication2||files..
[2020-01-09T20:15:53.445Z] DEBUG [1508] : Retrieving metadata for key: AWS::ElasticBeanstalk::Ext||ManifestFileS3Key..
[2020-01-09T20:15:56.071Z] INFO [1508] : Loading manifest from bucket 'elasticbeanstalk-eu-west-1-511205900284' using computed S3 key 'resources/environments/e-zcsy2pgich/_runtime/versions/manifest_1578600905237'.
[2020-01-09T20:15:56.520Z] INFO [1508] : Updated manifest cache: deployment ID 11 and serial 18.
[2020-01-09T20:15:56.520Z] DEBUG [1508] : Loaded definition of Command CMD-AppDeploy.
[2020-01-09T20:15:56.520Z] INFO [1508] : Executing Application update
[2020-01-09T20:15:56.520Z] INFO [1508] : Executing command: CMD-AppDeploy...
[2020-01-09T20:15:56.520Z] INFO [1508] : Executing command CMD-AppDeploy activities...
[2020-01-09T20:15:56.520Z] DEBUG [1508] : Setting environment variables..
[2020-01-09T20:15:56.520Z] INFO [1508] : Running AddonsBefore for command CMD-AppDeploy...
[2020-01-09T20:15:59.632Z] DEBUG [1508] : Running stages of Command CMD-AppDeploy from stage 0 to stage 1...
[2020-01-09T20:15:59.632Z] INFO [1508] : Running stage 0 of command CMD-AppDeploy...
[2020-01-09T20:15:59.632Z] INFO [1508] : Running leader election...
[2020-01-09T20:16:07.059Z] INFO [1508] : Instance is Leader.
[2020-01-09T20:16:07.059Z] DEBUG [1508] : Loaded 5 actions for stage 0.
[2020-01-09T20:16:07.059Z] INFO [1508] : Running 1 of 5 actions: DownloadSourceBundle...
[2020-01-09T20:16:14.038Z] INFO [1508] : Running 2 of 5 actions: EbExtensionPreBuild...
[2020-01-09T20:18:12.566Z] INFO [1508] : Running 3 of 5 actions: AppDeployPreHook...
[2020-01-09T20:22:02.966Z] INFO [1508] : Running 4 of 5 actions: EbExtensionPostBuild...
[2020-01-09T20:30:25.008Z] ERROR [1508] : Command execution failed: Activity failed. (ElasticBeanstalk::ActivityFatalError)
caused by: yarn install v1.21.1
warning package-lock.json found. Your project contains lock files generated by tools other than Yarn. It is advised not to mix package managers in order to avoid resolution inconsistencies caused by unsynchronized lock files. To clear this warning, remove package-lock.json.
[1/4] Resolving packages...
[2/4] Fetching packages...
info fsevents#1.2.9: The platform "linux" is incompatible with this module.
info "fsevents#1.2.9" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
Done in 162.47s.
yarn install v1.21.1
warning package-lock.json found. Your project contains lock files generated by tools other than Yarn. It is advised not to mix package managers in order to avoid resolution inconsistencies caused by unsynchronized lock files. To clear this warning, remove package-lock.json.
[1/4] Resolving packages...
[2/4] Fetching packages...
info fsevents#1.2.9: The platform "linux" is incompatible with this module.
info "fsevents#1.2.9" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
[4/4] Building fresh packages...
Done in 83.51s.
I, [2020-01-09T20:29:36.608192 #2970] INFO -- : Writing /var/app/ondeck/public/assets/manifest-cadda289ef9c70eaa0879a36e6263cb33f7523a16b3ef862e0b8609cdc2bdab1.js
I, [2020-01-09T20:29:36.609243 #2970] INFO -- : Writing /var/app/ondeck/public/assets/manifest-cadda289ef9c70eaa0879a36e6263cb33f7523a16b3ef862e0b8609cdc2bdab1.js.gz
I, [2020-01-09T20:29:36.609422 #2970] INFO -- : Writing /var/app/ondeck/public/assets/XXX-c0be3de839559053bb0a9486d5645ccba7a7452f6ef0370ee498e1fa59e364b2.png
I, [2020-01-09T20:29:36.609592 #2970] INFO -- : Writing /var/app/ondeck/public/assets/XXX-469a46bb9645a42d499c7f74ee69ffad4176e08c4373b6fe67a418e8289f3d83.png
I, [2020-01-09T20:29:36.609775 #2970] INFO -- : Writing /var/app/ondeck/public/assets/XXX-e57001fd85a8e1f4da2bb4bbd309a7d880a9d18e3447d743190ff9befb86413f.jpg
I, [2020-01-09T20:29:36.610062 #2970] INFO -- : Writing /var/app/ondeck/public/assets/application-e18be23bdc9236e71700193c31376705b918eab0738fdd68ef83e572da76c13d.css
I, [2020-01-09T20:29:36.610155 #2970] INFO -- : Writing /var/app/ondeck/public/assets/application-e18be23bdc9236e71700193c31376705b918eab0738fdd68ef83e572da76c13d.css.gz
I, [2020-01-09T20:29:36.610249 #2970] INFO -- : Writing /var/app/ondeck/public/assets/XXX-e57001fd85a8e1f4da2bb4bbd309a7d880a9d18e3447d743190ff9befb86413f.jpg
I, [2020-01-09T20:29:36.610500 #2970] INFO -- : Writing /var/app/ondeck/public/assets/asset-d0ff5974b6aa52cf562bea5921840c032a860a91a3512f7fe8f768f6bbe005f6.css
I, [2020-01-09T20:29:36.610582 #2970] INFO -- : Writing /var/app/ondeck/public/assets/asset-d0ff5974b6aa52cf562bea5921840c032a860a91a3512f7fe8f768f6bbe005f6.css.gz
I, [2020-01-09T20:29:36.690087 #2970] INFO -- : Writing /var/app/ondeck/public/assets/XXX-d0ff5974b6aa52cf562bea5921840c032a860a91a3512f7fe8f768f6bbe005f6.css
I, [2020-01-09T20:29:36.690356 #2970] INFO -- : Writing /var/app/ondeck/public/assets/XXX-d0ff5974b6aa52cf562bea5921840c032a860a91a3512f7fe8f768f6bbe005f6.css.gz
I, [2020-01-09T20:29:36.690743 #2970] INFO -- : Writing /var/app/ondeck/public/assets/main-e18be23bdc9236e71700193c31376705b918eab0738fdd68ef83e572da76c13d.css
I, [2020-01-09T20:29:36.690957 #2970] INFO -- : Writing /var/app/ondeck/public/assets/main-e18be23bdc9236e71700193c31376705b918eab0738fdd68ef83e572da76c13d.css.gz
I, [2020-01-09T20:29:36.691548 #2970] INFO -- : Writing /var/app/ondeck/public/assets/round-d0ff5974b6aa52cf562bea5921840c032a860a91a3512f7fe8f768f6bbe005f6.css
I, [2020-01-09T20:29:36.772944 #2970] INFO -- : Writing /var/app/ondeck/public/assets/round-d0ff5974b6aa52cf562bea5921840c032a860a91a3512f7fe8f768f6bbe005f6.css.gz
Compiling...
Compilation failed:
webpack not installed
Install webpack to start bundling: [32m
$ npm install --save-dev webpack
(ElasticBeanstalk::ExternalInvocationError)
[2020-01-09T20:30:25.009Z] ERROR [1508] : Command CMD-AppDeploy failed!
[2020-01-09T20:30:25.086Z] INFO [1508] : Command processor returning results:
{"status":"FAILURE","api_version":"1.0","results":[{"status":"FAILURE","msg":"(TRUNCATED)...NFO -- : Writing /var/app/ondeck/public/assets/round-d0ff5974b6aa52cf562bea5921840c032a860a91a3512f7fe8f768f6bbe005f6.css.gz\nCompiling...\nCompilation failed:\n\nwebpack not installed\n\nInstall webpack to start bundling: \u001b[32m\n$ npm install --save-dev webpack. \ncontainer_command 11_precompile in .ebextensions/yarn.config failed. For more detail, check /var/log/eb-activity.log using console or EB CLI","returncode":1,"events":[]}],"truncated":"true"}
Logically, I tried to install webpack again and again which should be normally already be done through the ebextensions file yarn.config:
commands:
# 01_remove_clean_and_install_latest_nodejs:
# run this command from /tmp directory
# cwd: /tmp
# test: '[ -f /usr/bin/node ] && echo "remove previous node"'
# command: 'sudo yum remove -y nodejs | sudo rm /etc/yum.repos.d/nodesource*'
02_node_get:
# run this command from /tmp directory
cwd: /tmp
# flag -y for no-interaction installation
command: 'sudo curl --silent --location https://rpm.nodesource.com/setup_13.x | sudo bash -'
03_node_install:
# run this command from /tmp directory
cwd: /tmp
command: 'sudo yum -y install nodejs'
04_yarn_get:
# run this command from /tmp directory
cwd: /tmp
# don't run the command if yarn is already installed (file /usr/bin/yarn exists)
test: '[ ! -f /usr/bin/yarn ] && echo "yarn not installed"'
command: 'sudo wget https://dl.yarnpkg.com/rpm/yarn.repo -O /etc/yum.repos.d/yarn.repo'
05_yarn_install:
# run this command from /tmp directory
cwd: /tmp
test: '[ ! -f /usr/bin/yarn ] && echo "yarn not installed"'
command: 'sudo yum -y install yarn'
06_mkdir_webapp_dir:
command: mkdir /home/webapp
ignoreErrors: true
07_chown_webapp_dir:
command: chown webapp:webapp /home/webapp
ignoreErrors: true
08_chmod_webapp_dir:
command: chmod 700 /home/webapp
ignoreErrors: true
09_update_bundler:
command: gem update bundler
ignoreErrors: true
container_commands:
10_install_webpack:
command: "sudo npm install --save-dev webpack"
11_precompile:
command: "bundle exec rake assets:precompile"
My other settings are RAILS_SKIP_ASSET_COMPILATION=true in eb
I tried installing webpack manually after ssh-ing into the machine via npm and yarn, with and without sudo, but nothing worked...
Each eb deploy takes roughly 20 mins and then times out into a "severe" state with the above error.
Edit
For some weird reason I got it to work now by running bundle exec rails webpacker:install in addition to having it in the Gemfile. Still not comfortable with the setup, but at least it's working for now
Update: We had two elasticsearch gems that were not being installed during bundle install but removing them from our gemfile allowed the deployment to go through. We noticed the issue locally as well with a bundle install that hung on "resolving dependencies" for about 4 hours, which is roughly how long it took our deployment to successfully go through. I'd start checking gems.
I'm not sure if this is related, but we have an open case with AWS support currently over a similar issue where deployments seemingly fail after hanging on some of AWS's own PreDeploy hook scripts (namely one that checks for Puma). We have been unable to successfully clone or deploy known working app code bundles to any new or existing instances. The issue started on the 7th of January, 2019.
AWS has since confirmed that the issue is on their end, but we have no remedy yet.
Sidenote: Code eventually will deploy after about 4 hours, but the instances all show degraded or error.

How is the correct syntax/call to get the sls-file working - salt

I'm trying to build a reactor sls file, which starts running when an event occurs.
The content of the sls file should be as the following cli commands:
sudo salt minion git.add /srv/salt .
sudo salt minion git.commit /srv/salt test
sudo salt minion git.push /srv/salt origin master identity=/home/autogit/.ssh/id_rsa
If i run the code bellow triggered by the reactor. I get the following error message.
[DEBUG ] Reactor is populating module client cache
[ERROR ] An un-handled exception from the multiprocessing process 'Reactor-9:1' was caught:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/salt/utils/process.py", line 765, in _run
return self._original_run()
File "/usr/lib/python2.7/dist-packages/salt/utils/reactor.py", line 271, in run
self.call_reactions(chunks)
File "/usr/lib/python2.7/dist-packages/salt/utils/reactor.py", line 228, in call_reactions
self.wrap.run(chunk)
File "/usr/lib/python2.7/dist-packages/salt/utils/reactor.py", line 330, in run
self.populate_client_cache(low)
File "/usr/lib/python2.7/dist-packages/salt/utils/reactor.py", line 324, in populate_client_cache
self.reaction_class[reaction_type](self.opts['conf_file'])
KeyError: u'module'
[CRITICAL] Engine 'reactor' could not be started!
I've tried different syntax (old style and new style) but couldn't figure out what the problem is. Always getting an KeyError: u'module' or u'git'.
Also tried it with runner function to run it locally on the master.
git pull:
module.run:
- git.pull:
- cwd: /srv/salt
- remote: git#git.xyz.com:user/sbt.git
- identity: /home/autogit/.ssh/id_rsa
- git.add:
- cwd: /srv/salt
- filename: .
- git.commit:
- cwd: /srv/salt
- remote: git#git.xyz.com:user/sbt.git
- git.push:
- cwd: /srv/salt
- remote: git#git.xyz.com:user/sbt.git
- identity: /home/autogit/.ssh/id_rsa
salt --versions-report
Salt Version:
Salt: 2019.2.0
Dependency Versions:
cffi: Not Installed
cherrypy: unknown
dateutil: 2.6.1
docker-py: Not Installed
gitdb: 2.0.3
gitpython: 2.1.8
ioflo: Not Installed
Jinja2: 2.10
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 1.0.7
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.15rc1 (default, Nov 12 2018, 14:31:15)
python-gnupg: 0.4.1
PyYAML: 3.12
PyZMQ: 16.0.2
RAET: Not Installed
smmap: 2.0.3
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.2.5
System Versions:
dist: Ubuntu 18.04 bionic
locale: UTF-8
machine: x86_64
release: 4.15.0-46-generic
system: Linux
version: Ubuntu 18.04 bionic
Since i'm quite new to Salt, hopefully you can give me a hint what i'm doing wrong:
You didn't provide the master config.
About module.run confusion: add in your settings (minion and maybe to master since I don't know your use-case)
use_superseded:
- module.run
That will enable your syntax, more doc about this here: https://docs.saltstack.com/en/latest/ref/states/all/salt.states.module.html#salt.states.module.run
In general: you are executing execution modules from the place that state modules are allowed only (the term module is heavily overused in salt...)
You didn't provide the full Master config. Reactor requires dedicated config to match events to sls files:
https://docs.saltstack.com/en/latest/ref/configuration/master.html#master-reactor-settings
You can also check the doc I've written some time ago about events and reactors:
https://github.com/kiemlicz/util/wiki/Salt-Events-and-Reactor
Assuming you've configured your events-to-sls-files-matching in master config, your provided sls:
git pull:
module.run:
- git.pull:
- cwd: /srv/salt
- remote: git#git.xyz.com:user/
...
will not work.
Mind that reaction happens on Salt Master thus the reaction sls file need to provide type of reaction (local, runner etc.) since it's no longer 'view of one minion' but possibly of tons of minions!
First create runner reaction type (which delegates to some orchestration sls file which will contain your logic wrapped with (I think) salt.function )
Help yourself with aforementioned github link to my attempt of explaining Reactor.
Refer to official doc as well: https://docs.saltstack.com/en/latest/topics/reactor/index.html

Vagrant - Ansible error installing Apache

I'm working on a project with Vagrant and Ansible and Virtualbox.
When I try to install Apache on an ubuntu precise (14.04) box, Vagrant fails. I improved the answer after.
It seems a known bug, but even if I'm installing a newer version, the error shows up.
I tried also as stated here, but with no luck.
How can I resolve this issue?
Thank you.
UPDATED ANSWER
This is the Ansible task.
Version 1:
- name: Install Apache
sudo: yes
apt: pkg=apache2 state=latest
register: apache2_apt
Output:
failed: [default] => {"failed": true}
stderr: E: Sub-process /usr/bin/dpkg returned an error code (1)
stdout: Reading package lists...
Building dependency tree...
Reading state information...
Suggested packages:
www-browser apache2-doc apache2-suexec-pristine apache2-suexec-custom
The following NEW packages will be installed:
apache2
0 upgraded, 1 newly installed, 0 to remove and 183 not upgraded.
Need to get 0 B/146 kB of archives.
After this operation, 460 kB of additional disk space will be used.
(Reading database ... 52932 files and directories currently installed.)
Unpacking apache2 (from .../apache2_2.4.12-1+deb.sury.org~precise+5_amd64.deb) ...
dpkg: error processing /var/cache/apt/archives/apache2_2.4.12-1+deb.sury.org~precise+5_amd64.deb (--unpack):
error setting ownership of `/var/www/html.dpkg-new': Operation not permitted
Processing triggers for man-db ...
Processing triggers for ureadahead ...
Errors were encountered while processing:
/var/cache/apt/archives/apache2_2.4.12-1+deb.sury.org~precise+5_amd64.deb
msg: '/usr/bin/apt-get -y -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" install 'apache2'' failed: E: Sub-process /usr/bin/dpkg returned an error code (1)
FATAL: all hosts have already failed -- aborting
Version 2:
- name: Install Apache
command: "sudo apt-get install apache2"
register: apache2_apt
Output:
failed: [default] => {"changed": true, "cmd": ["sudo", "apt-get", "install", "apache2"], "delta": "0:00:07.745095", "end": "2015-06-09 11:08:53.726031", "rc": 100, "start": "2015-06-09 11:08:45.980936", "warnings": []}
stderr: E: Sub-process /usr/bin/dpkg returned an error code (1)
stdout: Reading package lists...
Building dependency tree...
Reading state information...
Suggested packages:
www-browser apache2-doc apache2-suexec-pristine apache2-suexec-custom
The following NEW packages will be installed:
apache2
0 upgraded, 1 newly installed, 0 to remove and 183 not upgraded.
Need to get 0 B/146 kB of archives.
After this operation, 460 kB of additional disk space will be used.
(Reading database ... 52932 files and directories currently installed.)
Unpacking apache2 (from .../apache2_2.4.12-1+deb.sury.org~precise+5_amd64.deb) ...
dpkg: error processing /var/cache/apt/archives/apache2_2.4.12-1+deb.sury.org~precise+5_amd64.deb (--unpack):
error setting ownership of `/var/www/html.dpkg-new': Operation not permitted
Processing triggers for man-db ...
Processing triggers for ureadahead ...
ureadahead will be reprofiled on next reboot
Errors were encountered while processing:
/var/cache/apt/archives/apache2_2.4.12-1+deb.sury.org~precise+5_amd64.deb
FATAL: all hosts have already failed -- aborting
There are few possible issues for this
You need to disable apparmor or better add a rule to apparmor service for ability to have access by the script to /var/www within guest machine
There is a trouble with host machine permissions for /var/www folder. Try to check if the user has access to local folder, mapped as shared folder from host to guest - possibly you need to add permissions for local user at host machine.
try to use ansible-galaxy and search for already created role with fixes for both previous issues

Nutch 1.8 and Apache Solr 4.8 integration job fail

I am trying to crawl the web using Nutch 1.8 and Solr 4.8 on Windows 7.
bin/crawl urls newsolr http://localhost:8983/solr/ 1 -depth 1
I keep getting the following error
Indexer: java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
at org.apache.nutch.indexer.IndexingJob.index(IndexingJob.java:114)
at org.apache.nutch.indexer.IndexingJob.run(IndexingJob.java:176)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.indexer.IndexingJob.main(IndexingJob.java:186)
Here is the part of the log file:
2014-07-01 16:58:33,613 INFO solr.SolrMappingReader - source: content dest: content
2014-07-01 16:58:33,613 INFO solr.SolrMappingReader - source: title dest: title
2014-07-01 16:58:33,613 INFO solr.SolrMappingReader - source: host dest: host
2014-07-01 16:58:33,613 INFO solr.SolrMappingReader - source: segment dest: segment
2014-07-01 16:58:33,613 INFO solr.SolrMappingReader - source: boost dest: boost
2014-07-01 16:58:33,613 INFO solr.SolrMappingReader - source: digest dest: digest
2014-07-01 16:58:33,613 INFO solr.SolrMappingReader - source: tstamp dest: tstamp
2014-07-01 16:58:33,613 INFO solr.SolrMappingReader - source: url dest: id
2014-07-01 16:58:33,613 INFO solr.SolrMappingReader - source: url dest: url
2014-07-01 16:58:33,643 INFO solr.SolrIndexWriter - Indexing 1 documents
2014-07-01 16:58:33,773 WARN mapred.LocalJobRunner - job_local_0001
org.apache.solr.common.SolrException: Method Not Allowed
Method Not Allowed
request: http://localhost:8983/solr/
at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:430)
at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:244)
at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105)
at org.apache.nutch.indexwriter.solr.SolrIndexWriter.close(SolrIndexWriter.java:155)
at org.apache.nutch.indexer.IndexWriters.close(IndexWriters.java:118)
at org.apache.nutch.indexer.IndexerOutputFormat$1.close(IndexerOutputFormat.java:44)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:474)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:411)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:216)
2014-07-01 16:58:34,628 ERROR indexer.IndexingJob - Indexer: java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
at org.apache.nutch.indexer.IndexingJob.index(IndexingJob.java:114)
at org.apache.nutch.indexer.IndexingJob.run(IndexingJob.java:176)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.indexer.IndexingJob.main(IndexingJob.java:186)
Finally, Solr's error log:
org.apache.solr.common.SolrException: ERROR: [doc=http://.com/] unknown field 'tstamp' `
This is my first solr/nutch setup.
Just stop the solr instance and start it again. It should solve your problem.
The error occurred because you made changes to the schema file and didn't restart solr in order for the changes to be saved hence solr is unable to "see" the newly added fields.

what does "can not write log (is /dev/pts mounted") mean in apt-get in Ubuntu-on-Windows?

What exactly does the error message beginning with E: mean? I assume it has to do with the file structure on Ubuntu-on-Windows, but what exactly?
The following package was automatically installed and is no longer required:
os-prober
Use 'apt-get autoremove' to remove it.
The following extra packages will be installed:
libxslt1.1
The following NEW packages will be installed:
libxslt1.1 xmlstarlet
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 435 kB of archives.
After this operation, 1023 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu/ trusty/main libxslt1.1 amd64 1.1.28-2build1 [145 kB]
Get:2 http://archive.ubuntu.com/ubuntu/ trusty/universe xmlstarlet amd64 1.5.0-1 [290 kB]
Fetched 435 kB in 3s (134 kB/s)
**E: Can not write log (Is /dev/pts mounted?) - openpty (2: No such file or directory)**
Selecting previously unselected package libxslt1.1:amd64.
(Reading database ... 28723 files and directories currently installed.)
Preparing to unpack .../libxslt1.1_1.1.28-2build1_amd64.deb ...
Unpacking libxslt1.1:amd64 (1.1.28-2build1) ...
Selecting previously unselected package xmlstarlet.
Preparing to unpack .../xmlstarlet_1.5.0-1_amd64.deb .