I'm trying to run pig on tez on amazon emr 4.5.0. The configuration works without tez, I'm just trying to get it to work on Tez.
To create the cluster (from the command line), we're using (TEZ_VERSION is defined as 0.5.2):
--bootstrap-actions Path=s3://support.elasticmapreduce/tez/bigtop/install-tez.rb,Args=[-v,$TEZ_VERSION,--tez-site,tez.lib.uris=s3://support.elasticmapreduce/tez/$TEZ_VERSION/tez-$TEZ_VERSION-minimal.tar.gz]
Additionally, I'm overwriting the PIG_CLASSPATH:
--configurations file://pig_tez_classification.json
Containing:
[
{
"Classification": "hadoop-env",
"Properties": {
},
"Configurations": [
{
"Classification": "export",
"Properties": {
"PIG_CLASSPATH": "/etc/tez/conf"
},
"Configurations": [
]
}
]
}
]
The PIG_CLASSPATH is needed to prevent this error:
ERROR org.apache.pig.backend.hadoop.executionengine.tez.TezJob - Cannot submit DAG
org.apache.tez.dag.api.TezUncheckedException: Invalid configuration of tez jars, tez.lib.uris is not defined in the configuration
The tez.lib.uris overwrite is needed to prevent this error:
ERROR org.apache.pig.backend.hadoop.executionengine.tez.TezJob (PigTezLauncher-0): Cannot submit DAG
java.io.FileNotFoundException: File does not exist: hdfs://ip-172-31-3-207.eu-west-1.compute.internal:8020/apps/tez/0.5.2/tez-0.5.2-minimal.tar.gz
It appears the install script writes the tar.gz file to the right location in the hdfs, but when I log in over ssh afterwards, the file is not there. I think in EMR-4 the bootstrap actions are run at a different time, so before hdfs is available?
After all that, I still get this error:
WARN org.apache.pig.backend.hadoop.executionengine.tez.TezJob - Exception while gathering stats
java.lang.NullPointerException
at org.apache.pig.tools.pigstats.tez.TezDAGStats.accumulateStats(TezDAGStats.java:191)
at org.apache.pig.tools.pigstats.tez.TezPigScriptStats.accumulateStats(TezPigScriptStats.java:180)
at org.apache.pig.backend.hadoop.executionengine.tez.TezJob.run(TezJob.java:194)
at org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher$1.run(TezLauncher.java:167)
Trying with tez version 0.8.2 yields:
ERROR org.apache.pig.backend.hadoop.executionengine.tez.TezJob - Cannot submit DAG
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown
Which seems to be due to a difference in used tez version, because it still prints out:
INFO org.apache.pig.tools.pigstats.tez.TezPigScriptStats - Script Statistics:
HadoopVersion: 2.7.2-amzn-0
PigVersion: 0.14.0-amzn-0
TezVersion: 0.5.2
UserId: hadoop
So does anyone know how to get pig on tez on amazon emr (whatever versions) running?
I'm successfully running it with emr-4.4.0. However, I was unable to get pig to work correctly with the tar.gz uploaded to HDFS. Instead, I had to unpack the tarball and upload all the individual files, then set tez.lib.uris to hdfs:///apps/tez-0.8.2,hdfs:///apps/tez-0.8.2/lib
Related
I am trying to scan all images deployed on my EKS cluster I am setting up for high security (will be deployed to classified IL5 environment). Kubernetes v1.23, all worker nodes run on Bottlerocket OS.
I expect images to be scanned and available in the VulnerabilityReports CRD.
I was able to successfully install Falco to the cluster (uses containerd). However, when deploying the official Helm chart (0.6.0-rc3) the scan-vulnerability containers start and then immediately error out. I set this environment variable on the trivy-operator deployment:
- name: CONTAINER_RUNTIME_ENDPOINT
value: /run/containerd/containerd.sock
Output of run with -debug:
{
"level": "error",
"ts": 1668286646.865245,
"logger": "reconciler.vulnerabilityreport",
"msg": "Scan job container",
"job": "trivy-system/scan-vulnerabilityreport-74f54b6cd",
"container": "discovery",
"status.reason": "Error",
"status.message": "2022-11-12T20:57:13.674Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: unable to initialize a scanner: unable to initialize a docker scanner: 4 errors occurred:\n\t* unable to inspect the image (023620263533.dkr.ecr.us-gov-east-1.amazonaws.com/docker.io/istio/pilot:1.15.2): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?\n\t* unable to initialize Podman client: no podman socket found: stat podman/podman.sock: no such file or directory\n\t* containerd socket not found: /run/containerd/containerd.sock\n\t* GET https://023620263533.dkr.ecr.us-gov-east-1.amazonaws.com/v2/docker.io/istio/pilot/manifests/1.15.2: unexpected status code 401 Unauthorized: Not Authorized\n\n\n\n",
"stacktrace": "github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport.(*WorkloadController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller.go:551\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport.(*WorkloadController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller.go:376\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime#v0.13.1/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime#v0.13.1/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime#v0.13.1/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime#v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime#v0.13.1/pkg/internal/controller/controller.go:234"
}
I confirmed that bottlerocket uses containerd, as /run/containerd/containerd.sock is specified on my Falco deployment. Even when I mount this as volume onto the pod and set the CONTAINER_RUNTIME_ENDPOINT to this path I get the same error.
Edit
I added the following security context:
seLinuxOptions:
user: system_u
role: system_r
type: control_t
level: s0-s0:c0.c1023
Initially I mounted the dockershim.sock from the host to the pod, then realized that was not necessary, the error messages were a little misleading, it was really an authentication with ECR issue. Furthermore, the seLinux flags needed to be specified at the pod level, and not the container level.
my team is migrating to rush and I had a problem while setting it up: I'm getting a weird error when trying to add any 3rd party lib (let's say react) to package.json using rush add --package react (for example). The error log:
Found configuration in /home/ivan/dev/JS/eisbuk/rush.json
Rush Multi-Project Build Tool 5.66.2 - https://rushjs.io
Node.js version is 14.19.0 (LTS)
Found configuration in /home/ivan/dev/JS/eisbuk/rush.json
Starting "rush add"
Determining new version for dependency: react
No version selector was specified, so the version will be determined automatically.
Trying to acquire lock for pnpm-6.32.3
Acquired lock for pnpm-6.32.3
Found pnpm version 6.32.3 in /home/ivan/.rush/node-v14.19.0/pnpm-6.32.3
Symlinking "/home/ivan/dev/JS/eisbuk/common/temp/pnpm-local" --> "/home/ivan/.rush/node-v14.19.0/pnpm-6.32.3"
The "ensureConsistentVersions" policy is NOT active, so we will assign the latest version.
Querying NPM registry for latest version of "react"...
ERROR: The command failed with exit code 243
I tried to find a similar problem report related to rush and looked up the 243 error with npm, both to no avail. I'm using pnpm, however, the error persists with npm and yarn all the same (with yarn it's exit code 1, everything else is the same).
Here's my .npmrc (a default really):
registry=https://registry.npmjs.org/
always-auth=false
Interestingly enough, if I manually add a dependency to package.json and run rush update, the update goes well and the dep is added to node_modules and shrinkwrap (so it shouldn't be an npm registry problem).
Here's my rush.json:
{
"$schema": "https://developer.microsoft.com/json-schemas/rush/v5/rush.schema.json",
"rushVersion": "5.66.2",
"pnpmVersion": "6.32.3",
"pnpmOptions": {
"preventManualShrinkwrapChanges": true,
"useWorkspaces": true
},
"nodeSupportedVersionRange": ">=12.13.0 <13.0.0 || >=14.15.0 <15.0.0 || >=16.13.0 <17.0.0",
"gitPolicy": {},
"repository": {},
"eventHooks": {
"preRushInstall": [],
"postRushInstall": [],
"preRushBuild": [],
"postRushBuild": []
},
"variants": [],
"projects": [
/** ...other projects */
{
"packageName": "#eisbuk/functions",
"projectFolder": "packages/functions"
}
/** ...other projects */
]
}
The entire repo can be found at: https://github.com/eisbuk/EisBuk/tree/feature/rush
Solved!
In the end, I've managed to pinpoint it to the node version. I was using v14.19.0 and bumping to v16.14.2 solved it. I don't know if it's a known incompatibility, but here it is.
I'm trying to set up my environment to use Yocto's generated SDK to compile my out-of-tree module, but for some reason, I'm getting an error.
cp: cannot stat 'arch/arm/kernel/module.lds': No such file or directory
I'm using Poky distribution and meta-raspberrypi which is needed because I'm using the RPI ZeroW board.
Apart from this everything works fine. I'm able to compile the entire image and load it on the board.
Here is the line I've added to local.conf
TOOLCHAIN_TARGET_TASK_append = " kernel-devsrc"
as I've found in the documentation.
Also below you can find the whole log from the compilation.
DEBUG: Executing python function extend_recipe_sysroot
NOTE: Direct dependencies are ['virtual:native:/home/pp/yocto-hh/sources/poky/meta/recipes-bsp/u-boot/u-boot-tools_2020.07.bb:do_populate_sysroot', '/home/pp/yocto-hh/sources/poky/meta/recipes-devtools/binutils/binutils-cross_2.35.1.bb:do_populate_sysroot', '/home/pp/yocto-hh/sources/poky/meta/recipes-kernel/kmod/kmod-native_git.bb:do_populate_sysroot', '/home/pp/yocto-hh/sources/poky/meta/recipes-devtools/gcc/gcc-cross_10.2.bb:do_populate_sysroot', 'virtual:native:/home/pp/yocto-hh/sources/poky/meta/recipes-support/gmp/gmp_6.2.0.bb:do_populate_sysroot', 'virtual:native:/home/pp/yocto-hh/sources/poky/meta/recipes-devtools/pkgconfig/pkgconfig_git.bb:do_populate_sysroot', 'virtual:native:/home/pp/yocto-hh/sources/poky/meta/recipes-extended/xz/xz_5.2.5.bb:do_populate_sysroot', 'virtual:native:/home/pp/yocto-hh/sources/poky/meta/recipes-connectivity/openssl/openssl_1.1.1k.bb:do_populate_sysroot', 'virtual:native:/home/pp/yocto-hh/sources/poky/meta/recipes-devtools/bison/bison_3.7.2.bb:do_populate_sysroot', '/home/pp/yocto-hh/sources/poky/meta/recipes-devtools/quilt/quilt-native_0.66.bb:do_populate_sysroot', 'virtual:native:/home/pp/yocto-hh/sources/poky/meta/recipes-extended/bc/bc_1.07.1.bb:do_populate_sysroot', '/home/pp/yocto-hh/sources/poky/meta/recipes-core/glibc/glibc_2.32.bb:do_populate_sysroot', 'virtual:native:/home/pp/yocto-hh/sources/poky/meta/recipes-devtools/patch/patch_2.7.6.bb:do_populate_sysroot', '/home/pp/yocto-hh/sources/poky/meta/recipes-kernel/kern-tools/kern-tools-native_git.bb:do_populate_sysroot', 'virtual:native:/home/pp/yocto-hh/sources/poky/meta/recipes-devtools/pseudo/pseudo_git.bb:do_populate_sysroot', '/home/pp/yocto-hh/sources/poky/meta/recipes-devtools/gcc/gcc-runtime_10.2.bb:do_populate_sysroot']
NOTE: Installed into sysroot: []
NOTE: Skipping as already exists in sysroot: ['u-boot-tools-native', 'binutils-cross-arm', 'kmod-native', 'gcc-cross-arm', 'gmp-native', 'pkgconfig-native', 'xz-native', 'openssl-native', 'bison-native', 'quilt-native', 'bc-native', 'glibc', 'patch-native', 'kern-tools-native', 'pseudo-native', 'gcc-runtime', 'python3-native', 'gnu-config-native', 'autoconf-native', 'libtool-native', 'gtk-doc-native', 'automake-native', 'zlib-native', 'texinfo-dummy-native', 'readline-native', 'flex-native', 'attr-native', 'libmpc-native', 'mpfr-native', 'linux-libc-headers', 'gettext-minimal-native', 'libgcc', 'libnsl2-native', 'gdbm-native', 'libffi-native', 'bzip2-native', 'util-linux-native', 'sqlite3-native', 'libtirpc-native', 'm4-native', 'ncurses-native', 'libcap-ng-native', 'libpcre2-native']
DEBUG: Python function extend_recipe_sysroot finished
DEBUG: Executing shell function do_install
cp: cannot stat 'arch/arm/kernel/module.lds': No such file or directory
WARNING: exit code 1 from a shell command.
ERROR: Execution of '/home/pp/yocto-hh/build/tmp/work/hhctrl-poky-linux-gnueabi/kernel-devsrc/1.0-r0/temp/run.do_install.109942' failed with exit code 1:
cp: cannot stat 'arch/arm/kernel/module.lds': No such file or directory
WARNING: exit code 1 from a shell command
The command I am using to produce the SDK is:
bitbake name-of-my-image -c populate_sdk
What could be the problem here? Or how should I debug it? I've found a few touching on the subject and it seems that it should be already fixed, but for some reason, in my environment, it still does not work.
Missing the module.lds file in the latest kernel. Apply the following source code as a patch in the kernel and build the image.
diff -Naur a/arch/arm/kernel/module.lds b/arch/arm/kernel/module.lds
--- a/arch/arm/kernel/module.lds 1970-01-01 05:30:00.000000000 +0530
+++ b/arch/arm/kernel/module.lds 2020-02-28 21:53:45.000000000 +0530
## -0,0 +1,5 ##
+/* SPDX-License-Identifier: GPL-2.0 */
+SECTIONS {
+ .plt : { BYTE(0) }
+ .init.plt : { BYTE(0) }
+}
Telegraf version: Telegraf v1.0.1
I have the following files:
/etc/telegraf/telegraf.conf
main config file (which you create/get for free while installing telegraf). This file doesn't have any [[output.<plugin>]] stanza in it but does have valid default inputs.xx plugins enabled.
/etc/telegraf/telegraf.d/1-company-output-plugin.conf
(This file has valid outputs.<pluginname> plugin).
/etc/telegraf/telegraf.d/telegraf-additional-inputs-plugins.conf
(this file has additional inputs.<pluginname> plugins that I want to keep separate from the main conf file).
Running sudo service telegraf status shows Telegraf is running [OK]! and log file /var/log/telegrag/telegraf.log (looks good too without showing any errors).
If I make any changes to one of the above configuration files, I want to test those changes made (before I restart telegraf), so I'm running the following commands with -test or --test option but I'm getting the following error:
ubuntu#jenkins:~/test_aks_dir$ telegraf --config /etc/telegraf/telegraf.conf -test
2017/01/10 22:45:48 E! Error: no outputs found, did you provide a valid config file?
ubuntu#jenkins:~/test_aks_dir$ telegraf --config /etc/telegraf/telegraf.conf --test
2017/01/10 22:45:51 E! Error: no outputs found, did you provide a valid config file?
As you see above, both optons -test or --test worked but also gave a valid error message above.
As the above error was for not providing any configuration stanza for [[outputs.<plugin>]], I provided another --config file paramter for the file which contains the outputs plugin stanza, but now it gave an error for the missing inputs. file:
ubuntu#jenkins:~/test_aks_dir$ telegraf --config /etc/telegraf/telegraf.conf --config /etc/telegraf/telegraf.d/1-company-output-plugin.conf --test
2017/01/10 22:48:30 E! Error: no inputs found, did you provide a valid config file?
ubuntu#jenkins:~/test_aks_dir$
Note: The above command didn't error for providing multiple --config <someConfigFile> parameter values.
The same error comes if I want to test any change in the 3rd file : (/etc/telegraf/telegraf.d/telegraf-additional-inputs-plugins.conf)
How can I test an individual or selective configuration file changes using telegraf assuming I have multiple files here: /etc/telegraf/telegraf.d/*.conf ?
this is not possible at the moment, the only way to do it would be to specify the entire config directory
I am trying to run pig script in local mode on a single node cluster as given below.
hduser#ubuntu:~$ pig -x local -f "/home/hduser/ddsoft/pigscript/FirstUDF.pig"
But I am getting below error.
[main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 101: file
'/home/hduser/ddsoft/hive-0.13.1-bin/hcatalog/share/hcatalog/hcatalog-core-0.13.1.jar'
does not exist.
how do I register the jar file mentioned in the error message. I tried updating the .bashrc, but it didn’t fix the error.