The documentation says
In Amazon EMR releases 4.0 and greater, the only accepted parameter is
the application name. To pass arguments to applications, you supply a
configuration for each application.
But I cannot find an example that shows how to pass arguments in ami 4. All I can find are examples configuring exports such as below. I am trying to figure out how to set the version of Spark to use.
[
{
"Classification":"hadoop-env",
"Properties":{
},
"Configurations":[
{
"Classification":"export",
"Properties":{
"HADOOP_USER_CLASSPATH_FIRST":"true",
"HADOOP_CLASSPATH":"/path/to/my.jar"
}
}
]
}
]
You cannot set an arbitrary version of Spark to use like you could with 3.x AMI versions. Rather, the version of Spark (and other apps, of course) is determined by the release label. For example, the latest release is currently emr-5.2.1, which includes Spark 2.0.2. If you want a 1.x version of Spark, the latest version available is Spark 1.6.3 on release emr-4.8.3.
Related
I'm unable to find a buildpack builder that lends itself to Spring Webflux applications. Paketo, for example, has little room for customization and out of the box includes buildpacks that are not needed for a typical Webflux app (e.g. apache-tomcat). Are there any buildpacks tailored for webflux/jvm-reactive applications?
It doesn't look like you need to do anything specific here for Spring Webflux, the Java related CNBs will just do the right thing.
I took a sample app that I created with Spring Initializer (just added the Webflux starter) and ran pack build against it (if you run ./mvnw spring-boot:build-image, you should get very similar output).
That gave me the following output:
===> DETECTING
[detector] 6 of 17 buildpacks participating
[detector] paketo-buildpacks/bellsoft-liberica 3.2.0
[detector] paketo-buildpacks/maven 3.1.0
[detector] paketo-buildpacks/executable-jar 3.1.0
[detector] paketo-buildpacks/apache-tomcat 2.2.0
[detector] paketo-buildpacks/dist-zip 2.2.0
[detector] paketo-buildpacks/spring-boot 3.2.0
At first glance, that might seem odd. Why is Tomcat there? Looking at the Tomcat CNB, that is expected though. The Tomcat CNB is always going to return a successful detection.
Note how pass is hard coded to true.
result := libcnb.DetectResult{
Pass: true,
Plans: []libcnb.BuildPlan{
{
Requires: []libcnb.BuildPlanRequire{
{Name: "jre", Metadata: map[string]interface{}{"launch": true}},
{Name: "jvm-application"},
},
},
},
}
The reason this is OK is because at build time, the Tomcat CNB will immediately exit (no-op) if there is no WEB-INF directory, and for a Spring WebFlux app there won't be.
file := filepath.Join(context.Application.Path, "WEB-INF")
if _, err := os.Stat(file); err != nil && !os.IsNotExist(err) {
return libcnb.BuildResult{}, fmt.Errorf("unable to stat file %s\n%w", file, err)
} else if os.IsNotExist(err) {
return libcnb.BuildResult{}, nil
}
You can confirm this by looking at the full output of pack build and looking for the presence of Paketo Apache Tomcat Buildpack x.x.x (where x.x.x is the version number). If the Tomcat CNB were running and performing any work, you'd see that line output.
The paketo-buildpack/dist-zip CNB works the same way, it's just looking for <APPLICATION_ROOT>/*/bin/* to exist.
So in summary, the image that's generated should be specific to your application and shouldn't contain unnecessary stuff. If you're running Spring WebFlux, then you shouldn't have Tomcat installed into your image. In addition, you get all the optimizations provided by using Cloud Native buildpacks.
I'm trying to get double underscore for configurations to work in ASP.Net core 3.1 as described here: https://learn.microsoft.com/en-us/aspnet/core/fundamentals/configuration/?tabs=basicconfiguration&view=aspnetcore-3.1#environment-variables
I've written some code for configurations that does this manually
check Env Var
check AppSettings...json files
but I saw that in .net 3.1 this is supposed to be supported by default with one caveat around hierarchal settings on unix platforms. In this case you can't use this syntax
Configuration["SomeSection:SomeConfig"]
to resolve this environment variable
## Environment Var
SomeConfig:SomeConfig
Because linux doesn't support colons in Env Var names. This would however work if code was deployed to a windows machine. So instead .net seems to indicate you can use this syntax
Configuration["SomeSection__SomeConfig"]
which will resolve any of the following configurations
// AppSettings...json
{
"SomeSection" {
"SomeConfig": "some value"
}
}
or
# Environment Var on linux
SomeConfig__SomeConfig
or
// Environment Var on windows
SomeConfig:SomeConfig
Does anyone know what I'm doing wrong?
Edit:
Note that I know the answer to this questions and will happily answer it if it's is reopened.
I had misunderstood the documentation. The way it works is that if you use this code in your Startup.cs
Configuration["SomeSection:SomeConfig"]
it will correctly resolve
// AppSettings...json
{
"SomeSection" {
"SomeConfig": "some value"
}
}
from your various appsettings*.json files.
However, if you define an environment variable with the name SomeSection:SomeConfig that will be used instead of the appsettings.json value which is a great feature. This is often desirable when running the app in a docker container.
However, if are on a Linux operation system it's not possible to create an environment variable with the name SomeSection:SomeConfig. So if you are on Linux machine or in a Linux container, instead if you name your environment variable SomeSection__SomeConfig with a __ (double underscore) instead of a : then you could access like below:
Configuration["SomeSection:SomeConfig"]
Is there a way to find, via an API or CLI, the latest available version of Gradle?
I'm working on a tool to check versions of dependencies and compare them to the latest versions, and I need a way to determine what the latest version of Gradle that is available.
To be clear, I am not looking for the version of Gradle I already have. I know how to get that any number of ways. I'm just looking for some officially maintained endpoint I can call to determine the latest version available.
Gradle has an API to retrieve all sorts of information:
https://services.gradle.org/
For the current version:
GET https://services.gradle.org/versions/current
{
"version" : "6.8.1",
"buildTime" : "20210122132008+0000",
"current" : true,
"snapshot" : false,
"nightly" : false,
"releaseNightly" : false,
"activeRc" : false,
"rcFor" : "",
"milestoneFor" : "",
"broken" : false,
"downloadUrl" : "https://services.gradle.org/distributions/gradle-6.8.1-bin.zip",
"checksumUrl" : "https://services.gradle.org/distributions/gradle-6.8.1-bin.zip.sha256",
"wrapperChecksumUrl" : "https://services.gradle.org/distributions/gradle-6.8.1-wrapper.jar.sha256"
}
You can get the data using curl and then use jq to extract the version key.
Node.js has in-built JSON support so this will be even easier.
CURRENT_GRADLE_VERSION="$(curl -s https://services.gradle.org/versions/current | jq -r '.version')"
echo "${CURRENT_GRADLE_VERSION}" # prints 6.8.1
Trying to achieve this proposal to make a div size based on its background image I'm using Less 2.5.1 compiled by Web Essentials 2013 for Update 4 version 2.5.4 and some functions do not resolve:
My Less File:
.myClass{
height:image-height("myUrl.png"); //<-- Do not resolve
width:percentage(1/2); //<--Resolves
}
Resolved Css File:
.myClass{
height:image-height("myUrl.png"); //Not good
width:50%; //Good
}
I'm having this problem with image-height, image-width and image-size. Not with the rest of Misc Functions
There's no problem resolving the image by the compiler, because it's working fine when I used it like:
background-image:url("myUrl.png");
In the documentation there's a note for these methods saying:
Note: this function needs to be implemented by each environment. It is
currently only available in the node environment.
In the WebEssentials documentation it says:
Web Essentials uses the node-less compiler and it always uses the
latest version
And
NodeJS compilation - Web Essentials uses NodeJS to run the compiler.
It's the fastest and most accurate compiler for LESS available.
I'm missing something that I don't know. What does that note mean? What
should I do?
I've create a jsfiddle to share with you my goal.
I have a legacy Symfony 2.0 project (version 2.0.11 to be precise) to which I'd like to add behat tests. As it's Symfony 2.0, it uses the deps vendor system rather than composer. I'm not in a position to upgrade the Symfony version or switch to composer at the moment.
I attempted to install behat using the following deps settings:
[Mink]
target=/Behat/Mink
git=git://github.com/Behat/Mink.git
version=v1.3.3
[MinkBundle]
target=/Behat/MinkBundle
git=git://github.com/Behat/MinkBundle.git
[BehatBundle]
target=/Behat/BehatBundle
git=git://github.com/Behat/BehatBundle.git
[Gherkin]
target=/Behat/Gherkin
git=git://github.com/Behat/Gherkin.git
version=v2.1.1
[Behat]
target=/Behat/Behat
git=git://github.com/Behat/Behat.git
version=v2.3.5
[Goutte]
target=/Goutte
git=git://github.com/fabpot/Goutte.git
(Yes, I know that the BehatBundle etc are outdated, but it looked like I'd need these outdated versions given that I'm using deps and sf2.0.)
When I run vendor/Behat/Behat/bin/behat, I then get the issue described here:
PHP Warning: require_once(behat/autoload.php): failed to open stream: No such file or directory in /home/sam/wo-code/PersonaBubble/vendor/Behat/Behat/bin/behat on line 23
PHP Fatal error: require_once(): Failed opening required 'behat/autoload.php' (include_path='.:/usr/share/php:/usr/share/pear') in /home/sam/wo-code/PersonaBubble/vendor/Behat/Behat/bin/behat on line 23
I realised that behat 2.3.5 doesn't actually have an autoload.php file. I looked through Behat's tags on Github and realised that 2.1.3 was the latest version which had an autoload.php (actually autoload.php.dist, though every earlier version also had autoload.php.dist rather than autoload.php, so I assumed that this was it).
I therefore changed my behat version number in deps to v2.1.3, deleted my vendors and re-installed. The behat command had then changed, so I ran:
php vendor/Behat/Behat/bin/behat.php
I now saw this error:
PHP Fatal error: Class 'Symfony\Component\Console\Application' not found in /home/sam/wo-code/PersonaBubble/vendor/Behat/Behat/src/Behat/Behat/Console/BehatApplication.php on line 26
Does anyone know what the correct versions of behat etc are that I should use to get it working with Symfony 2.0 and deps? Or is there some other step that I'm missing.
PS I ended up running behat via PHAR (although this had other problems so I abandoned it as not worth it). However, I really want to know how to do it via standard vendors install, hence this post.
I'm not in a position to upgrade the Symfony version or switch to
composer at the moment.
I understand what you say but example below might give you a bit of hint I hope! I hope it helps a bit.
I'm sharing what I have been using for all my Symfony2 projects. Behat+Mink+Selenium
CONPOSER:
You need certain versions so that eveyone use same versions of everthing.
mySymfonyProject/composer.json:
"require": {
"behat/behat": "2.5.*#stable",
"behat/behat-bundle": "1.0.0",
"behat/symfony2-extension": "1.1.2",
"behat/mink": "1.5.0",
"behat/mink-extension": "~1.3",
"behat/mink-selenium2-driver": "1.1.1",
"behat/mink-goutte-driver": "1.0.9"
},
"config": {
"bin-dir": "bin"
},
"minimum-stability": "dev",
BEHAT
mySymfonyProject/behat.yml:
default:
context:
class: FeatureContext
extensions:
Behat\Symfony2Extension\Extension:
mink_driver: true
kernel:
env: test
debug: true
Behat\MinkExtension\Extension:
base_url: 'http://mysymfonyproject.local/app_test.php/'
javascript_session: selenium2
browser_name: firefox
goutte: ~
selenium2: ~
paths:
features: %behat.paths.base%/src
bootstrap: %behat.paths.features%/Context
SELENIUM
Download into your project. It is here, make sure you download 2.43.1 version which is in the middle of the page.
Run it: java -jar selenium-server-standalone-2.43.1.jar
CONTEXT FEATURE
mySymfonyProject/src/Site/CommonBundle/Features/Context/FeatureContext.php
<?php
namespace Site\CommonBundle\Features\Context;
use Behat\MinkExtension\Context\MinkContext;
use Behat\Symfony2Extension\Context\KernelAwareInterface;
use Symfony\Component\HttpKernel\KernelInterface;
class FeatureContext extends MinkContext implements KernelAwareInterface
{
/**
* Hold Symfony kernel object.
*
* #var object Kernel Object.
*/
protected $kernel;
/**
* Helps to use doctrine and entity manager.
*
* #param KernelInterface $kernelInterface Interface for getting Kernel.
*/
public function setKernel(KernelInterface $kernelInterface)
{
$this->kernel = $kernelInterface;
}
//And your own methods
}
TESTS
When you have feature files you can run them like this (this runs all in one go. for more info read behat doc):
bin/behat #SiteCommonBundle