Programmatically determine latest Gradle version? - api

Is there a way to find, via an API or CLI, the latest available version of Gradle?
I'm working on a tool to check versions of dependencies and compare them to the latest versions, and I need a way to determine what the latest version of Gradle that is available.
To be clear, I am not looking for the version of Gradle I already have. I know how to get that any number of ways. I'm just looking for some officially maintained endpoint I can call to determine the latest version available.

Gradle has an API to retrieve all sorts of information:
https://services.gradle.org/
For the current version:
GET https://services.gradle.org/versions/current
{
"version" : "6.8.1",
"buildTime" : "20210122132008+0000",
"current" : true,
"snapshot" : false,
"nightly" : false,
"releaseNightly" : false,
"activeRc" : false,
"rcFor" : "",
"milestoneFor" : "",
"broken" : false,
"downloadUrl" : "https://services.gradle.org/distributions/gradle-6.8.1-bin.zip",
"checksumUrl" : "https://services.gradle.org/distributions/gradle-6.8.1-bin.zip.sha256",
"wrapperChecksumUrl" : "https://services.gradle.org/distributions/gradle-6.8.1-wrapper.jar.sha256"
}
You can get the data using curl and then use jq to extract the version key.
Node.js has in-built JSON support so this will be even easier.
CURRENT_GRADLE_VERSION="$(curl -s https://services.gradle.org/versions/current | jq -r '.version')"
echo "${CURRENT_GRADLE_VERSION}" # prints 6.8.1

Related

GCSToBigQueryOperator not working in composer-2.1.0-airflow-2.3.4

After a recent upgrade to composer-2.1.0-airflow-2.3.4 the GCSToBigQueryOperator is no longer able to find data in buckets to upload to BigQuery.
All other aspects of the DAGs still work.
The usage is as follows
gcs_to_bq = GCSToBigQueryOperator(
task_id = f"transfer_{data_type}_to_bq_task",
bucket = os.environ["GCS_BUCKET"],
source_objects = file_names,
destination_project_dataset_table = os.environ["GCP_PROJECT"] + f".creditsafe.{data_type}",
schema_object = f"dags/schema/creditsafe/{data_type}.json",
source_format = "CSV",
field_delimiter = '|',
quote_character = "",
max_bad_records = 0,
create_disposition = "CREATE_IF_NEEDED",
ignore_unknown_values = True,
allow_quoted_newlines = True,
allow_jagged_rows = True,
write_disposition = "WRITE_TRUNCATE",
gcp_conn_id = 'google_cloud_default',
skip_leading_rows = 1,
dag = dag
)
The error from the API is
google.api_core.exceptions.NotFound: 404 GET
{ "error": { "code": 400, "message": "Unknown output format: media:", "errors": [ { "message": "Unknown output format: media:", "domain": "global", "reason": "invalidAltValue", "locationType": "parameter", "location": "alt" } ] } }
The error delivered by Cloud Composer is
google.api_core.exceptions.NotFound: 404 GET https://storage.googleapis.com/download/storage/v1/b/[BUCKET_HIDDEN]/o/data%2Fcreditsafe%2FCD01%2Ftxt%2F%2A.txt?alt=media: No such object: [BUCKET_HIDDEN]/data/creditsafe/CD01/txt/*.txt: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)
I can't see the cause of the error. The reference to the GCS location has not changed and appears correct while the gcp_conn_id appears sufficient for all other tasks. I'm at a loss.
A fix for this above issue has now been made
https://github.com/apache/airflow/pull/28444
It is unclear how long it will take for this to be integrated into the Cloud Composer libraries.
GCSToBigQueryOperator does not support wildcard *.csv. For your requirement, you can try the below steps:
You can attach to a pod in the composer environment by running the below commands :
gcloud container clusters get-credentials --region __GCP_REGION__ __GKE_CLUSTER_NAME__
kubectl get pods -n [Namespace]
kubectl exec -it [Worker] -n [Namespace] -- bash
You can run the below command to identify the google provider package,
pip list | grep -i goo | grep provider
If the output of the above command is a different version from 8.3.0 then change the version to apache-airflow-providers-google ==8.3.0.
The release 8.5.0 of apache-airflow-providers-google that cames with airflow (>2.3.4 and <2.5.1) introduced several critical regressions nottably:
GCSToBigquery operator is broken as it ignores its options https://github.com/apache/airflow/pull/27961
This means that all custom settings specified in the operator (delimiter, formatting, null values, wildcards on files) are no longer sent to BigQuery leading to unexpected results.
Until Google releases a composer version based on airflow 2.5.1 the workaround is to upgrade the apache-airflow-providers-google library (or to use a composer version based on apache <=2.2.5).
No need to connect via gcloud/kubectl to change the apache-airflow-providers-google version, you can change it directly in the Composer UI, via PyPi Packages page (or via the terraform provider).
I can confirm that on the latest (today) composer : composer-1.20.4-airflow-2.4.3 configuring the apache-airflow-providers-google ==8.8.0 (latest) solves those issues for me.
But as mentioned previously this is only an workaround and your mileage might vary...
configuring custom PyPI packages

Does zef recognize and handle pre-release modules?

I have a module App::Assixt, to which I've recently made a relatively large change. I've of course tested it on my local system, but would like to give it some field testing before calling it a "stable release".
Is there a way to "tag" this release as an "alpha", or "release-candidate", so this release will only be installed by people who have opted to use unstable/testing releases?
zef will treat versions the same as Perl 6:
# true because "1.0.a" < "1.0.0"
$ perl6 -e 'say Version.new("1.0.PREVIEW") < Version.new("1.0")'
True
If a Foo:ver<1.0.PREVIEW> is installed it can be used as Foo:ver<1.0.PREVIEW> or Foo:ver<1.0>. This means any systems that have Foo:ver<1.0.PREVIEW> installed would need to uninstall it to upgrade / install Foo:ver<1.0> in the future ( unless using --force-install ), but also that authors can write code for the final version without declaring the extra .PREVIEW everywhere.
This is not very useful in regards to publishing -- zef will grab the newest version by default despite the user not having opted in to whatever versioning scheme is in use. Since the user wants to opt-in for this, there are two options.
1) Create e.g. unstable, testing, stable` indexes and convince people to use them
This could be done by copying the zef config to %*ENV<XDG_CONFIG_HOME>/zef/config.json and incorporate the following:
"Repository" : [
{
"short-name" : "unstable",
"enabled" : 0,
"module" : "Zef::Repository::Ecosystems",
"options" : {
"name" : "unstable",
"mirrors" : [ "/path/or/url/to/package/list.json" ]
}
}
]
Which allows:
zef install Foo::Bar --unstable
2) Give out a link to the resource without publishing it
zef install https://github.com/ugexe/Perl6-Text--Table--Simple.git#v0.0.4
zef install https://github.com/ugexe/Perl6-Text--Table--Simple/archive/v0.0.3.zip

How to configure applications in ami 4

The documentation says
In Amazon EMR releases 4.0 and greater, the only accepted parameter is
the application name. To pass arguments to applications, you supply a
configuration for each application.
But I cannot find an example that shows how to pass arguments in ami 4. All I can find are examples configuring exports such as below. I am trying to figure out how to set the version of Spark to use.
[
{
"Classification":"hadoop-env",
"Properties":{
},
"Configurations":[
{
"Classification":"export",
"Properties":{
"HADOOP_USER_CLASSPATH_FIRST":"true",
"HADOOP_CLASSPATH":"/path/to/my.jar"
}
}
]
}
]
You cannot set an arbitrary version of Spark to use like you could with 3.x AMI versions. Rather, the version of Spark (and other apps, of course) is determined by the release label. For example, the latest release is currently emr-5.2.1, which includes Spark 2.0.2. If you want a 1.x version of Spark, the latest version available is Spark 1.6.3 on release emr-4.8.3.

Installing behat on a Symfony 2.0 project using deps - what versions?

I have a legacy Symfony 2.0 project (version 2.0.11 to be precise) to which I'd like to add behat tests. As it's Symfony 2.0, it uses the deps vendor system rather than composer. I'm not in a position to upgrade the Symfony version or switch to composer at the moment.
I attempted to install behat using the following deps settings:
[Mink]
target=/Behat/Mink
git=git://github.com/Behat/Mink.git
version=v1.3.3
[MinkBundle]
target=/Behat/MinkBundle
git=git://github.com/Behat/MinkBundle.git
[BehatBundle]
target=/Behat/BehatBundle
git=git://github.com/Behat/BehatBundle.git
[Gherkin]
target=/Behat/Gherkin
git=git://github.com/Behat/Gherkin.git
version=v2.1.1
[Behat]
target=/Behat/Behat
git=git://github.com/Behat/Behat.git
version=v2.3.5
[Goutte]
target=/Goutte
git=git://github.com/fabpot/Goutte.git
(Yes, I know that the BehatBundle etc are outdated, but it looked like I'd need these outdated versions given that I'm using deps and sf2.0.)
When I run vendor/Behat/Behat/bin/behat, I then get the issue described here:
PHP Warning: require_once(behat/autoload.php): failed to open stream: No such file or directory in /home/sam/wo-code/PersonaBubble/vendor/Behat/Behat/bin/behat on line 23
PHP Fatal error: require_once(): Failed opening required 'behat/autoload.php' (include_path='.:/usr/share/php:/usr/share/pear') in /home/sam/wo-code/PersonaBubble/vendor/Behat/Behat/bin/behat on line 23
I realised that behat 2.3.5 doesn't actually have an autoload.php file. I looked through Behat's tags on Github and realised that 2.1.3 was the latest version which had an autoload.php (actually autoload.php.dist, though every earlier version also had autoload.php.dist rather than autoload.php, so I assumed that this was it).
I therefore changed my behat version number in deps to v2.1.3, deleted my vendors and re-installed. The behat command had then changed, so I ran:
php vendor/Behat/Behat/bin/behat.php
I now saw this error:
PHP Fatal error: Class 'Symfony\Component\Console\Application' not found in /home/sam/wo-code/PersonaBubble/vendor/Behat/Behat/src/Behat/Behat/Console/BehatApplication.php on line 26
Does anyone know what the correct versions of behat etc are that I should use to get it working with Symfony 2.0 and deps? Or is there some other step that I'm missing.
PS I ended up running behat via PHAR (although this had other problems so I abandoned it as not worth it). However, I really want to know how to do it via standard vendors install, hence this post.
I'm not in a position to upgrade the Symfony version or switch to
composer at the moment.
I understand what you say but example below might give you a bit of hint I hope! I hope it helps a bit.
I'm sharing what I have been using for all my Symfony2 projects. Behat+Mink+Selenium
CONPOSER:
You need certain versions so that eveyone use same versions of everthing.
mySymfonyProject/composer.json:
"require": {
"behat/behat": "2.5.*#stable",
"behat/behat-bundle": "1.0.0",
"behat/symfony2-extension": "1.1.2",
"behat/mink": "1.5.0",
"behat/mink-extension": "~1.3",
"behat/mink-selenium2-driver": "1.1.1",
"behat/mink-goutte-driver": "1.0.9"
},
"config": {
"bin-dir": "bin"
},
"minimum-stability": "dev",
BEHAT
mySymfonyProject/behat.yml:
default:
context:
class: FeatureContext
extensions:
Behat\Symfony2Extension\Extension:
mink_driver: true
kernel:
env: test
debug: true
Behat\MinkExtension\Extension:
base_url: 'http://mysymfonyproject.local/app_test.php/'
javascript_session: selenium2
browser_name: firefox
goutte: ~
selenium2: ~
paths:
features: %behat.paths.base%/src
bootstrap: %behat.paths.features%/Context
SELENIUM
Download into your project. It is here, make sure you download 2.43.1 version which is in the middle of the page.
Run it: java -jar selenium-server-standalone-2.43.1.jar
CONTEXT FEATURE
mySymfonyProject/src/Site/CommonBundle/Features/Context/FeatureContext.php
<?php
namespace Site\CommonBundle\Features\Context;
use Behat\MinkExtension\Context\MinkContext;
use Behat\Symfony2Extension\Context\KernelAwareInterface;
use Symfony\Component\HttpKernel\KernelInterface;
class FeatureContext extends MinkContext implements KernelAwareInterface
{
/**
* Hold Symfony kernel object.
*
* #var object Kernel Object.
*/
protected $kernel;
/**
* Helps to use doctrine and entity manager.
*
* #param KernelInterface $kernelInterface Interface for getting Kernel.
*/
public function setKernel(KernelInterface $kernelInterface)
{
$this->kernel = $kernelInterface;
}
//And your own methods
}
TESTS
When you have feature files you can run them like this (this runs all in one go. for more info read behat doc):
bin/behat #SiteCommonBundle

Behat 2.4 upgrade issues, or Unrecognized options "extensions" under "behat" - error when attempting to activate sahi extension via behat.yml

Fairly new to this Behat stuff, and I've run into a roadblock I can't seem to get around. I've been staring at the docs, googling like it's my job, and doing my best to refrain from tossing my computer off the fire escape.
I'm working with a fairly complex project, and I'm not the one who set it up. So I'm a little lost in some areas.
Currently, I'm trying to use the Sahi driver, because selenium isn't cutting it for some dynamic forms I need to test. I can run the tests fine with the default selenium driver, but the tests fail because it doesn't adequately trigger JavaScript events on form input. Specifically, it'll work with the workarounds covered in that link, but only if I have the browser in focus. Which means it fails when the tests are run in sauce or via jenkins with xvfb.
I'm explaining all this only because this is my larger issue, which I'm attempting to address by using the Sahi driver. Which brings me to:
[Symfony\Component\Config\Definition\Exception\InvalidConfigurationException]
Unrecognized options "extensions" under "behat"
That's what I get when I try to activate the Sahi driver for a particular profile in my bahat.yml the way the documentation says to.
Here's the default profile and the profile I'm currently working with in my behat.yml (slightly modifies for public consumption):
default:
paths:
features: 'features'
bootstrap: '%behat.paths.features%/bootstrap'
sahi:
extensions:
Behat\MinkExtension\Extension:
sahi: ~
context:
class: 'FeatureContext'
parameters:
environment: 'staging'
mink: 'sahi'
Fwiw, the tests are on a vm, which I ssh -X into, then run the test using
$ behat --tags #test_name_tag --profile=sahi
When I'm using the default selenium driver and the #javascript tag, the browser pops up and the tests run and pass (assuming I keep the browser in focus, of course).
I installed the additional drivers using composer:
{
"require": {
"behat/behat": "2.4.*#stable",
"behat/mink": "1.4#stable",
"behat/mink-extension": "*",
"behat/mink-selenium2-driver": "*",
"behat/mink-sahi-driver": "*"
}
}
I've added use Behat\Mink\Driver\SahiDriver; to my MinkContext.php, EnvironmentContext.php and FeatureContext.php, though I'm guessing that's probably either overkill or not necessary. It doesn't seem to be making a difference at this point, though. I get the same error with or without it.
I also added a sahi.php which lives in features/bootstrap/mink:
<?php
return
array(
'default_session' => 'sahi',
'sahi' => array(
'capabilities' => array(
'browserName' => 'firefox',
'browserVersion' => 7,
),
),
);
I thought maybe adding a directory in features/bootstrap called exensions might help for some reason. Even stuck a file in there called sahi.php. That didn't help much.
I think that covers everything. Thanks in advance for any help, and if this is covered elsewhere, please direct me to it, because I've spent countless hours looking and haven't found anything that helps me.
Update:
I uninstalled the old versions of behat, mink and gherkin, and installed 2.4, et al as per this https://lestbddphp.wordpress.com/2012/08/31/behatcomposer/
I've been making my way through "Migrating from Behat 2.3 to 2.4" in the docs. (Sorry, SO won't let me post any more links, but it's in the official Behat docs.)
My composer.json:
{
"require": {
"behat/behat": "2.4.*#stable",
"behat/mink": "1.4#stable",
"behat/mink-goutte-driver": "*",
"behat/symfony2-extension": "*",
"symfony/class-loader": "2.1.*",
"symfony/form": "2.1.*",
"symfony/validator": "2.1.*",
"behat/mink-selenium-driver": "*",
"behat/mink-selenium2-driver": "*",
"behat/mink-extension": "*",
"behat/mink-sahi-driver": "*"
},
"minimum-stability": "dev",
"config": {
"bin-dir": "bin/"
}
}
I moved my behat.yml file to the root of the project, as directed. I updated my default profile to:
default:
paths:
features: 'features'
bootstrap: '%behat.paths.features%/bootstrap'
extensions:
Behat\Symfony2Extension\Extension:
mink_driver: true
kernel:
env: test
debug: true
Behat\MinkExtension\Extension:
default_session: symfony2
sahi: ~
though I'm not entirely sure that's what I need. Just going by the example given in the docs.
I updated my vendor/autoload.php by replacing the require_once with require:
<?php
// autoload.php generated by Composer
require __DIR__ . '/composer' . '/autoload_real.php';
return ComposerAutoloaderInit::getLoader();
but I'm a little confused by this, because that file is different from the example code in the docs. If I were to add the line in the docs here, instead of what was already there, then it would just be loading itself. (I tried. It barfed.) Am I completely dense, or is the wording here confusing/misleading? Did I do this correctly?
As I mentioned before, I have 3 context files in features/bootstrap:
FeatureContext.php
EnvironmentContext.php
MinkContext.php
When running the tests via cli, I pass it a --profile, and then it uses the appropriate profile in behat.yml. In almost all of the profiles, FeatureContext is used.
context:
class: 'FeatureContext'
FeatureContext then gets EnvironmentContext and MinkContext, from what I can tell. So, theoretically, everything should be working there.
Only it's not.
$ bin/behat --profile=sahi
[ReflectionException]
Class AppKernel does not exist
Before I added all the Symfony stuff, I was getting this:
Warning: require(Behat\Symfony2Extension\Extension): failed to open stream: No such file or directory in /path/to/project/vendor/behat/behat/src/Behat/Behat/Extension/ExtensionManager.php on line 112
Fatal error: require(): Failed opening required 'Behat\Symfony2Extension\Extension' (include_path='/usr/share/pear:/usr/share/php:/usr/share/git core/templates/hooks:.') in /path/to/project/vendor/behat/behat/src/Behat/Behat/Extension/ExtensionManager.php on line 112
Which is why I added the Symfony stuff via composer.
Also possibly of note: when I forgot to pass it a --profile, before installing the Symfony stuff via composer, I got this:
Notice: Undefined index: environment in /home/lbaron/development/BeHat-Functional/features/bootstrap/FeatureContext.php on line 43
Warning: include(/path/to/project/features/bootstrap/environment/.php): failed to open stream: No such file or directory in /path/to/project/features/bootstrap/FeatureContext.php on line 44
Warning: include(): Failed opening '/path/to/project/features/bootstrap/environment/.php' for inclusion (include_path='/usr/share/pear:/usr/share/php:/usr/share/git-core/templates/hooks:.') in /path/to/project/features/bootstrap/FeatureContext.php on line 44
Catchable fatal error: Argument 1 passed to EnvironmentContext::__construct() must be an array, boolean given, called in /path/to/project/features/bootstrap/FeatureContext.php on line 44 and defined in /path/to/project/features/bootstrap/EnvironmentContext.php on line 27
Which I guess is to be expected.
So I'm at a loss now. Ideas?
I'm going to keep banging on it to see if I can figure it out, but any ideas/input would be greatly appreciated.
Update again:
Removing the extensions section from yml gives me this:
Catchable fatal error: Argument 2 passed to Symfony\Component\BrowserKit\Client::__construct() must be an instance of Symfony\Component\BrowserKit\History, array given, called in /usr/share/pear/mink/src/Behat/Mink/Behat/Context/MinkContext.php on line 163 and defined in /home/lbaron/development/BeHat-Functional/vendor/symfony/browser-kit/Symfony/Component/BrowserKit/Client.php on line 52
Current state of behat.yml:
default:
paths:
features: 'features'
bootstrap: '%behat.paths.features%/bootstrap'
formatter:
parameters:
language: 'en'
extensions:
Behat\MinkExtension\Extension:
sahi: ~
goutte: ~
You are running a version of behat which is older that 2.4 (the current version). I can tell because the command you use is "behat" instead of "bin/behat". Older versions had a different architecture and did not use extensions. The documentation on the behat.org website is all for the new 2.4 version and, as far as I know, does not have the documentation for older versions available anymore. You should upgrade your behat version to 2.4, there is a guide on how to do this here