Attaching attachments in Allure reports using cucumber jvm after hooks - cucumber-jvm

With Allure report framework when a step fails we can attach a screenshot or logs by calling a method with #Attachment annotation.
#Attachment(value = "Message", type = "text/plain")
public String attachLog(){
return "Hello, Test failed!";
}
But this means I have to explicitly call this method ( attachLog() ) in every step before assertions. Which seems unreasonable.
In CucumberJvm the "after" hooks are a great way to attach screenshots or logs. we do this by checking the scenario status and attach screenshots/logs based on the outcome.
I tried invoking the above method ( attachLog() ) in the cucumberJvm "after" hook. But unfortunately did not work.
Is there a solution to make this work?
Cheers
Vinod

You can override test failure method from ru.yandex.qatools.allure.cucumberjvm.AllureRunListener
public class CustomAllureListener extends AllureRunListener {
#Override
public void testFailure(Failure failure) {
super.testFailure(failure);
if (!failure.getDescription().isSuite()) { // check is needed to avoid double attaching
attachFailed();
}
}
#Attachment(value = "Message", type = "text/plain")
public String attachFailed(){
return "Test failed!";
}
}
And don't forget to change listener in pom.xml file
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.18.1</version>
<configuration>
<testFailureIgnore>false</testFailureIgnore>
<argLine>
-javaagent:${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar
</argLine>
<properties>
<property>
<name>listener</name>
<value>com.mycompany.testing.CustomAllureListener</value>
</property>
</properties>
</configuration>
<dependencies>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjweaver</artifactId>
<version>${aspectj.version}</version>
</dependency>
</dependencies>
</plugin>

Related

Bean Validation on Jax-RS Resource stops working while using CDI on Apache TomEE 8.0.10

I'm having troubles getting bean validation to work with the following minimalised project consisting only of this three java files plus pom.xml. I'm using Apache TomEE 8.0.10.
LoginMessage.java
package org.example;
import lombok.Getter;
import lombok.Setter;
import lombok.ToString;
import javax.validation.constraints.NotBlank;
#Getter
#Setter
#ToString
public class LoginMessage {
#NotBlank
private String username;
#NotBlank
private String password;
}
SessionService.java
package org.example;
import lombok.extern.java.Log;
import javax.enterprise.context.RequestScoped;
#Log
#RequestScoped
public class SessionService {
public void login(final LoginMessage loginMessage) {
log.info(loginMessage.toString());
}
}
SessionController.java
package org.example;
import lombok.extern.java.Log;
import javax.inject.Inject;
import javax.validation.Valid;
import javax.ws.rs.Consumes;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.core.MediaType;
#Log
#Path("/session")
public class SessionController {
#Inject
private SessionService sessionService;
#POST
#Consumes(MediaType.APPLICATION_JSON)
public void postLoginMessage(#Valid final LoginMessage loginMessage) {
sessionService.login(loginMessage);
}
}
pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<packaging>war</packaging>
<groupId>org.example</groupId>
<artifactId>beanval</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
<configuration>
<source>11</source>
<target>11</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<version>3.3.2</version>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>javax</groupId>
<artifactId>javaee-api</artifactId>
<version>8.0.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.22</version>
</dependency>
</dependencies>
</project>
If you post an empty JSON object it ignores the #Valid annotation in SessionController#postLoginMessage() and directly outputs the log message containing the toString() content of the LoginMessage object through SessionService#login() method.
POST http://localhost:8080/beanval-1.0-SNAPSHOT/session
Content-Type: application/json
{
}
13-Mar-2022 01:30:39.700 INFORMATION [http-nio-8080-exec-6] SessionService.login LoginMessage(username=null, password=null)
If you remove or comment out the #RequestScoped annotation from SessionService and post the empty JSON-Object after restart of TomEE then bean validation works and logs:
13-Mar-2022 01:52:51.594 WARNUNG [http-nio-8080-exec-6] org.apache.cxf.jaxrs.validation.ValidationExceptionMapper.toResponse Value (null) of SessionController.postLoginMessage.arg0.password: must not be blank
13-Mar-2022 01:52:51.595 WARNUNG [http-nio-8080-exec-6] org.apache.cxf.jaxrs.validation.ValidationExceptionMapper.toResponse Value (null) of SessionController.postLoginMessage.arg0.username: must not be blank
I would like to use CDI in combination with Bean-Validation in JAX-RS Resource.
What am I doing wrong?
Thanks in advance.
This appears to be a bug in OpenWebBeans or TomEE. So what's happening is the first the actual instance of the bean is managed by JAX-RS, and the second, the bean is managed by the CDI container. In the second case, there needs to be some sort of interceptor the invokes the Bean Validation framework.
I would start a discussion on the mailing list and open a bug on in the JIRA. If you can create a sample project that reproduces the problem it helps the devs out tremendously.
As a workaround, you can #Inject private Validator validator and if there are any constraint violations returned, throw new ConstraintViolationException(constraintViolations);.
After all to come up to some kind of a solution I will stop using bean validation at controller layer. It works at service layer and so I can continue to work on my web app.
The solution is using the #Valid annotation in SessionService#login() method and remove it from SessionController#postLoginMessage() method.
If it is really a bug in TomEE the alternative could also be to use another application server until it is fixed.

Unable to use aspectJ interceptor in a non-spring project

I am trying to use Aspectj to execute some code after some method execution. I cannot use spring AOP as the project is a non-spring project and at this point of time I cannot change it to spring project. I have tried with a very simple implementation as below but it is not at all working:
POM of my project
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>test.aspectj</groupId>
<artifactId>HelloAspectj</artifactId>
<version>0.0.1-SNAPSHOT</version>
<properties>
<maven.compiler.plugin.version>3.5.1</maven.compiler.plugin.version>
</properties>
<dependencies>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjrt</artifactId>
<version>1.9.6</version>
</dependency>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjweaver</artifactId>
<version>1.9.6</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>${maven.compiler.plugin.version}</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
</build>
</project>
A normal class and method after which the aspect methods will run:
package tester;
public class HelloWorld {
private String name;
public void setName(String name) {
this.name = name;
}
public void printHello() {
System.out.println("Spring 3 : Hello ! " + name);
}
}
Aspect class
package tester;
import org.aspectj.lang.annotation.After;
import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.annotation.Before;
#Aspect
public class TestAspect {
#Before(" call(void java.io.PrintStream.println(String)) " +
"&& !within(net.andrewewhite.aspects..*)")
public void beforePrintlnCall() {
System.out.println("About to make call to print Hello World");
}
#After(" call(void java.io.PrintStream.println(String)) " +
"&& !within(net.andrewewhite.aspects..*)")
public void afterPrintlnCall() {
System.out.println("Just made call to print Hello World");
}
}
Main class
package tester;
public class MainApp {
public static void main(String[] args) {
// TODO Auto-generated method stub
System.out.println("Hello World");
}
}
aop.xml
<aspectj>
<aspects>
<aspect name="tester.TestAspect"/>
</aspects>
</aspectj>
Project Structure:
Now i am expecting that it will price About to make call to print Hello World & Just made call to print Hello World BUt it is only printing Hello World
can someone help here..
If you want to use compile-time weaving, use
Mojohaus AspectJ Maven plugin until Java 8 and AspectJ 1.8.x or
Nick Wongdev's AspectJ Maven fork for Java 9+.
Javac via Maven Compiler plugin is not enough.
If you wish to use load-time weaving (LTW), it should be okay to compile your aspects with Javac via Maven Compiler plugin, as long as you only use annotation-driven #AspectJ syntax. For native AspectJ syntax you always need the AspectJ compiler Ajc via AspectJ Maven plugin, no matter what. For LTW you also need the weaving agent on the Java command line via -javaagent:/path/to/aspectjweaver.jar. There is also a hot-attachment option, but that is advanced stuff and you need to know what you are doing and the application must know that it wants to attach the weaver, so let's not talk about this here, as you are clearly a beginner.
All of this has been documented on the AspectJ website and the AspectJ Maven website. I have also answered numerous questions about AspectJ + Maven here, you should easily find some. Before asking questions, you should really search first. This website does not replace manuals and tutorials.

Cucumber: undefined scenario/steps when running a bunch of test with tags (annotations) using POM.xml

I am facing a problem when executing cucumber scripts having tags using POM.XML. I wish to run it without a runner class.
When I use the running class, the scripts run well and I get the below output.
/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/bin/java -ea -Didea.test.cyclic.buffer.size=1048576 "-javaagent:/Applications/IntelliJ IDEA CE.app/Contents/lib/idea_rt.jar=51160:/Applications/IntelliJ IDEA CE.app/Contents/bin" -Dfile.encoding=UTF-8 -classpath "/Applications/IntelliJ IDEA CE.app/Contents/lib/idea_rt.jar:/Applications/IntelliJ IDEA CE.app/Contents/plugins/junit/lib/junit-rt.jar:/Applications/IntelliJ IDEA CE.app/Contents/plugins/junit/lib/junit5-rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/ext/jaccess.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/ext/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/ext/nashorn.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/jfxswt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/lib/packager.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/lib/tools.jar:/Users/umeshanand/Documents/Selenium/Frameworks/FW.BDD/target/test-classes:/Users/umeshanand/.m2/repository/info/cukes/cucumber-java/1.2.2/cucumber-java-1.2.2.jar:/Users/umeshanand/.m2/repository/org/seleniumhq/selenium/selenium-java/3.7.1/selenium-java-3.7.1.jar:/Users/umeshanand/.m2/repository/org/seleniumhq/selenium/selenium-api/3.7.1/selenium-api-3.7.1.jar:/Users/umeshanand/.m2/repository/org/seleniumhq/selenium/selenium-chrome-driver/3.7.1/selenium-chrome-driver-3.7.1.jar:/Users/umeshanand/.m2/repository/org/seleniumhq/selenium/selenium-edge-driver/3.7.1/selenium-edge-driver-3.7.1.jar:/Users/umeshanand/.m2/repository/org/seleniumhq/selenium/selenium-firefox-driver/3.7.1/selenium-firefox-driver-3.7.1.jar:/Users/umeshanand/.m2/repository/org/seleniumhq/selenium/selenium-ie-driver/3.7.1/selenium-ie-driver-3.7.1.jar:/Users/umeshanand/.m2/repository/org/seleniumhq/selenium/selenium-opera-driver/3.7.1/selenium-opera-driver-3.7.1.jar:/Users/umeshanand/.m2/repository/org/seleniumhq/selenium/selenium-remote-driver/3.7.1/selenium-remote-driver-3.7.1.jar:/Users/umeshanand/.m2/repository/org/seleniumhq/selenium/selenium-safari-driver/3.7.1/selenium-safari-driver-3.7.1.jar:/Users/umeshanand/.m2/repository/org/seleniumhq/selenium/selenium-support/3.7.1/selenium-support-3.7.1.jar:/Users/umeshanand/.m2/repository/net/bytebuddy/byte-buddy/1.7.5/byte-buddy-1.7.5.jar:/Users/umeshanand/.m2/repository/org/apache/commons/commons-exec/1.3/commons-exec-1.3.jar:/Users/umeshanand/.m2/repository/commons-codec/commons-codec/1.10/commons-codec-1.10.jar:/Users/umeshanand/.m2/repository/commons-logging/commons-logging/1.2/commons-logging-1.2.jar:/Users/umeshanand/.m2/repository/com/google/code/gson/gson/2.8.2/gson-2.8.2.jar:/Users/umeshanand/.m2/repository/com/google/guava/guava/23.0/guava-23.0.jar:/Users/umeshanand/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/Users/umeshanand/.m2/repository/com/google/errorprone/error_prone_annotations/2.0.18/error_prone_annotations-2.0.18.jar:/Users/umeshanand/.m2/repository/com/google/j2objc/j2objc-annotations/1.1/j2objc-annotations-1.1.jar:/Users/umeshanand/.m2/repository/org/codehaus/mojo/animal-sniffer-annotations/1.14/animal-sniffer-annotations-1.14.jar:/Users/umeshanand/.m2/repository/org/apache/httpcomponents/httpclient/4.5.3/httpclient-4.5.3.jar:/Users/umeshanand/.m2/repository/org/apache/httpcomponents/httpcore/4.4.6/httpcore-4.4.6.jar:/Users/umeshanand/.m2/repository/junit/junit/4.12/junit-4.12.jar:/Users/umeshanand/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar:/Users/umeshanand/.m2/repository/info/cukes/cucumber-core/1.2.2/cucumber-core-1.2.2.jar:/Users/umeshanand/.m2/repository/info/cukes/cucumber-html/0.2.3/cucumber-html-0.2.3.jar:/Users/umeshanand/.m2/repository/info/cukes/cucumber-jvm-deps/1.0.3/cucumber-jvm-deps-1.0.3.jar:/Users/umeshanand/.m2/repository/info/cukes/gherkin/2.12.2/gherkin-2.12.2.jar:/Users/umeshanand/.m2/repository/info/cukes/cucumber-junit/1.2.2/cucumber-junit-1.2.2.jar:test/java/.:src/test/java/stepDefs" com.intellij.rt.execution.junit.JUnitStarter -ideVersion5 -junit4 runner.runnertest
objc[976]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/bin/java (0x102c0d4c0) and /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/libinstrument.dylib (0x10352b4e0). One of the two will be used. Which one is undefined.
Launch browser
open google
enter powercor
validate results
Launch browser
open google
enter powercor
validate results
Launch browser
open google
enter powercor
validate results
3 Scenarios (3 passed)
12 Steps (12 passed)
0m0.125s
When I execute using a POM.XML, I get the below message (please note I am trying to execute one of the three tags #google1, #google2, #google3)
--- exec-maven-plugin:1.2.1:java (default) # FW_BDD ---
Feature: First test to launch google
#google1
Scenario: Launch Google and search something # features/googlenew.feature:4
Given I open Chrome browser
And user open google
When user enter powercor in textbox
Then search results should should poercor.com.au
1 Scenarios (1 undefined)
4 Steps (4 undefined)
0m0.000s
You can implement missing steps with the snippets below:
#Given("^I open Chrome browser$")
public void i_open_Chrome_browser() throws Throwable {
// Write code here that turns the phrase above into concrete actions
throw new PendingException();
}
#Given("^user open google$")
public void user_open_google() throws Throwable {
// Write code here that turns the phrase above into concrete actions
throw new PendingException();
}
#When("^user enter powercor in textbox$")
public void user_enter_powercor_in_textbox() throws Throwable {
// Write code here that turns the phrase above into concrete actions
throw new PendingException();
}
#Then("^search results should should poercor\\.com\\.au$")
public void search_results_should_should_poercor_com_au() throws Throwable {
// Write code here that turns the phrase above into concrete actions
throw new PendingException();
}
Process finished with exit code 1
Execution happens through below :
clean install -DtagArg=#google1
POM.XML is:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>FW_BDD</groupId>
<artifactId>FW_BDD</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<tagArg>~#ignore</tagArg>
<format>pretty</format>
</properties>
<dependencies>
<dependency>
<groupId>info.cukes</groupId>
<artifactId>cucumber-java</artifactId>
<version>1.2.2</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.seleniumhq.selenium/selenium-java -->
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>3.7.1</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
</dependency>
<dependency>
<groupId>info.cukes</groupId>
<artifactId>cucumber-core</artifactId>
<version>1.2.2</version>
</dependency>
<dependency>
<groupId>info.cukes</groupId>
<artifactId>cucumber-junit</artifactId>
<version>1.2.2</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.7.0</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.20.1</version>
<configuration>
<additionalClasspathElements>
<additionalClasspathElement>test/java/.</additionalClasspathElement>
<additionalClasspathElement>src/test/java/stepDefs</additionalClasspathElement>
<additionalClasspathElement>test/java/.</additionalClasspathElement>
</additionalClasspathElements>
<skipTests>true</skipTests>
<!--<includes>
<include>**/*test.java</include>
</includes>-->
</configuration>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.2.1</version>
<executions>
<execution>
<phase>test</phase>
<goals>
<goal>java</goal>
</goals>
<configuration>
<classpathScope>test</classpathScope>
<mainClass>cucumber.api.cli.Main</mainClass>
<arguments>
<argument>--plugin</argument>
<argument>junit:reports/junit.xml</argument>
<argument>--plugin</argument>
<argument>json:reports/json.json</argument>
<argument>--plugin</argument>
<argument>html:reports/</argument>
<argument>--plugin</argument>
<argument>pretty</argument>
<argument>--strict</argument>
<argument>--glue</argument>
<argument>target/test-classes</argument>
<argument>target/test-classes/.</argument>
<argument>--tags</argument>
<argument>~#ignore</argument>
<argument>--tags</argument>
<argument>${tagArg}</argument>
</arguments>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
I tried using groovy over Java and things work very well. I use IntelliJ and am not sure what I am missing.
The "--glue" argument is incorrect. This should be the full package name for the step definition classes instead of "target/test-classes". For example, if those classes are located in the "com.cucumber.steps" package, then the POM should look like:
<argument>--glue</argument>
<argument>com.cucumber.steps</argument>
...
<argument>src/test/resources/features</argument>

Read data from Redis to Flink

I have been trying to find a connector to read data from Redis to Flink. Flink's documentation contains the description for a connector to write to Redis. I need to read data from Redis in my Flink job. In Using Apache Flink for data streaming, Fabian has mentioned that it is possible to read data from Redis. What is the connector that can be used for the purpose?
We are running one in production that looks roughly like this
class RedisSource extends RichSourceFunction[SomeDataType] {
var client: RedisClient = _
override def open(parameters: Configuration) = {
client = RedisClient() // init connection etc
}
#volatile var isRunning = true
override def cancel(): Unit = {
isRunning = false
client.close()
}
override def run(ctx: SourceContext[SomeDataType]): Unit = while (isRunning) {
for {
data <- ??? // get some data from the redis client
} yield ctx.collect(SomeDataType(data))
}
}
I think it really depends on what you need to fetch from redis. The above could be used to fetch a message from a list/queue, transform/push and then delete it form the queue.
Redis also supports Pub/Sub, so it would possible to subscribe, grab the SourceConext and push messages downstream.
Currently, Flink Redis Connector is not available but it can be implemented by extending RichSinkFunction/SinkFunction class.
public class RedisSink extends RichSinkFunction<String> {
#Override
public void open(Configuration parameters) throws Exception {
//open redis connection
}
#Override
public void invoke(String map) throws Exception {
//sink data to redis
}
#Override
public void close() throws Exception {
super.close();
}
}
There's been a bit of discussion about having a streaming redis source connector for Apache Flink (see FLINK-3033), but there isn't one available. It shouldn't be difficult to implement one, however.
One of challenges in getting your Flink program to use Jedis to talk to Redis is getting the appropriate libraries into the JAR file you submit to Flink. Absent this, you will get call stacks indicating certain classes are undefined. Here is a snippet of a Maven pom.xml I created to move Redis and its dependent component, apache commons-pool2, into my JAR.
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.9</version>
<executions>
<execution>
<id>unpack</id>
<!-- executed just before the package phase -->
<!-- https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/linking.html -->
<phase>prepare-package</phase>
<goals>
<goal>unpack</goal>
</goals>
<configuration>
<artifactItems>
<artifactItem>
<groupId>org.apache.commons</groupId>
<artifactId>commons-pool2</artifactId>
<version>2.4.2</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>${project.build.directory}/classes</outputDirectory>
<includes>org/apache/commons/**</includes>
</artifactItem>
<artifactItem>
<groupId>redis.clients</groupId>
<artifactId>jedis</artifactId>
<version>2.9.0</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>${project.build.directory}/classes</outputDirectory>
<includes>redis/clients/**</includes>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>

how to fail maven build on dependency conflict

Im working on a large multi-module project which uses an internal framework as one of its dependencies. The framework version is set in the top level pom at the beginning of a project, and has to stay constant. If any submodule uses a different version, I want the build to fail.
Ive tried declaring the dependency as a single version:
<dependency>
<groupId>framework_snt</groupId>
<artifactId>SFP</artifactId>
<version>[6.1]</version>
<type>pom</type>
</dependency>
Ive tried using the enforcer plugin with banned dependencies:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<version>1.0.1</version>
<executions>
<execution>
<id>enforce-versions</id>
<goals>
<goal>enforce</goal>
</goals>
<configuration>
<rules>
<requireJavaVersion>
<version>[1.6.0-21]</version>
</requireJavaVersion>
<requireMavenVersion>
<version>[3.0.3]</version>
</requireMavenVersion>
<bannedDependencies>
<excludes>
<exclude>framework_snt:SFP</exclude>
</excludes>
<includes>
<include>framework_snt:SFP:6.1.2</include>
</includes>
</bannedDependencies>
</rules>
</configuration>
</execution>
</executions>
</plugin>
Ive also tried adding the <DependencyConvergence/> tag as mentioned here, but none of these approaches work.
So given this top level pom fragment:
<project>
<groupId>glb</groupId>
<artifactId>GLB</artifactId>
<packaging>pom</packaging>
<name>Global</name>
<version>1.0</version>
........
<dependencies>
<dependency>
<groupId>framework_snt</groupId>
<artifactId>SFP</artifactId>
<version>[6.1.2]</version>
<type>pom</type>
</dependency>
</dependencies>
.....
</project>
And this (invalid) submmodule:
<project>
<groupId>glb</groupId>
<artifactId>CORE</artifactId>
<packaging>jar</packaging>
<name>Core</name>
<version>1.0</version>
<parent>
<groupId>glb</groupId>
<artifactId>GLB</artifactId>
<version>1.0</version>
</parent>
<dependencies>
<dependency>
<groupId>framework_snt</groupId>
<artifactId>SFP</artifactId>
<version>6.3</version>
<type>pom</type>
</dependency>
</dependencies>
</project>
how do I setup maven so the build will fail when using the child module above, but when I remove the tag <version>6.3</version> it succeeds (or I change the version to match the one from the top level pom?
Well after much search and trying stuff with various plugins and getting nowhere, I decided to write my own plugin to do conflict checking. Using the dependency:tree plugin source as a base I wrote the following:
/**
* Walks the dependency tree looking for anything marked as a conflict, and fails the build if one is found
* code ripped from the dependency:tree plugin
*
* #param rootNode
* the dependency tree root node to check
*
*/
private void checkForConflicts( DependencyNode rootNode ) throws MojoExecutionException
{
StringWriter writer = new StringWriter();
boolean conflicts=false;
// build the tree
DependencyNodeVisitor visitor = new SerializingDependencyNodeVisitor( writer, SerializingDependencyNodeVisitor.STANDARD_TOKENS);
visitor = new BuildingDependencyNodeVisitor( visitor );
rootNode.accept( visitor );
getLog().debug("Root: "+rootNode.toNodeString() );
DependencyNode node;
Artifact actual, conflict;
DependencyTreeInverseIterator iter = new DependencyTreeInverseIterator(rootNode);
while (iter.hasNext() )
{
node = (DependencyNode)iter.next();
if (node.getState() == DependencyNode.OMITTED_FOR_CONFLICT)
{
actual = node.getArtifact();
conflict = node.getRelatedArtifact();
getLog().debug("conflict node: " + node.toNodeString() );
getLog().debug("conflict with: "+conflict.toString() );
getLog().error(actual.getGroupId()+":"+actual.getArtifactId()+":"+actual.getType()+" Conflicting versions: "+actual.getVersion()+" and "+conflict.getVersion() );
conflicts=true;
}
}
if (conflicts)
throw new MojoExecutionException( "version conflict detected");
}
You can call this from your existing plugin using the following:
try
{
rootNode = dependencyTreeBuilder.buildDependencyTree( project, localRepository, artifactFactory, artifactMetadataSource, null, artifactCollector );
checkForConflicts( rootNode );
}
catch ( DependencyTreeBuilderException e)
{
throw new MojoExecutionException( "Cannot build project dependency tree", e);
}
Then in your plugin pom.xml, include dependencies from the existing dependency plugin source and your good to go.
This works for our project -- but if anyone knows a cleaner or better way, please let me know.