I have the below code in driverscript. I need to embed TestNG to it. How can I do this? ActionKeywords.java is defining all the action keywords.
Where should I give the TestNG annotation?
import java.io.FileInputStream;
import java.lang.reflect.Method;
import java.util.Properties;
import org.apache.log4j.xml.DOMConfigurator;
import config.ActionKeywords;
import config.Constants;
import utility.ExcelUtils;
import utility.Log;
public class DriverScript {
public static Properties OR;
public static ActionKeywords actionKeywords;
public static String sActionKeyword;
public static String sPageObject;
public static Method method[];
public static int iTestStep;
public static int iTestLastStep;
public static String sTestCaseID;
public static String sRunMode;
public static String sData;
public static boolean bResult;
public DriverScript() throws NoSuchMethodException, SecurityException{
actionKeywords = new ActionKeywords();
method = actionKeywords.getClass().getMethods();
}
public static void main(String[] args) throws Exception {
ExcelUtils.setExcelFile(Constants.Path_TestData);
DOMConfigurator.configure("log4j.xml");
String Path_OR = Constants.Path_OR;
FileInputStream fs = new FileInputStream(Path_OR);
OR= new Properties(System.getProperties());
OR.load(fs);
DriverScript startEngine = new DriverScript();
startEngine.execute_TestCase();
}
private void execute_TestCase() throws Exception {
int iTotalTestCases = ExcelUtils.getRowCount(Constants.Sheet_TestCases);
System.out.println("Total TC count" + iTotalTestCases);
for(int iTestcase=1;iTestcase<iTotalTestCases;iTestcase++){
bResult = true;
sTestCaseID = ExcelUtils.getCellData(iTestcase, Constants.Col_TestCaseID, Constants.Sheet_TestCases);
sRunMode = ExcelUtils.getCellData(iTestcase, Constants.Col_RunMode,Constants.Sheet_TestCases);
if (sRunMode.equals("Yes")){
Log.startTestCase(sTestCaseID);
iTestStep = ExcelUtils.getRowContains(sTestCaseID, Constants.Col_TestCaseID, Constants.Sheet_TestSteps);
iTestLastStep = ExcelUtils.getTestStepsCount(Constants.Sheet_TestSteps, sTestCaseID, iTestStep);
bResult=true;
for (;iTestStep<iTestLastStep;iTestStep++){
sActionKeyword = ExcelUtils.getCellData(iTestStep, Constants.Col_ActionKeyword,Constants.Sheet_TestSteps);
sPageObject = ExcelUtils.getCellData(iTestStep, Constants.Col_PageObject, Constants.Sheet_TestSteps);
sData = ExcelUtils.getCellData(iTestStep, Constants.Col_DataSet, Constants.Sheet_TestSteps);
execute_Actions();
if(bResult==false){
ExcelUtils.setCellData(Constants.KEYWORD_FAIL,iTestcase,Constants.Col_Result,Constants.Sheet_TestCases);
Log.endTestCase(sTestCaseID);
break;
}
}
if(bResult==true){
ExcelUtils.setCellData(Constants.KEYWORD_PASS,iTestcase,Constants.Col_Result,Constants.Sheet_TestCases);
Log.endTestCase(sTestCaseID);
}
}
}
}
private static void execute_Actions() throws Exception {
for(int i=0;i<method.length;i++){
if(method[i].getName().equals(sActionKeyword)){
method[i].invoke(actionKeywords,sPageObject, sData);
if(bResult==true){
ExcelUtils.setCellData(Constants.KEYWORD_PASS, iTestStep, Constants.Col_TestStepResult, Constants.Sheet_TestSteps);
break;
}else{
ExcelUtils.setCellData(Constants.KEYWORD_FAIL, iTestStep, Constants.Col_TestStepResult, Constants.Sheet_TestSteps);
ActionKeywords.closeBrowser("","");
break;
}
}
}
}
}
Related
My situation asks for a bit more complex serialisation. I have a class Available (this is a very simplified snippet):
public class Available<T> {
private T value;
private boolean available;
...
}
So a POJO
class Tmp {
private Available<Integer> myInt = Available.of(123);
private Available<Integer> otherInt = Available.clean();
...
}
would normally result in
{"myInt":{available:true,value:123},"otherInt":{available:false,value:null}}
However, I want a serialiser to render the same POJO like this:
{"myInt":123}
What I have now:
public class AvailableSerializer extends JsonSerializer<Available<?>> {
#Override
public void serialize(Available<?> available, JsonGenerator jsonGenerator, SerializerProvider provider) throws IOException, JsonProcessingException {
if (available != null && available.isAvailable()) {
jsonGenerator.writeObject(available.getValue());
}
// MISSING: nothing at all should be rendered here for the field
}
#Override
public Class<Available<?>> handledType() {
#SuppressWarnings({ "unchecked", "rawtypes" })
Class<Available<?>> clazz = (Class) Available.class;
return clazz;
}
}
A test
#Test
public void testSerialize() throws Exception {
SimpleModule module = new SimpleModule().addSerializer(new AvailableSerializer());
ObjectMapper objectMapper = new ObjectMapper();
objectMapper.registerModule(module);
System.out.println(objectMapper.writeValueAsString(new Tmp()));
}
outputs
{"myInt":123,"otherInt"}
Can anyone tell me how to do the "MISSING"-stuff? Or if I'm doing it all wrong, how do I do it then?
The restriction I have is that I don't want the developers to add #Json...-annotations all the time to fields of type Available. So the Tmp-class above is an example of what a typical using class should look like. If that's possible...
Include.NON_DEFAULT
If we assume that your clean method is implemented in this way:
class Available<T> {
public static final Available<Object> EMPTY = clean();
//....
#SuppressWarnings("unchecked")
static <T> Available<T> clean() {
return (Available<T>) EMPTY;
}
}
You can set serialisation inclusion to JsonInclude.Include.NON_DEFAULT value and it should skip values set to EMPTY (default) values. See below example:
import com.fasterxml.jackson.annotation.JsonInclude;
import com.fasterxml.jackson.core.JsonGenerator;
import com.fasterxml.jackson.databind.JsonSerializer;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializerProvider;
import com.fasterxml.jackson.databind.module.SimpleModule;
import com.fasterxml.jackson.databind.ser.BeanPropertyWriter;
import java.io.IOException;
public class JsonApp {
public static void main(String[] args) throws Exception {
SimpleModule module = new SimpleModule();
module.addSerializer(new AvailableSerializer());
ObjectMapper objectMapper = new ObjectMapper();
objectMapper.registerModule(module);
objectMapper.setSerializationInclusion(JsonInclude.Include.NON_DEFAULT);
System.out.println(objectMapper.writeValueAsString(new Tmp()));
}
}
class AvailableSerializer extends JsonSerializer<Available<?>> {
#Override
public void serialize(Available<?> value, JsonGenerator jsonGenerator, SerializerProvider provider) throws IOException {
jsonGenerator.writeObject(value.getValue());
}
#Override
#SuppressWarnings({"unchecked", "rawtypes"})
public Class<Available<?>> handledType() {
return (Class) Available.class;
}
}
Above code prints:
{"myInt":123}
Custom BeanPropertyWriter
If you do not want to use Include.NON_DEFAULT you can write your custom BeanPropertyWriter and skip all values you want. See below example:
import com.fasterxml.jackson.core.JsonGenerator;
import com.fasterxml.jackson.databind.BeanDescription;
import com.fasterxml.jackson.databind.JsonSerializer;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationConfig;
import com.fasterxml.jackson.databind.SerializerProvider;
import com.fasterxml.jackson.databind.module.SimpleModule;
import com.fasterxml.jackson.databind.ser.BeanPropertyWriter;
import com.fasterxml.jackson.databind.ser.BeanSerializerModifier;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
public class JsonApp {
public static void main(String[] args) throws Exception {
SimpleModule module = new SimpleModule();
module.addSerializer(new AvailableSerializer());
module.setSerializerModifier(new BeanSerializerModifier() {
#Override
public List<BeanPropertyWriter> changeProperties(SerializationConfig config, BeanDescription beanDesc, List<BeanPropertyWriter> beanProperties) {
List<BeanPropertyWriter> writers = new ArrayList<>(beanProperties.size());
for (BeanPropertyWriter writer : beanProperties) {
if (writer.getType().getRawClass() == Available.class) {
writer = new SkipNotAvailableBeanPropertyWriter(writer);
}
writers.add(writer);
}
return writers;
}
});
ObjectMapper objectMapper = new ObjectMapper();
objectMapper.registerModule(module);
System.out.println(objectMapper.writeValueAsString(new Tmp()));
}
}
class AvailableSerializer extends JsonSerializer<Available<?>> {
#Override
public void serialize(Available<?> value, JsonGenerator jsonGenerator, SerializerProvider provider) throws IOException {
jsonGenerator.writeObject(value.getValue());
}
#Override
#SuppressWarnings({"unchecked", "rawtypes"})
public Class<Available<?>> handledType() {
return (Class) Available.class;
}
}
class SkipNotAvailableBeanPropertyWriter extends BeanPropertyWriter {
SkipNotAvailableBeanPropertyWriter(BeanPropertyWriter base) {
super(base);
}
#Override
public void serializeAsField(Object bean, JsonGenerator gen, SerializerProvider prov) throws Exception {
// copier from super.serializeAsField(bean, gen, prov);
final Object value = (_accessorMethod == null) ? _field.get(bean) : _accessorMethod.invoke(bean, (Object[]) null);
if (value == null || value instanceof Available && !((Available) value).isAvailable()) {
return;
}
super.serializeAsField(bean, gen, prov);
}
}
Above code prints:
{"myInt":123}
After Michał Ziober's answer I had to look for something regarding Include.NON_DEFAULT and the default object and ran into this answer explaining Include.NON_EMPTY that Google didn't return in my first research (thanks Google).
So things become easier, it's now:
public class AvailableSerializer extends JsonSerializer<Available<?>> {
#Override
public void serialize(Available<?> available, JsonGenerator jsonGenerator, SerializerProvider provider) throws IOException, JsonProcessingException {
jsonGenerator.writeObject(available.getValue());
}
#Override
public Class<Available<?>> handledType() {
#SuppressWarnings({ "unchecked", "rawtypes" })
Class<Available<?>> clazz = (Class) Available.class;
return clazz;
}
#Override
public boolean isEmpty(SerializerProvider provider, Available<?> value) {
return value == null || !value.isAvailable();
}
}
with the test
#Test
public void testSerialize() throws Exception {
SimpleModule module = new SimpleModule().addSerializer(availableSerializer);
objectMapper.registerModule(module);
objectMapper.configOverride(Available.class).setInclude(
// the call comes from JavaDoc of objectMapper.setSerializationInclusion(...)
JsonInclude.Value.construct(JsonInclude.Include.NON_EMPTY, JsonInclude.Include.ALWAYS));
Tmp tmp = new Tmp();
assertThat(objectMapper.writeValueAsString(tmp)).isEqualTo("{\"myInt\":123}");
tmp.otherInt.setValue(123);
assertThat(objectMapper.writeValueAsString(tmp)).isEqualTo("{\"myInt\":123,\"otherInt\":123}");
}
So please, if you upvote my answer please also upvote Michał Ziober's as that's also working with a mildly different approach.
ReporterClass.Java:
package POM_Classes;
import com.aventstack.extentreports.AnalysisStrategy;
import com.aventstack.extentreports.ExtentReports;
import com.aventstack.extentreports.ExtentTest;
import com.aventstack.extentreports.reporter.ExtentHtmlReporter;
public class ReporterClass {
public static ExtentHtmlReporter html;
public ExtentReports extent;
public ExtentTest test, suiteTest;
public String testCaseName, testNodes, testDescription, category, authors;
public void startResult() {
html = new ExtentHtmlReporter("./reports/result.html");
html.setAppendExisting(true);
extent = new ExtentReports();
extent.attachReporter(html);
}
/*public ExtentTest startTestModule(String testCaseName, String testDescription) {
suiteTest = extent.createTest(testCaseName, testDescription);
return suiteTest;
}*/
public ExtentTest startTestCase(String testName) {
System.out.println(testName);
test = extent.createTest(testName);
return test;
}
public void reportStep(String desc, String status) {
if (status.equalsIgnoreCase("PASS")) {
test.pass(desc);
} else if (status.equalsIgnoreCase("FAIL")) {
test.fail(desc);
} else if (status.equalsIgnoreCase("WARNING")) {
test.warning(desc);
} else if (status.equalsIgnoreCase("INFO")) {
test.info(desc);
}
}
public void endTestcase() {
extent.setAnalysisStrategy(AnalysisStrategy.CLASS);
}
public void endResult() {
extent.flush();
}
}
Usage:
#Test
public void 961_NavigateToMyAlertsAndAddNewAlerts()
throws IOException, InterruptedException, ATUTestRecorderException, APIException {
driver = launchTargetUrl(this.getClass().getSimpleName().toString());
startResult();
test = startTestCase(Thread.currentThread().getStackTrace()[1].getMethodName().toString());
LoginApplication(driver,transactionusername, transactionuserpassword,test);
HomePage homePage = new HomePage(driver,test);
homePage.MyAlerts.click();
MyAlerts myalerts = new MyAlerts(driver,test);
String selectedcardName;
selectedcardName = driver.findElement(myalerts.cardName).getText().trim();
System.out.println(selectedcardName);
}
#AfterMethod
public void afterMethod(ITestResult result) throws ATUTestRecorderException {
resultOfTest(result);
endTestcase();
endResult();
closeBrowsers(driver);
}
The test case which first gets completed has the report and if the another test case is completed then that result overrides the old one..
If I change public static ExtentReports extent; then it maintains only thread so it logs only one test case and the other parallel execution is not even recorded.. How to resolve this?
Ok, here you go. I haven't tested this yet, but this should allow you to use Extent with parallel usage.
Reporter:
public abstract class ReporterClass {
private static final ExtentReports EXTENT = ExtentManager.getInstance();
public ExtentTest test, suiteTest;
public String testCaseName, testNodes, testDescription, category, authors;
public synchronized ExtentTest startTestCase(String testName) {
System.out.println(testName);
return ExtentTestManager.createTest(testName);
}
public synchronized void reportStep(String desc, String status) {
if (status.equalsIgnoreCase("PASS")) {
ExtentTestManager.getTest().pass(desc);
} else if (status.equalsIgnoreCase("FAIL")) {
ExtentTestManager.getTest().fail(desc);
} else if (status.equalsIgnoreCase("WARNING")) {
ExtentTestManager.getTest().warning(desc);
} else if (status.equalsIgnoreCase("INFO")) {
ExtentTestManager.getTest().info(desc);
}
}
public synchronized void endResult() {
EXTENT.flush();
}
#BeforeMethod
public synchronized void beforeMethod(Method method) {
startTestCase(method.getName());
}
#AfterMethod
public synchronized void afterMethod(ITestResult result) throws ATUTestRecorderException {
reportStep(result.getThrowable(), result.getStatus());
endResult();
closeBrowsers(driver);
}
}
Base:
public abstract class BaseClass extends ReporterClass {
// .. abstractions
}
Extent utils:
public class ExtentTestManager {
static Map<Integer, ExtentTest> extentTestMap = new HashMap<Integer, ExtentTest>();
private static final ExtentReports EXTENT = ExtentManager.getInstance();
public static synchronized ExtentTest getTest() {
return extentTestMap.get((int) (long) (Thread.currentThread().getId()));
}
public static synchronized ExtentTest createTest(String testName) {
return createTest(testName, "");
}
public static synchronized ExtentTest createTest(String testName, String desc) {
ExtentTest test = EXTENT.createTest(testName, desc);
extentTestMap.put((int) (long) (Thread.currentThread().getId()), test);
return test;
}
}
public class ExtentManager {
private static ExtentReports extent;
public synchronized static ExtentReports getInstance() {
if (extent == null) {
createInstance("reports/extent.html");
}
return extent;
}
public synchronized static ExtentReports createInstance(String fileName) {
ExtentHtmlReporter htmlReporter = new ExtentHtmlReporter(fileName);
htmlReporter.config().setTestViewChartLocation(ChartLocation.BOTTOM);
htmlReporter.config().setChartVisibilityOnOpen(true);
htmlReporter.config().setTheme(Theme.STANDARD);
htmlReporter.config().setDocumentTitle(fileName);
htmlReporter.config().setEncoding("utf-8");
htmlReporter.config().setReportName(fileName);
htmlReporter.setAppendExisting(true);
extent = new ExtentReports();
extent.setAnalysisStrategy(AnalysisStrategy.CLASS);
extent.attachReporter(htmlReporter);
return extent;
}
}
Finally, your slim tests. Notice there is 0 lines of reporter code here - see ReporterClass.
public class MyTestsClass extends BaseClass {
#Test
public void 961_NavigateToMyAlertsAndAddNewAlerts()
throws IOException, InterruptedException, ATUTestRecorderException, APIException {
driver = launchTargetUrl(this.getClass().getSimpleName().toString());
LoginApplication(driver,transactionusername, transactionuserpassword,test);
HomePage homePage = new HomePage(driver,test);
homePage.MyAlerts.click();
MyAlerts myalerts = new MyAlerts(driver,test);
String selectedcardName;
selectedcardName = driver.findElement(myalerts.cardName).getText().trim();
System.out.println(selectedcardName);
}
}
//Add below class in your Project. First you need to add the respective object and
//call them respectively. Declaring Extent and Driver as static is a big problem in
//Parallel/execution.
//Avoid using static as far as possible by using the below class.
import java.util.ArrayList;
import java.util.List;
import org.openqa.selenium.WebDriver;
import com.aventstack.extentreports.ExtentReports;
import com.aventstack.extentreports.ExtentTest;
public class WebDriverFactory {
private static ThreadLocal<WebDriver> drivers=new ThreadLocal<>();
private static List<WebDriver> storeDrivers=new ArrayList<WebDriver>();
private static List<ExtentTest> extent=new ArrayList<ExtentTest>();
private static ThreadLocal<ExtentTest> reports=new ThreadLocal<ExtentTest>();
static
{
Runtime.getRuntime().addShutdownHook(new Thread(){
public void run()
{
storeDrivers.stream().forEach(WebDriver::quit);
}
});
}
public static WebDriver getDriver()
{
return drivers.get();
}
public static ExtentTest getextentReportObject()
{
return reports.get();
}
public static void addDriver(WebDriver driver)
{
storeDrivers.add(driver);
drivers.set(driver);
}
public static void addExtentReportObject(ExtentTest report)
{
extent.add(report);
reports.set(report);
}
public static void removeDriver()
{
storeDrivers.remove(drivers.get());
drivers.remove();
}
}
//Add and Invoke the object in the following way
/*** Add and invoke the object in the below fashion **/
WebDriverFactory.addExtentReportObject(extent.createTest("Monitor Scenario
").createNode("Monitor Page Validation"));
WebDriverFactory.getextentReportObject().assignCategory("#SmokeTest");
I want to use jvm parameter "-javaagent" to modify Account.class, make it can count execution time,but there have a error inforamtion,i don't know why.
source code：
Account.class
public class Account {
public void operation(){
System.out.println("operations.....");
try{
Thread.sleep(10);
}catch(InterruptedException e){
e.printStackTrace();
}
}
}
TimeStat.class
public class TimeStat {
static ThreadLocal<Long> t=new ThreadLocal<>();
public static void start(){
t.set(System.currentTimeMillis());
}
public static void end(){
long time=System.currentTimeMillis();
System.out.println(Thread.currentThread().getStackTrace()[2]+"speed:");
System.out.println(time);
}
}
Main class
public class RunAccountMain {
public static void main(String[] args){
Account account=new Account();
account.operation();
System.out.println(Account.class.getName());
}
}
I can use some classes to modify Account.class in the follow:
PreMainTraceAgent .class
import java.lang.instrument.ClassFileTransformer;
import java.lang.instrument.IllegalClassFormatException;
import java.lang.instrument.Instrumentation;
import java.lang.instrument.UnmodifiableClassException;
import java.security.ProtectionDomain;
import org.objectweb.asm.ClassReader;
import org.objectweb.asm.ClassWriter;
public class PreMainTraceAgent {
public static void premain(String agentArgs,Instrumentation inst)throws ClassNotFoundException,UnmodifiableClassException{
System.out.println("agentArgs:"+agentArgs);
inst.addTransformer(new ClassFileTransformer() {
#Override
public byte[] transform(ClassLoader classLoader, String classname,
Class<?> classbeing, ProtectionDomain protectionDomain,
byte[] classfilebuffer)
throws IllegalClassFormatException {
if(classname.equals("asmtimer.Account")){
System.out.println("meet asmtimer/Account");
ClassReader cr=new ClassReader(classfilebuffer);
ClassWriter cw=new ClassWriter(ClassWriter.COMPUTE_MAXS|
ClassWriter.COMPUTE_FRAMES);
TimeStatClassAdapter adapter=new TimeStatClassAdapter(cw);
cr.accept(adapter, ClassReader.SKIP_DEBUG);
return cw.toByteArray();
}else{
System.out.println(classname);
return classfilebuffer;
}
}
},true);
}
}
TimeStatClassAdapter.class
import org.objectweb.asm.ClassAdapter;
import org.objectweb.asm.ClassVisitor;
import org.objectweb.asm.MethodVisitor;
public class TimeStatClassAdapter extends ClassAdapter {
public TimeStatClassAdapter(ClassVisitor cv){
super(cv);
}
public MethodVisitor visitMethod(final int access,final String name,
final String desc,final String signature,
final String[] exceptions){
MethodVisitor mv=cv.visitMethod(access, name, desc, signature, exceptions);
MethodVisitor wrappedMv=mv;
if(mv!=null){
if(name.equals("operation")){
wrappedMv=new TimeStatMethodAdapter(mv);
}
}
return wrappedMv;
}
}
TimeStatMethodAdapter.class
import org.objectweb.asm.MethodAdapter;
import org.objectweb.asm.MethodVisitor;
import org.objectweb.asm.Opcodes;
public class TimeStatMethodAdapter extends MethodAdapter{
public TimeStatMethodAdapter(MethodVisitor ss){
super(ss);
}
#Override
public void visitCode(){
mv.visitMethodInsn(Opcodes.INVOKESTATIC, "asmtimer/TimeStat",
"start", "()V");
super.visitCode();
}
#Override
public void visitInsn(int opcode){
if(opcode>=Opcodes.IRETURN&&opcode<=Opcodes.RETURN){
mv.visitMethodInsn(Opcodes.INVOKESTATIC, "asmtimer/TimeStat",
"end", "()V");
}
mv.visitInsn(opcode);
}[enter image description here][1]
}
MAINFEST.MF
Manfest-Version: 1.0
Premain-Class: asmtimer.PreMainTraceAgent
Can-Redine-Classes: true
Can-Retransform-Classes: true
Console error informattion:
enter image description here
jvm parameter:
enter image description here
Jar file have exsit in "C:\jat.jar", If you have any idle to help me ,please save you words!!!
I'm trying to implement a hadoop job, that counts how often a object (Click) appears in a dataset.
Therefore i wrote a custom file input format. The record reader seems to read only the first line of the given file and the close the input stream.
Here is the code:
The Pojo class:
package model;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import org.apache.hadoop.io.WritableComparable;
public class Click implements WritableComparable<Click> {
private String user;
private String clickStart;
private String date;
private String clickTarget;
#Override
public void write(DataOutput out) throws IOException {
out.writeUTF(user);
out.writeUTF(clickStart);
out.writeUTF(date);
out.writeUTF(clickTarget);
}
#Override
public void readFields(DataInput in) throws IOException {
user = in.readUTF();
clickStart = in.readUTF();
date = in.readUTF();
clickTarget = in.readUTF();
}
public int compareTo(Click arg0) {
int response = clickTarget.compareTo(arg0.clickTarget);
if (response == 0) {
response = date.compareTo(arg0.date);
}
return response;
}
public String getUser(String user) {
return this.user;
}
public void setUser(String user) {
this.user = user;
}
public String getClickStart() {
return clickStart;
}
public void setClickStart(String clickStart) {
this.clickStart = clickStart;
}
public String getDate() {
return date;
}
public void setDate(String date) {
this.date = date;
}
public String getClickTarget() {
return clickTarget;
}
public void setClickTarget(String clickTarget) {
this.clickTarget = clickTarget;
}
public String toString() {
return clickStart + "\t" + date;
}
}
Here is the FileInputFormat class:
package ClickAnalysis;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import model.Click;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.mapreduce.lib.input.FileSplit;
import org.apache.hadoop.mapreduce.InputSplit;
import org.apache.hadoop.mapreduce.RecordReader;
import org.apache.hadoop.mapreduce.TaskAttemptContext;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.util.StringUtils;
import org.apache.tools.ant.types.CommandlineJava.SysProperties;
public class ClickAnalysisInputFormat extends FileInputFormat<Click, IntWritable>{
#Override
public RecordReader<Click, IntWritable> createRecordReader(
InputSplit split, TaskAttemptContext context) throws IOException,
InterruptedException {
System.out.println("Creating Record Reader");
return new ClickReader();
}
public static class ClickReader extends RecordReader<Click, IntWritable> {
private BufferedReader in;
private Click key;
private IntWritable value;
#Override
public void initialize(InputSplit inputSplit, TaskAttemptContext context) throws IOException, InterruptedException {
key = new Click();
value = new IntWritable(1);
System.out.println("Starting to read ...");
FileSplit split = (FileSplit) inputSplit;
Configuration conf = context.getConfiguration();
Path path = split.getPath();
InputStream is = path.getFileSystem(conf).open(path);
in = new BufferedReader(new InputStreamReader(is));
}
#Override
public boolean nextKeyValue() throws IOException, InterruptedException {
String line = in.readLine();
System.out.println("line: " + line);
boolean hasNextKeyValue;
if (line == null) {
System.out.println("line is null");
hasNextKeyValue = false;
} else {
String[] click = StringUtils.split(line, '\\', ';');
System.out.println(click[0].toString());
System.out.println(click[1].toString());
System.out.println(click[2].toString());
key.setClickStart(click[0].toString());
key.setDate(click[1].toString());
key.setClickTarget(click[2].toString());
value.set(1);
System.out.println("done with first line");
hasNextKeyValue = true;
}
System.out.println(hasNextKeyValue);
return hasNextKeyValue;
}
#Override
public Click getCurrentKey() throws IOException, InterruptedException {
return this.key;
}
#Override
public IntWritable getCurrentValue() throws IOException, InterruptedException {
return this.value;
}
#Override
public float getProgress() throws IOException, InterruptedException {
return 0;
}
public void close() throws IOException {
in.close();
System.out.println("in closed");
}
}
}
The Mapper class:
package ClickAnalysis;
import java.io.IOException;
import model.Click;
import model.ClickStartTarget;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.Mapper;
import org.jruby.RubyProcess.Sys;
public class ClickAnalysisMapper extends Mapper<Click, IntWritable, Click, IntWritable> {
private static final IntWritable outputValue = new IntWritable();
#Override
protected void map(Click key, IntWritable value, Context context) throws IOException, InterruptedException {
System.out.println("Key: " + key.getClickStart() + " " + key.getDate() + " " + key.getClickTarget() + " Value: " + value);
outputValue.set(value.get());
System.out.println(outputValue.get());
context.write(key, outputValue);
System.out.println("nach context");
}
}
Partitioner class:
package ClickAnalysis;
import model.Click;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.mapreduce.Partitioner;
public class ClickAnalysisPartitioner extends Partitioner<Click, IntWritable> {
#Override
public int getPartition(Click key, IntWritable value, int numPartitions) {
System.out.println("in Partitioner drinnen");
int partition = numPartitions;
return partition;
}
}
Hadoop Job, which is triggered via an Restful web service call in a servlet container, but this shouldn't be the problem:
package ClickAnalysis;
import model.Click;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.*;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat;
public class ClickAnalysisJob {
public int run() throws Exception {
// TODO Auto-generated method stub
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "ClickAnalysisJob");
job.setJarByClass(ClickAnalysisJob.class);
// Job Input path
FileInputFormat.setInputPaths(job, "hdfs://localhost:9000/user/hadoop/testdata1.csv");
// Job Output path
Path out = new Path("hdfs://localhost:9000/user/hadoop/clickdataAnalysis_out");
FileOutputFormat.setOutputPath(job, out);
out.getFileSystem(conf).delete(out,true);
job.setMapperClass(ClickAnalysisMapper.class);
job.setReducerClass(Reducer.class);
job.setPartitionerClass(ClickAnalysisPartitioner.class);
//job.setReducerClass(ClickAnalysisReducer.class);
job.setInputFormatClass(ClickAnalysisInputFormat.class);
job.setOutputFormatClass(SequenceFileOutputFormat.class);
job.setOutputKeyClass(Click.class);
job.setOutputValueClass(IntWritable.class);
job.setMapOutputKeyClass(Click.class);
job.setMapOutputValueClass(IntWritable.class);
System.out.println("in run drinnen");
//job.setGroupingComparatorClass(ClickTargetAnalysisComparator.class);
job.setNumReduceTasks(1);
int result = job.waitForCompletion(true)? 0:1;
return result;
}
}
Next the dataset (example):
/web/big-data-test-site/test-seite-1;2014-07-08;ein ziel
/web/big-data-test-site/test-seite-1;2014-07-08;ein anderes ziel
/web/big-data-test-site/test-seite-1;2014-07-08;ein anderes ziel
/web/big-data-test-site/test-seite-1;2014-07-08;ein ziel
/web/big-data-test-site/test-seite-1;2014-07-08;ein drittes ziel
/web/big-data-test-site/test-seite-1;2014-07-08;ein ziel
/web/big-data-test-site/test-seite-1;2014-07-08;ein viertes ziel
/web/big-data-test-site/test-seite-1;2014-07-08;ein ziel
When I run the program the syso's are showing following:
in run drinnen
Creating Record Reader
Starting to read ...
line: /web/big-data-test-site/test-seite-1;2014-07-08;ein ziel
/web/big-data-test-site/test-seite-1
2014-07-08
ein ziel
done with first line
true
Key: /web/big-data-test-site/test-seite-1 2014-07-08 ein ziel Value: 1
1
in closed
analyze Method: 1
From that i conclude that the record reader only reads the first line.
Why is this happening and how is it fixed?
I have a simple class that I want to deserialize into JSON using Jackson. I want to rename one field logically in my JSON and the other I want to have the same name as defined in my Java class.
#JsonSerialize(include = Inclusion.NON_NULL)
public static class Manifest {
public Manifest(){
this.files = new ArrayList<String>();
}
#JsonProperty("manifest-version")
private String manifestVersion;
private ArrayList<String> files;
#JsonIgnore
public String getManifestVersion() {
return manifestVersion;
}
#JsonIgnore
public void setManifestVersion(String manifestVersion) {
this.manifestVersion = manifestVersion;
}
public ArrayList<String> getFiles() {
return files;
}
public void setFiles(ArrayList<String> files) {
this.files = files;
}
public void addFile(String file) {
this.files.add(file);
}
}
I'm expecting the #JsonIgnore for the getter/setter to cause manifestVersion to not become a JSON property (But should create a JSON property for manifest-version, where I have the #JsonProperty defined.
Expected output is
{
"manifest-version" : "2.0"
}
Actual output is
{
"manifest-version" : "2.0",
"manifestVersion":"2.0"
}
Any help would be appreciated.
I tried executing your code with Jackson 2.2 and i'm getting the expected output
import java.util.ArrayList;
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
import com.fasterxml.jackson.databind.annotation.JsonSerialize.Inclusion;
public class Test {
#JsonSerialize(include = Inclusion.NON_NULL)
public static class Manifest {
public Manifest(){
this.files = new ArrayList<String>();
}
#JsonProperty("manifest-version")
private String manifestVersion;
private ArrayList<String> files;
#JsonIgnore
public String getManifestVersion() {
return manifestVersion;
}
#JsonIgnore
public void setManifestVersion(String manifestVersion) {
this.manifestVersion = manifestVersion;
}
public ArrayList<String> getFiles() {
return files;
}
public void setFiles(ArrayList<String> files) {
this.files = files;
}
public void addFile(String file) {
this.files.add(file);
}
}
public static void main(String[] args) throws JsonProcessingException {
ObjectMapper obj = new ObjectMapper();
Manifest m = new Manifest();
m.setManifestVersion("2.0");
System.out.println(obj.writeValueAsString(m));
}
}
Output: {"files":[],"manifest-version":"2.0"}
what version of jackson are you using?