the type must implement the inherited abstract method Reducer.reduce(Object, Iterator, OutputCollector, Reporter) - mapper

I am new to Hadoop, and this is my first Hadoop program.
I am trying to create a Mapper class called WordMapper, but it throws be the below error.
The type WordMapper must implement the inherited abstract method Mapper.map(Object, Object, OutputCollector, Reporter)
public class WordMapper extends MapReduceBase implements Mapper
{
public void map(WritableComparable key, Writable values, OutputCollector output, Reporter reporter) throws IOException
{
String line=values.toString();
StringTokenizer tok=new StringTokenizer(line);
while(tok.hasMoreTokens())
{
String t=tok.nextToken();
output.collect(new Text(t), new IntWritable(1));
}
}
}
Can someone please tell me where i am going wrong and suggest to overcome the problem

try to fulfill your Mapper parameter like this:
public class WCMapper extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable>
public void map(LongWritable key, Text values, OutputCollector output, Reporter reporter)

You haven't provided any of the type parameters. Mapper is a generic interface; it's parameterized with type parameters for the input and output key and value types. Fill in K1, V1, K2, and V2 in the following code with the types you need:
public class WordMapper extends MapReduceBase implements Mapper<K1, V1, K2, V2> {
public void map(K1 key,
V1 value,
OutputCollector<K2, V2> output,
Reporter reporter)
throws IOException {
whatever();
}
}

Related

Using formatter in web flux

I am trying to use Formatter in webflux application but its throws
java.lang.IllegalStateException: Iterating over a toIterable() /
toStream() is blocking, which is not supported in thread
reactor-http-nio-2
Exception indicate i can't use block and method is expecting an object of PetType. I wanted to know if there is any other way to do this
#Component
public class PetTypeFormatter implements Formatter<PetType> {
private final PetTypeService petTypeServive;
public PetTypeFormatter(PetTypeService petTypeServive) {
this.petTypeServive = petTypeServive;
}
#Override
public String print(PetType petType, Locale locale) {
return petType.getName();
}
#Override
public PetType parse(String text, Locale locale) throws ParseException
{
Iterable<PetType> findPetTypes = petTypeServive.findAll().toIterable();
for (PetType type : findPetTypes)
{
if (type.getName().equals(text)) {
return type;
}
}
throw new ParseException("type not found: " + text, 0);
}
}
Edit:
The method signature of controller which i am using is
#PostMapping("/pets/new")
public String processCreationForm(#ModelAttribute("owner") Owner owner, #Valid Pet pet,BindingResult result, ModelMap model)
and the Pet class petType property which i was setting through the custom formatter when using webmvc
Edit2:
#Setter
#Getter
public class Pet
{
private String id;
private PetType petType;
#DateTimeFormat(pattern = "yyyy-MM-dd")
private LocalDate birthDate;
private String name;
}
#Setter
#Getter
public class PetType
{
private String name;
#Override
public String toString() {
return name;
}
}
You are trying to implement blocking business logic in a formatter.
The purpose of the Formatter<T> interface is to write custom parsing of strings, for example json strings, csv strings etc. and parse these into an object.
What you are doing is making a database call in a formatter, which is NOT the purpose of the formatter interface.
Since you have not shown us:
the purpose of the formatter
where the formatter is used
whats in the passed string into the formatter
what your request looks like
What a Pet class is
What a PetType is
I can't help you more than this. You are trying to do a blocking call in a webflux application in an interface that does not allow reactive coding (it returns a concrete value), You need to rethink your solution to the problem.
Please explain what your problem is and what it is you want to do, and not the problem with the code, the problem you are trying to solve, and we might be able to help you more.

LocalDateTime Serialization with Jackson results in fields being present twice

I have a custom serializer class
public class LocalDateTimeSerializer extends StdSerializer<LocalDateTime>{
#Override
public void serialize(LocalDateTime value, JsonGenerator jgen, SerializerProvider provider) throws IOException, JsonGenerationException {
// custom code to serialize
jgen.writeString(newValue);
}
I use it as below on a class variable:
#JsonSerialize(using = LocalDateTimeSerializer.class)
private LocalDateTime localDateTime;
Now, when I serialize this is json, the response datetime is serialized as expected according to the custom logic. All good here.
But there are some cases, when a field is mapped twice, for no apparent reason. The class variable is declared exactly the same to serialize with the custom serializer, but the output would have two fields for the variable. One would be the expected field serialized. The second would be the non-serialized field. Look at the following output:
"localDateTime": //as expected,
"localdateTime":{"year":2017,"month":"JUNE","monthValue":6,"dayOfMonth":29,"hour":20,"minute":0,"second":0,"nano":0,"dayOfWeek":"SATURDAY","dayOfYear":180,"chronology":{"calendarType":"iso8601","id":"ISO"}}
This is randomly happening for say two fields of a class.
Any idea what could cause this?
UPDATE:
I think it was something to do with how the getter/setter for the entity was named as.
#JsonSerialize(using = LocalDateTimeSerializer.class)
private LocalDateTime aOldDate;
#JsonSerialize(using = LocalDateTimeSerializer.class)
private LocalDateTime aNewDate;
public LocalDateTime getaOldDate() {
return aOldDate;
}
public void setaOldDate(LocalDateTime aOldDate) {
this.aOldDate= aOldDate;
}
public LocalDateTime getANewDate() {
return aNewDate;
}
public void setANewDate(LocalDateTime aNewDate) {
this.aNewDate= aNewDate;
}
If I try serializing this, the aNewDate field would be present twice in the json, and the aOldDate would be present once. It is probably because the 'A' is capital in the getter/setter for aNewDate, and this causes some problems. Not entirely sure what happens but relates to section 8.8 in this document: https://download.oracle.com/otn-pub/jcp/7224-javabeans-1.01-fr-spec-oth-JSpec/beans.101.pdf

How do I programmatically configure ObjectMapper in Java/Jackson to instantiate Java beans?

I have a number of Java bean interfaces like this:
public interface Dog
{
String getName();
void setName( final String value );
}
I also auto-generate bean implementations like this:
public final class DogImpl implements Dog
{
public String getName()
{
return m_name;
}
public void setName( final String value )
{
m_value = value;
}
private volatile String m_value;
}
ObjectMapper works perfectly except when I start nesting these beans like this:
public interface Dog
{
String getName();
void setName( final String value );
Dog getParent();
void setParent( final Dog value );
}
I get this error:
abstract types either need to be mapped to concrete types, have custom deserializer, or contain additional type information
It's complaining because the bean definition is an interface and not the concrete type. My question is if there is a way for me to define the mapping of interface types to concrete types for the ObjectMapper via a module or something?
Specifically, I can get a Map< Class< ? >, Class< ? > > of api type to implementation concrete type, but have no idea how to "give this" to the ObjectMapper so it understands how to look up the concrete types from the api types so it can instantiate them. How do I accomplish this?
This can be done using a SimpleAbstractTypeResolver.
This link shows you how to add the mappings to the resolver: Jackson - How to specify a single implementation for interface-referenced deserialization?
And this is how you add the resolver to an ObjectMapper:
final SimpleAbstractTypeResolver resolver = new SimpleAbstractTypeResolver();
for ( final Class< ? > api : apis )
{
resolver.addMapping( api, getConcreteImpl( api ) );
}
final SimpleModule module = new SimpleModule();
module.setAbstractTypes( resolver );
mapper.registerModule( module );

Morphia Interface for List of enum does not work (unmarshalling)

I have the following interface
#JsonTypeInfo(use = JsonTypeInfo.Id.CLASS, include = JsonTypeInfo.As.PROPERTY, property = "className")
public interface InfoChartInformation {
public String name();
}
And the following implementation (enum):
public class InfoChartSummary {
public static enum Immobilien implements InfoChartInformation {
CITY, CONSTRUCTION_DATE;
}
public static enum Cars implements InfoChartInformation {
POWER, MILEAGE;
}
}
Then I use all of It in the following entity:
#Entity(noClassnameStored = true)
#Converters(InfoChartInformationMorphiaConverter.class)
public class TestEntity{
#Id
public ObjectId id;
#Embedded
public List<InfoChartInformation> order;
}
Jackson, in order to detect the type on the unmarshalling time, will add to every enum on the list the className.
I thought morphia would do the same, but there's no field className in the List of enum and the unmarshalling cannot be done correctly: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassCastException: java.lang.String cannot be cast to com.mongodb
.DBObject
I guess the correct behavior should be to save all the enum route (package+name), not only the enum name. At least in that way the unmarshalling could be performed. There's a way morphia supports that by default or I need to create my own converter (similar to this) ?
I tried creating a Custom Converter:
public class InfoChartInformationMorphiaConverter extends TypeConverter{
public InfoChartInformationMorphiaConverter() {
super(InfoChartInformation.class);
}
#Override
public Object decode(Class targetClass, Object fromDBObject, MappedField optionalExtraInfo) {
if (fromDBObject == null) {
return null;
}
String clazz = fromDBObject.toString().substring(0, fromDBObject.toString().lastIndexOf("."));
String value = fromDBObject.toString().substring(fromDBObject.toString().lastIndexOf(".") + 1);
try {
return Enum.valueOf((Class)Class.forName(clazz), value);
} catch (ClassNotFoundException e) {
return null;
}
}
#Override
public Object encode(final Object value, final MappedField optionalExtraInfo) {
return value.getClass().getName() + "." + ((InfoChartInformation) value).name();
}
}
Then, I added the converter information to morphia morphia.getMapper().getConverters().addConverter(new InfoChartInformationMorphiaConverter());.
However, when serializing (or marshalling) the object to save it into the database, the custom converter is ignored and the Enum is saved using the default Morphia converter (only the enum name).
If I use in the TestEntity class only an attribute InfoChartInformation; instead of the List<>InfoChartInformation>, my customer converter will work. However I need support for List
Use:
public class InfoChartInformationMorphiaConverter extends TypeConverter implements SimpleValueConverter
It is a marker interface required to make your Convertor work.

PIG Algebraic UDF with arguments : Combiner optimizer error

I'm trying to build a Pig UDF that performs some aggregation on a variable of type double. To do so, I built an algebraic UDF called Aggreg. It is called in the following script:
REGISTER 'Test.jar';
DEFINE Aggreg com.pig.test.Agreg();
records = LOAD '/tmp/Test.csv' USING PigStorage(',') AS (v1:chararray, v2:double);
grouped_rec = GROUP records ALL;
test = FOREACH grouped_rec GENERATE Aggreg(records.v2) AS val;
DUMP test;
This works fine as it is. Then, I wanted to use the arguments for this UDF so I added a public constructor with one String argument.
I just changed the DEFINE statement in the previous script but haven't yet used the argument in the UDF Java code:
DEFINE Aggreg com.pig.test.Agreg('Test');
And now I get the following error:
ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2018: Internal error. Unable to introduce the combiner for optimization.
Any ideas where this could come from?
Using Algebraic interface, you must implement two constructors in classes Initial, Intermed and Final, the default constructor and constructor with the parameter you use.
static public class Initial extends EvalFunc<Tuple> {
public Initial(){}
public Initial(String str){Aggreg.string=trs;}
#Override
public Tuple exec(Tuple input) throws IOException {
...
}
}
static public class Intermed extends EvalFunc<Tuple> {
public Intermed(){}
public Intermed(String str){Aggreg.string=trs;}
#Override
public Tuple exec(Tuple input) throws IOException {
...
}
}
static public class Final extends EvalFunc<Tuple> {
public Final(){}
public Final(String str){Aggreg.string=trs;}
#Override
public Tuple exec(Tuple input) throws IOException {
...
}
}
public String getInitial() {
return Initial.class.getName();
}
public String getIntermed() {
return Intermed.class.getName();
}
public String getFinal() {
return Final.class.getName();
}