I need to convert a pig formatted map string representation (e.g. "['k1'#'v1','k2'#'v2']") to Map object.
I'd prefer using pig implementation rather than implementing it myself.
I found a method for an opposite conversion: DataType.mapToString(Map m)
If you really stick to Pig, you may come up with the followings:
...
import org.apache.pig.builtin.Utf8StorageConverter;
...
public class Test {
public static void main(String[] args) throws IOException {
String mapStr = "[k1#v1,k2#v2]";
Utf8StorageConverter converter = new Utf8StorageConverter();
Map<String, Object> map = converter.bytesToMap(mapStr.getBytes());
for (Map.Entry<String, Object> entry : map.entrySet()) {
System.out.println(entry.getKey() + " :: " + entry.getValue());
}
}
}
I assume you have Pig v0.10.0
Related
I have the following class:
public class ChronicleMapIndex<K, V> implements Index<K, V> {
private ChronicleMap<K, V> index;
private Map<String, String> characteristicsMap;
public void buildIndex(String name, Map<String, String> characteristicsMap, Path indexPath, Class<?> keyType, Class<?> valueType){
this.characteristicsMap = characteristicsMap;
String filename = name + ".bin";
Path indexFilePath = Paths.get(indexPath + filename);
try {
index = (ChronicleMap<K, V>) ChronicleMap
.of(keyType, valueType)
.name(name)
.entries(Long.parseLong(characteristicsMap.get("entries")))
.averageValueSize(Double.parseDouble(characteristicsMap.get("averageValueSize")))
.averageKeySize(Double.parseDouble(characteristicsMap.get("averageKeySize")))
.createOrRecoverPersistedTo(indexFilePath.toFile(), true);
} catch (IOException e) {
e.printStackTrace();
}
}
#Override
public V get(K key) {
return index.get(key);
}
#Override
public void put(K key, V value) {
index.put(key, value);
}
}
I basically want to store a chronicle maps map in a wrapper function. Here, K and V are the same as the keyType and valueType values passed. So I would like the ChronicleMap map to have the same key and value type as K and V.
However, when I create it, I get the following error:
net.openhft.chronicle.hash.ChronicleHashRecoveryFailedException: java.lang.AssertionError: java.lang.NoSuchMethodException: sun.nio.ch.FileChannelImpl.map0(int,long,long)
at net.openhft.chronicle.map.ChronicleMapBuilder.openWithExistingFile(ChronicleMapBuilder.java:1877)
at net.openhft.chronicle.map.ChronicleMapBuilder.createWithFile(ChronicleMapBuilder.java:1701)
at net.openhft.chronicle.map.ChronicleMapBuilder.recoverPersistedTo(ChronicleMapBuilder.java:1655)
at net.openhft.chronicle.map.ChronicleMapBuilder.createOrRecoverPersistedTo(ChronicleMapBuilder.java:1638)
at net.openhft.chronicle.map.ChronicleMapBuilder.createOrRecoverPersistedTo(ChronicleMapBuilder.java:1629)
at edu.upf.taln.indexer.index.chroniclemap.ChronicleMapIndex.buildIndex(ChronicleMapIndex.java:27)
in this line:
.createOrRecoverPersistedTo(indexFilePath.toFile(), true);
I was wondering if this error is because I'm doing something wrong with generics.
I am using ChronicleMap 3.19.4 and JNA 5.5.0 in Windows 10.
Here is an individual test you can run easily:
Map<String, String> characteristicsMap = new HashMap<>();
characteristicsMap.put("entries", Long.toString(123));
characteristicsMap.put("averageKeySize", Integer.toString(5));
characteristicsMap.put("averageValueSize", Integer.toString(5));
String name = "test";
Path indexPath = Paths.get("D:/trabajo"); // substitute this as needed
ChronicleMapIndex<String, String> index = new ChronicleMapIndex<>();
index.buildIndex(name, characteristicsMap, indexPath ,String.class, String.class);
This has nothing to do with generics. You're using Java 12+, and it's not supported. The internal sun.nio.ch.FileChannelImpl has changed somewhere between version 11 to 14, and is not backwards-compatible.
Our support policy is to only support LTS versions of Java. The highest we support right now is Java 11. Next LTS version (TBC) is 17, and that what we plan to support next.
I am using a library com.fasterxml.jackson library for JsonSchema,
I am creating an IntegerSchema object, when I set range for integer schema using below code:
main(){
IntegerSchema intSchema = new IntegerSchema();
// setMaximum accepts Double object
intSchema.setMaximum(new Double(102000000));
// setMaximum accepts Double object
intSchema.setMinimum(new Double(100));
printJsonSchema(intSchema);
}
public void printJsonSchema(JsonSchema schema){
ObjectMapper mapper = new ObjectMapper();
try {
logger.info(mapper.writeValueAsString(schema));
} catch (JsonProcessingException e) {
throw new IllegalStateException(e);
}
}
When I convert IntegerSchema to string using ObjectMapper getting below response:
{"type":"integer","maximum":1.02E8,"minimum":100.0}
maximum and minimum values are getting converted to scientific notation.
But I need output in non scientific notation as below:
{"type":"integer","maximum":102000000,"minimum":100}
I cannot change IntegerSchema class.
Please suggest how to get the required output without extending IntegerSchema class?
Thanks in advance
this is a java issue somewhat I believe. If you debug your program, your Double will always be displayed scientifically, so what we'll want is to force it into a String. This can be achieved in Java in multiple ways, and you can look it up here:
How to print double value without scientific notation using Java?
In terms of your specific question about Jackson, I've written up some code for you:
public class ObjectMapperTest {
public static void main(String[] args) throws JsonProcessingException {
IntegerSchema schema = new IntegerSchema();
schema.type = "Int";
schema.max = 10200000000d;
schema.min = 100d;
ObjectMapper m = new ObjectMapper();
System.out.println(m.writeValueAsString(schema));
}
public static class IntegerSchema {
#JsonProperty
String type;
#JsonProperty
double min;
#JsonProperty
#JsonSerialize(using=MyDoubleDesirializer.class)
double max;
}
public static class MyDoubleDesirializer extends JsonSerializer<Double> {
#Override
public void serialize(Double value, JsonGenerator gen, SerializerProvider serializers)
throws IOException, JsonProcessingException {
// TODO Auto-generated method stub
BigDecimal d = new BigDecimal(value);
gen.writeNumber(d.toPlainString());
}
}
}
The trick is to register a custom Serializer for your Double value. This way, you can control what you want.
I am using the BigDecimal value to create a String representation of your Double. The output then becomes (for the specific example):
{"type":"Int","min":100.0,"max":10200000000}
I hope that solves your problem.
Artur
Feature.WRITE_BIGDECIMAL_AS_PLAIN
set this for your Object Mapper
I know I am answering late, but something I faced may help other
While converting a BigDecimal I have faced below is working
mapper = mapper.setNodeFactory(JsonNodeFactory.withExactBigDecimals(true));
while this is not working for me
mapper.configure(JsonGenerator.Feature.WRITE_BIGDECIMAL_AS_PLAIN, true);
Update for Jakson 2.9.10:
Property WRITE_BIGDECIMAL_AS_PLAIN replaced to com.fasterxml.jackson.core.JsonGenerator. You could use:
mapper.enable(JsonGenerator.Feature.WRITE_BIGDECIMAL_AS_PLAIN);
If you are using ValueToTree then no need of any factory settings. only problem with ValueToTree is it is converting as TextNode (String Fromat), So if you have any logic based on ObjectNodes it will not work
You should use
mapper.configure(DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS, true);
To avoid scientific notation on floating numbers.
You can find an example below.
ObjectMapper mapper = new ObjectMapper();
mapper.configure(DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS, true);
String test ="{\"doubleValue\" : 0.00001}";
try {
System.out.println(mapper.readTree(test).toPrettyString());
} catch (JsonProcessingException e) {
e.printStackTrace();
}
Output
{
"doubleValue" : 0.00001
}
All the examples I've seen that use aspect oriented programming for logging either log just class, method name and duration, and if they log parameters and return values they simply use ToString(). I need to have more control over what is logged. For example I want to skip passwords, or in some cases log all properties of an object but in other cases just the id property.
Any suggestions? I looked at AspectJ in Java and Unity interception in C# and could not find a solution.
You could try introducing parameter annotations to augment your parameters with some attributes. One of those attributes could signal to skip logging the parameter, another one could be used to specify a converter class for the string representation.
With the following annotations:
#Documented
#Retention(RetentionPolicy.RUNTIME)
#Target(ElementType.METHOD)
public #interface Log {
}
#Documented
#Retention(RetentionPolicy.RUNTIME)
#Target(ElementType.PARAMETER)
public #interface SkipLogging {
}
#Documented
#Retention(RetentionPolicy.RUNTIME)
#Target(ElementType.PARAMETER)
public #interface ToStringWith {
Class<? extends Function<?, String>> value();
}
the aspect could look like this:
import java.lang.reflect.Parameter;
import java.util.function.Function;
import java.util.stream.Collectors;
import java.util.stream.IntStream;
import org.aspectj.lang.reflect.MethodSignature;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public aspect LoggingAspect {
private final static Logger logger = LoggerFactory.getLogger(LoggingAspect.class);
pointcut loggableMethod(): execution(#Log * *..*.*(..));
before(): loggableMethod() {
MethodSignature signature = (MethodSignature) thisJoinPoint.getSignature();
Parameter[] parameters = signature.getMethod()
.getParameters();
String message = IntStream.range(0, parameters.length)
.filter(i -> this.isLoggable(parameters[i]))
.<String>mapToObj(i -> toString(parameters[i], thisJoinPoint.getArgs()[i]))
.collect(Collectors.joining(", ",
"method execution " + signature.getName() + "(", ")"));
Logger methodLogger = LoggerFactory.getLogger(
thisJoinPointStaticPart.getSignature().getDeclaringType());
methodLogger.debug(message);
}
private boolean isLoggable(Parameter parameter) {
return parameter.getAnnotation(SkipLogging.class) == null;
}
private String toString(Parameter parameter, Object value) {
ToStringWith toStringWith = parameter.getAnnotation(ToStringWith.class);
if (toStringWith != null) {
Class<? extends Function<?, String>> converterClass =
toStringWith.value();
try {
#SuppressWarnings("unchecked")
Function<Object, String> converter = (Function<Object, String>)
converterClass.newInstance();
String str = converter.apply(value);
return String.format("%s='%s'", parameter.getName(), str);
} catch (Exception e) {
logger.error("Couldn't instantiate toString converter for logging "
+ converterClass.getName(), e);
return String.format("%s=<error converting to string>",
parameter.getName());
}
} else {
return String.format("%s='%s'", parameter.getName(), String.valueOf(value));
}
}
}
Test code:
public static class SomethingToStringConverter implements Function<Something, String> {
#Override
public String apply(Something something) {
return "Something nice";
}
}
#Log
public void test(
#ToStringWith(SomethingToStringConverter.class) Something something,
String string,
#SkipLogging Class<?> cls,
Object object) {
}
public static void main(String[] args) {
// execution of this method should log the following message:
// method execution test(something='Something nice', string='some string', object='null')
test(new Something(), "some string", Object.class, null);
}
I used Java 8 Streams API in my answer for it's compactness, you could convert the code to normal Java code if you don't use Java 8 features or need better efficiency. It's just to give you an idea.
A library is using Map to use some extra information. This map eventually is being converted a JSON object and I need to set request information to display for debugging purposes as this:
map.put("request", requestString);
I am considering to use Jackson specifically to create a JSON without quotes and want to set as requestString.
I am building necessary information regarding Request and building a Map including request headers, parameters, method etc.
Jackson is creating perfectly valid JSON with quotes but when I set this generated value inside map, It is displayed ugly because of having escaped quotes.
So Jackson is creating this:
{
method : "POST",
path : "/register"
}
When I set this in map, it turns to this:
{
method : \"POST\",
path : \"/register\"
}
Consider this as a huge map including all parameters and other information about request.
What I would like to want this:
{
method : POST,
path : /register
}
I know that this is not a valid JSON but I am using this as a String to a Map which is accepting String values.
public class UnQuotesSerializer extends NonTypedScalarSerializerBase<String>
{
public UnQuotesSerializer() { super(String.class); }
/**
* For Strings, both null and Empty String qualify for emptiness.
*/
#Override
public boolean isEmpty(String value) {
return (value == null) || (value.length() == 0);
}
#Override
public void serialize(String value, JsonGenerator jgen, SerializerProvider provider) throws IOException {
jgen.writeRawValue(value);
}
#Override
public JsonNode getSchema(SerializerProvider provider, Type typeHint) {
return createSchemaNode("string", true);
}
#Override
public void acceptJsonFormatVisitor(JsonFormatVisitorWrapper visitor, JavaType typeHint) throws JsonMappingException {
if (visitor != null) visitor.expectStringFormat(typeHint);
}
}
and
ObjectMapper objectMapper = new ObjectMapper();
SimpleModule module = new SimpleModule("UnQuote");
module.addSerializer(new UnQuotesSerializer());
objectMapper.configure(JsonGenerator.Feature.QUOTE_FIELD_NAMES, false);
objectMapper.configure(JsonParser.Feature.ALLOW_UNQUOTED_FIELD_NAMES, true);
objectMapper.configure(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS, true);
objectMapper.registerModule(module);
This is generating without quotes strings.
The following test passes (Jackson 2.5.0)
#Test
public void test() throws Exception {
ObjectMapper mapper = new ObjectMapper();
Map map = new HashMap();
map.put("method", "POST");
map.put("request", "/register");
String s = mapper.writeValueAsString(map);
Map map2 = mapper.readValue(s, Map.class);
Assert.assertEquals(map, map2);
}
so your pseudo JSON without quotes does not seem the way to go
I need to customize serialization of a POJO in Jackson so that I can apply filter on the properties based on user input
I applied the following annotations on the POJO.
#JsonFilter("userFilter")
#JsonSerialize(using = UserSerializer.class)
The custom serializer class is as below.
public class UserSerializer extends JsonSerializer<User> {
#Override
public void serialize(User value, JsonGenerator jgen,
SerializerProvider provider) throws IOException,
JsonProcessingException {
ObjectMapper mapper = new ObjectMapper();
SimpleFilterProvider sfp = new SimpleFilterProvider();
// create a set that holds name of User properties that must be serialized
Set userFilterSet = new HashSet<String>();
userFilterSet.add("firstName");
userFilterSet.add("corporateOrgs");
userFilterSet.add("rights");
userFilterSet.add("requirements");
sfp.addFilter("userFilter",SimpleBeanPropertyFilter.filterOutAllExcept(userFilterSet));
// create an objectwriter which will apply the filters
ObjectWriter writer = mapper.writer(sfp);
String json = writer.writeValueAsString(value);
}
}
I can see that Jackson is trying to serialize the POJO using the custom serializer defined. However it ends up in infinite recursion/stackoverflow as writer.writeValueAsString(value) ends up calling the custom serializer again.
Obviously I have not got some basic stuff right here. If the filtering is done outside the serialize method (for example in a method called from main() ), filtering works as expected.
can anyone please provide insight/link to documentation on how to make use of custom serialization to leverage filtering.
Fields can be filtered out with JsonFilter, or you can create a custom JsonSerialize serializer that writes out only certain fields.
Independent of the use of a JsonFilter, the attempt to recursively reserialize the same object to be serialized (first parameter of the overwritten serialize method) in a user-defined serializer with an object mapper will result in an infinite loop. Instead, in a custom serializer you would rather use the JsonGenerator methods (second parameter of the overridden serialize method) to write out field name/values.
In the following answer both variants (#JsonFilter and #JsonSerialize) are demonstrated, where only a part of the available fields are serialized to JSON.
#JsonFilter
To apply filters to properties based on user input, you do not need to extend JsonSerializer. Instead, you annotate the POJO with JsonFilter and just apply the filtering.
A self-contained example based on your code would look like this:
package com.software7.test;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.ser.impl.SimpleBeanPropertyFilter;
import com.fasterxml.jackson.databind.ser.impl.SimpleFilterProvider;
import java.util.HashSet;
import java.util.Set;
public class Main {
public static void main(String[] args) {
Main m = new Main();
try {
m.serialize();
} catch (JsonProcessingException e) {
e.printStackTrace();
}
}
void serialize() throws JsonProcessingException {
ObjectMapper mapper = new ObjectMapper();
SimpleFilterProvider sfp = new SimpleFilterProvider();
Set<String> userFilterSet = new HashSet<>();
userFilterSet.add("firstName");
userFilterSet.add("corporateOrgs");
userFilterSet.add("rights");
userFilterSet.add("requirements");
sfp.addFilter("UserFilter",
SimpleBeanPropertyFilter.filterOutAllExcept(userFilterSet));
mapper.setFilterProvider(sfp);
User user = new User("Brownrigg", "Don", "none", "+rwx", "n/a",
"some", "superfluous", "properties");
System.out.println(user);
System.out.println(">>>> serializing >>>>");
String s = mapper.writeValueAsString(user);
System.out.println(s);
}
}
User POJO
package com.software7.test;
import com.fasterxml.jackson.annotation.JsonFilter;
#JsonFilter("UserFilter")
public class User {
public String lastName;
public String firstName;
public String corporateOrgs;
public String rights;
public String requirements;
public String a, b, c;
public User(String lastName, String firstName, String corporateOrgs, String rights, String requirements,
String a, String b, String c) {
this.lastName = lastName;
this.firstName = firstName;
this.corporateOrgs = corporateOrgs;
this.rights = rights;
this.requirements = requirements;
this.a = a;
this.b = b;
this.c = c;
}
#Override
public String toString() {
return "User{" +
"lastName='" + lastName + '\'' +
", firstName='" + firstName + '\'' +
", corporateOrgs='" + corporateOrgs + '\'' +
", rights='" + rights + '\'' +
", requirements='" + requirements + '\'' +
", a='" + a + '\'' +
", b='" + b + '\'' +
", c='" + c + '\'' +
'}';
}
}
Test
The debug output of the above program would look like this:
User{lastName='Brownrigg', firstName='Don', corporateOrgs='none', rights='+rwx', requirements='n/a', a='some', b='superfluous', c='properties'}
>>>> serializing >>>>
{"firstName":"Don","corporateOrgs":"none","rights":"+rwx","requirements":"n/a"}
The test is successful! As you can see, the properties lastName, a, b and c are removed.
#JsonSerialize Alternative
If you want to use a customer serializer instead you can do like so:
Replace the annotation:
#JsonFilter("UserFilter")
with
#JsonSerialize(using = UserSerializer.class)
but do not use both.
The UserSerializer class could look like this:
package com.software7.test;
import com.fasterxml.jackson.core.JsonGenerator;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.JsonSerializer;
import com.fasterxml.jackson.databind.SerializerProvider;
import java.io.IOException;
public class UserSerializer extends JsonSerializer<User> {
#Override
public void serialize(User user, JsonGenerator jsonGenerator, SerializerProvider serializerProvider)
throws IOException, JsonProcessingException {
jsonGenerator.writeStartObject();
jsonGenerator.writeObjectField("firstName", user.firstName);
jsonGenerator.writeObjectField("corporateOrgs", user.corporateOrgs);
jsonGenerator.writeObjectField("rights", user.rights);
jsonGenerator.writeObjectField("requirements", user.requirements);
jsonGenerator.writeEndObject();
}
}
Finally, the serialization method would look like this:
void serialize() throws JsonProcessingException {
ObjectMapper mapper = new ObjectMapper();
User user = new User("Brownrigg", "Don", "none", "+rwx", "n/a",
"some", "superfluous", "properties");
System.out.println(user);
System.out.println(">>>> serializing >>>>");
String s = mapper.writeValueAsString(user);
System.out.println(s);
}
The result would be the same in this example. Which variant is better suited depends on the specific use case or personal preferences.