List of map as inout of custom Hive UD - hive

Question

I'm created a UDF and expect List<MAP<STRING,STRING>> argument. It works fine in unit test. However when I use it in hql. I cannot get any value of the map by key. it's always null.
However log shows every key has non-null value. don't know why.
error message：
Caused by: java.lang.RuntimeException: service_type cannot be
null:{value=告訴%{name}為甚麼你需要更改訂單,
source_hash=d793db7dee0d1941600c29427383bce8c03ebd84,
source_locale=en, source_updated_at=1501377418000, content_type=PLAIN,
service_type=HUMAN}
// UDF code:
public class MusselContentUDF extends GenericUDF {
private static final Log LOG = LogFactory.getLog(MusselContentUDF.class);
private ListObjectInspector listObjectInspector;
private MapObjectInspector mapObjectInspector;
#Override
public ObjectInspector initialize(ObjectInspector[] arguments) throws UDFArgumentException {
ObjectInspector a = arguments[0];
if (!(a instanceof ListObjectInspector)) {
throw new UDFArgumentException("first argument must be a list / array");
}
this.listObjectInspector = (ListObjectInspector) a;
if (!(listObjectInspector.getListElementObjectInspector() instanceof MapObjectInspector)) {
throw new UDFArgumentException("element must be map type");
}
this.mapObjectInspector =
(MapObjectInspector) (listObjectInspector.getListElementObjectInspector());
return PrimitiveObjectInspectorFactory.javaByteArrayObjectInspector;
}
#Override
public byte[] evaluate(DeferredObject[] arguments) throws HiveException {
List<MusselContentUnit> contentUnits =
this.listObjectInspector
.getList(arguments[0].get())
.stream()
.map(
e -> {
Map<?, ?> map = mapObjectInspector.getMap(e);
if (map.get("service_type") == null) {
LOG.error("service_type cannot be null:" + map);
throw new RuntimeException("service_type cannot be null:" + map);
}
if (map.get("value") == null) {
LOG.error("value cannot be null:" + map);
throw new RuntimeException("value cannot be null:" + map);
}
if (map.get("source_hash") == null) {
LOG.error("source_hash cannot be null:" + map);
throw new RuntimeException("source_hash cannot be null:" + map);
}
if (map.get("source_locale") == null) {
LOG.error("source_locale cannot be null:" + map);
throw new RuntimeException("source_locale cannot be null:" + map);
}
if (map.get("source_updated_at") == null) {
LOG.error("source_updated_at cannot be null:" + map);
throw new RuntimeException("source_updated_at cannot be null:" + map);
}
return MusselContentUnit.builder()
.serviceType((String) map.get("service_type"))
.value((String) map.get("value"))
.sourceContentDescriptor(
SourceContentDescriptor.builder()
.sourceHash((String) map.get("source_hash"))
.sourceLocale((String) map.get("source_locale"))
.sourceUpdatedAt(Long.valueOf((String) map.get("source_updated_at")))
.contentType((String) map.get("content_type"))
.build())
.build();
})
.collect(Collectors.toList());
try {
return ThriftCodec.serialize(MusselContent.builder().units(contentUnits).build(), true);
} catch (TException e) {
throw new HiveException("Cannot parse idl request content");
}
}
#Override
public String getDisplayString(String[] children) {
return "MusselContentUDF(" + children[0] + ")";
}
}
hql code:
q4 AS (
SELECT MUSSEL_PRIMARY_KEY(publisher_name, model, field_name, shard_num) AS primary_key
,CONCAT(field_name, '.', locale) AS secondary_key
,MAP(
'service_type', service_type,
'value', value,
'source_hash', source_hash,
'source_locale', source_locale,
'source_updated_at', source_updated_at,
'content_type', content_type
) AS content_unit
FROM q3
)
,
q5 AS (
SELECT primary_key
,secondary_key
,COLLECT_LIST(content_unit) AS content_units
FROM q4
GROUP BY primary_key, secondary_key
)

Answer 1

The map keys may not be strings when you get the map from mapObjectInspector. Currently I see two possible ways how to solve the problem:
Try to use getMapValueElement(Object data, Object key) method to get the map value
To work out keys and values ObjectInspectors, e.g.,
protected transient PrimitiveObjectInspector keyOI;
protected transient PrimitiveObjectInspector valueOI;
keyOI = (PrimitiveObjectInspector) this.mapObjectInspector.getMapKeyObjectInspector();
valueOI = (PrimitiveObjectInspector) this.mapObjectInspector.getMapValueObjectInspector();
then use getPrimitiveJavaObject to get the Java object and cast it to String, this way you can cast Map<?, ?> map to Map<String, String>, and then use new_map.get("service_type").
Unit tests are not realy helpful, I would suggest you to use HiveRunner instead