How to put Arrow vector to Ignite - ignite

Question

I try to create some arrow vectors and put them directly in ignite, but it failed, my questions are:
Does Ignite support store arrow data?
If Ignite support arrow data, how to store arrow vectors in ignite? Do I need to serialize data before put them in ignite?
My code:
private static final int DATA_SIZE = 1000;
try(Ignite ignite = Ignition.start("examples/config/example-ignite.xml")) {
BufferAllocator bufferAllocator = new RootAllocator(1024 * 1024);
IntVector intVector = new IntVector("int_vector", bufferAllocator);
VarCharVector varCharVector = new VarCharVector("varchar_vector", bufferAllocator);
intVector.allocateNew(DATA_SIZE);
intVector.setValueCount(DATA_SIZE);
varCharVector.allocateNew(DATA_SIZE);
varCharVector.setValueCount(DATA_SIZE);
for (int i = 0; i < DATA_SIZE; i++) {
intVector.set(i, i);
varCharVector.set(i, new Text("value-" + i));
}
try (IgniteCache<String, ValueVector> cache = ignite.getOrCreateCache(CACHE_NAME)) {
cache.put("key_int", intVector);
cache.put("key_varchar", varCharVector);
} finally {
ignite.destroyCache(CACHE_NAME);
}
}
Exception:
[2021-05-17 17:17:38,844][ERROR][main][BinaryContext] Failed to deserialize object [typeName=io.netty.buffer.PoolArena$DirectArena]
class org.apache.ignite.binary.BinaryObjectException: Failed to read field [name=chunkListMetrics]
at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(BinaryFieldAccessor.java:192)
at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:888)
Caused by: class org.apache.ignite.binary.BinaryObjectException: Failed to unmarshal object with optimized marshaller
at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1788)
at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1958)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: [clsLdr=sun.misc.Launcher$AppClassLoader#18b4aac2, err=Failed to deserialize object [typeName=java.util.Collections$UnmodifiableList]]
at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:242)
Caused by: java.io.IOException: Failed to deserialize object [typeName=java.util.Collections$UnmodifiableList]
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
Caused by: java.io.IOException: Failed to deserialize field [name=arena]
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:524)
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:608)
Caused by: java.io.IOException: Failed to deserialize object [typeName=io.netty.buffer.PoolArena$DirectArena]
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
Caused by: java.io.IOException: Failed to deserialize field [name=activeBytesHuge]
Caused by: java.io.IOException: Failed to deserialize object [typeName=io.netty.util.internal.LongAdderCounter]
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:350)
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:473)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:440)
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:517)
... 121 more
Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:604)
at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:953)
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
... 125 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:601)
... 127 more
Caused by: java.io.InvalidObjectException: Proxy required
at java.util.concurrent.atomic.LongAdder.readObject(LongAdder.java:265)
... 132 more
[17:17:38] Ignite node stopped OK [uptime=00:00:01.107]
Exception in thread "main" java.lang.NullPointerException
at org.apache.ignite.examples.storageEngine.ArrowGandivaTest.main(ArrowGandivaTest.java:40)

Answer 1

I'm not sure what happens here since it is actually possible to store LongAdder in cache.
I guess you are better off serializing those vectors explicitly, if only because they can't preserve a (shared) pool allocator after being put and get from/to Ignite without extra effort.