Serializing multiple files to binary Avro format - serialization

Question

I currently have a directory of files taking up a large amount of space. I'd like to compress and serialize each of these files to the binary Avro format to free up disk space. The schema for the Avro data file would just be a record containing a single field to store the content of the original file. I'm considering making use of a utility in the avro-tools jar provided by Apache to serialize each file. The utility takes an Avro schema and an input file containing records to be serialized in the JSON format and produces an Avro data file:
$ java -jar ~/avro-tools-1.7.6.jar fromjson --schema-file twitter.avsc twitter.json > twitter.avro
I'd like to write a bash script that executes this tool for each file in the folder, but I'm not sure how to form the equivalent record for each file in the JSON format that the tool expects. Each of the files is hundreds of MB in size. I was wondering if forming the JSON for a file ({'content': 'file content..'}) could be done using text processing commands or tools (awk, sed, etc.) from the command line. Also, are there better ways to accomplish of achieving the larger task, of migrating multiple files to Avro storage, than the approach I'm working on?
Thanks.

Answer 1

Looks good so far. You'll need a schema, something like
{
'type': 'record',
'fields':[{'name': 'content', 'type': bytes}]
}
Although I sympathize with the desire not to fire up a real programming language, the Java Avro API (for example) makes it easier create records, and easier to get the bytes from a file that's on your system.
Hope this helps,
Julian

Answer 2

I'm afraid you can't use Apache avro in such way. When you make avro schema, you need to determine each complex type (records, enums, arrays, maps, unions and fixed) with a primitive types (null, boolean, int, long, float, double, bytes, string), but cant describe a whole file in any way.
You can read this to find out what this library is.
If you will use a schema like this
{
'type': 'record',
'fields':[{'name': 'content', 'type': bytes}]
}
to serialize a file (even that file contains Json string), you will get AvroTypeException, cause avro-tools will expect field named "content" with type "bytes" and there is no such fields in used file.
If you want to serialize a file which contains a Json-string, you must describe in schema all fields in that Json-string.