Tweak schema of CTAS parquet table in apache drill: make element required instead of optional - schema

Question

I'd like to produce a parquet file with a very specific schema using apache drill. I join two tables with CTAS like:
CREATE TABLE synthetic1 AS (
SELECT e1.returneddocids AS returneddocids, e1.pathinfo AS pathinfo, c1.counters AS counters
FROM dfs.`/tmp/tier1.parquet` e1 LEFT JOIN dfs.tmp.shadow3 c1 ON TRUE LIMIT 100
);
The resulting file schema looks like this:
message root {
optional group returneddocids {
repeated group list {
optional binary element (UTF8); // need this one as required, not optional
}
}
optional binary pathinfo (UTF8);
optional group counters {
repeated group list {
optional group element { // need this as required
optional binary name (UTF8); // need this as required
optional int32 value; // need this as required
}
}
}
}
I wonder how can I tweak the CTAS query so optional elements above changed to required?

Answer 1

It's quite convoluted, you can use CREATE OR REPLACE SCHEMA to apply constraints. In my case this kind of works (not exactly, though could be helpful for others struggling with a similar problem):
ALTER SESSION SET `store.table.use_schema_file` = true;
ALTER SESSION SET `exec.storage.enable_v3_text_reader` = true;
CREATE OR REPLACE SCHEMA (
returneddocids STRUCT<`list` STRUCT<`element` ARRAY<VARCHAR>>> NOT NULL,
pathinfo VARCHAR NOT NULL,
counters STRUCT<`list` STRUCT<`element` ARRAY<VARCHAR>>>
) FOR TABLE synthetic1;