how to query differently-structured files? - azure-data-lake

Question

Is it possible to execute a query against files that have different schemas?
I have 2 sets of files in the same directory. The second type has an extra field.
Type 1
id, first, last
1, liza, smith
Type 2
id, first, last, state
4, alex, gordon, CT
Desired Result
1, liza
4, alex
How do we query files with different schemas, but where you want the same output fields?
Here's what I have:
SELECT id, first
FROM "/one 1300/{files}.csv"
USING Extractors.Csv();
#interestingRows = SELECT id, first FROM #interestingRows;
OUTPUT #interestingRows
TO #uriPrefix + "/one 1300/output/output.csv"
USING Outputters.Csv();

Answer 1

The CSV outputter not solve your problem.
You will need a custom extractor to solve this.
I recommend you to use Flexible Extractor
check this:
https://github.com/Azure/usql/tree/master/Examples/FlexibleSchemaExtractor
https://blogs.msdn.microsoft.com/mrys/2016/08/15/how-to-deal-with-files-containing-rows-with-different-column-counts-in-u-sql-introducing-a-flexible-schema-extractor/
The other solutions is to extract data with diferent schema separatly

Answer 2

How about importing the column as one using a delimiter you know does not exist in the data, then splitting it afterwards using the Split method of the string class? Something like this:
#working =
EXTRACT wholeRow string
FROM "/one 1300/{*}.csv"
USING Extractors.Text(delimiter:'|');
#working =
SELECT
wholeRow.Split(',')[0] AS id,
wholeRow.Split(',')[1] AS first,
wholeRow.Split(',')[2] AS last
FROM #working;
OUTPUT #working
TO "/output/output.csv"
USING Outputters.Csv(quoting:false);

Answer 3

Since you said that this two types are actualy in the same file, assuming that they are like this:
You just extract it with all the columns and set quoting to false:
//Extract the data
#extractedData =
EXTRACT id int,
first string,
last string,
state string
FROM "data.csv"
USING Extractors.Csv(skipFirstNRows : 1, quoting: false);
Then you just select the fields you need and output them:
//Select the fields
#finalData = SELECT id, first FROM #extractedData;
//Output the data
OUTPUT #finalData
TO "/Desired Result.csv"
USING Outputters.Csv(quoting: false);
The desired result: