Does Spark SQL provide an API to parse a SQL statement and corresponding DDL and infer data types of the select list? - apache-spark-sql

Question

I'm reviewing Spark SQL for a project and I see all the pieces of the API I need (SQL Parser, Dataset, Encoder, LogicalPlan, etc) however I'm having difficulty tying them together the way I'd like.
Essentially I want the following functionality:
var ddl = parseDDL(RAW_DDL);
var query = parseQuery("SELECT c1, c2 FROM t1 WHERE c2='value'", ddl);
var selectFields = query.getSelectFields();
for(var field: selectFields) {
var name = field.getName();
var type = field.getType(); // <~~~ want this in terms of `t1` from ddl
...
}
The type information for the select list in terms of the DDL is what I'm after.
Ideally I'd like a soup-to-nuts example with Spark SQL, if possible.
UPDATE
To clarify let's say I have an SQL schema file with several CREATE TABLE statements:
File: com/abc/MovieDb.sql
CREATE TABLE Movie (
Title varchar(255),
Year integer,
etc.
);
CREATE TABLE Actor (
FirstName varchar(255),
LastName varchar(255),
etc.
);
etc.
I want to use Spark SQL to parse a number of arbitrary SQL SELECT statements against this schema. Importantly, I want to get type information about the select list of each query in terms of the Movie, Actor, etc. tables and columns in the schema. For example:
SELECT Title, Year FROM Movie WHERE Year > 1990
I want to parse this query against the schema and get the type information for the select list. Again the queries are arbitrary, however the schema is stable so something like:
var parser = createParser(schema);
var query = parser.parseQuery(arbitraryQuery);
var selectedFields = query.getSelectedFields();
for (var field: selectedFields) {
var name = field.getName();
var type = field.getType();
}
Most important is the field.getType() call.
I assumed this would be an easy Yes or No type question, but perhaps my use-case is off the beaten path. Time to dive into it myself...

Answer 1

to get columns information here is what can be done
suppose you have dataframe with columns A,B,C,D in it
val inputDf= Seq(("foo","Bar",0,0.0)).toDf("a","b","c","d")
val newDf = inputDf.select("a","c")
val columnInfo= newDf.dtypes // should give you something like (("a","StringType"),("c","IntegarType"))
again this is not tested code but generally this is how you can get the column names and their types.