Spark SQL CTE ignoring namespace in query - sql

Question

Running Spark locally spark-sql or through pyspark spark.sql(...), if I use a CTE in a query and then reference the CTE with an incorrect namespace / database, the query works just fine (unexpected). When I run the query in production (on Databricks), I get a Table or view not found error (expected).
Unexpected pass behavior can be reproduced locally via spark-sql:
WITH myview AS (
SELECT 1 AS column
)
SELECT
*
FROM
invalid_namespace.myview;
Which returns "1" when I expect it to fail.
Can someone help me make this fail locally so that we can properly test before we deploy?
Exact steps to reproduce from a terminal:
$ spark-sql
...
spark-sql> WITH some_new_cte AS (SELECT 1 AS column)
> SELECT * FROM namespace_does_not_exist.some_new_cte;
...
1
Time taken: 2.294 seconds, Fetched 1 row(s)
spark-sql>

Answer 1

If you look at the query plan, it actually failed to parse
== Parsed Logical Plan ==
CTE [myview]
: +- SubqueryAlias `myview`
: +- Project [1 AS column#0]
: +- OneRowRelation
+- 'Project [*]
+- 'UnresolvedRelation `invalid_namespace`.`myview`
== Analyzed Logical Plan ==
column: int
Project [column#0]
+- SubqueryAlias `myview`
+- Project [1 AS column#0]
+- OneRowRelation
== Optimized Logical Plan ==
Project [1 AS column#0]
+- OneRowRelation
== Physical Plan ==
*(1) Project [1 AS column#0]
+- Scan OneRowRelation[]
The reason your query returned "1" was because spark saw that your view was in the same query, so it just ignored your namespace. If the namespace really doesn't exist, it will fail.

Answer 2

This seems to be a bug that affects versions of Spark 2.4.0 through 2.4.5 (I did not check <2.4.0). It appears to have been fixed in 2.4.6 and continues to work as expected in 3.0.0.
Unexpected success in 2.4.0:
$ spark-submit --version
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.0
/_/
Using Scala version 2.11.12, OpenJDK 64-Bit Server VM, 1.8.0_262
Branch
Compiled by user on 2018-10-29T06:22:05Z
Revision
Url
Type --help for more information.
$ echo "WITH mycte AS (SELECT 1 AS column) SELECT * FROM bad_namespace.mycte;" | SPARK_CONF_DIR=spark_conf spark-sql
20/07/17 10:59:05 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark master: local[*], Application Id: local-1595008748636
spark-sql> WITH mycte AS (SELECT 1 AS column) SELECT * FROM bad_namespace.mycte;
1
Time taken: 1.822 seconds, Fetched 1 row(s)
spark-sql>
Tested but unshown: 2.4.1, 2.4.2, 2.4.3, 2.4.4 (all succeed unexpectedly)
Unexpected success in 2.4.5:
$ spark-submit --version
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.5
/_/
Using Scala version 2.11.12, OpenJDK 64-Bit Server VM, 1.8.0_262
Branch HEAD
Compiled by user centos on 2020-02-02T19:38:06Z
Revision cee4ecbb16917fa85f02c635925e2687400aa56b
Url https://gitbox.apache.org/repos/asf/spark.git
Type --help for more information.
$ echo "WITH mycte AS (SELECT 1 AS column) SELECT * FROM bad_namespace.mycte;" | SPARK_CONF_DIR=spark_conf spark-sql
20/07/17 10:59:55 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark master: local[*], Application Id: local-1595008798737
spark-sql> WITH mycte AS (SELECT 1 AS column) SELECT * FROM bad_namespace.mycte;
1
Time taken: 2.155 seconds, Fetched 1 row(s)
spark-sql>
Expected failure in 2.4.6:
$ spark-submit --version
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.6
/_/
Using Scala version 2.11.12, OpenJDK 64-Bit Server VM, 1.8.0_262
Branch HEAD
Compiled by user holden on 2020-05-29T23:47:51Z
Revision 807e0a484d1de767d1f02bd8a622da6450bdf940
Url https://gitbox.apache.org/repos/asf/spark.git
Type --help for more information.
$ echo "WITH mycte AS (SELECT 1 AS column) SELECT * FROM bad_namespace.mycte;" | SPARK_CONF_DIR=spark_conf spark-sql
20/07/17 11:00:40 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark master: local[*], Application Id: local-1595008843321
spark-sql> WITH mycte AS (SELECT 1 AS column) SELECT * FROM bad_namespace.mycte;
20/07/17 11:00:44 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
20/07/17 11:00:44 WARN ObjectStore: Failed to get database bad_namespace, returning NoSuchObjectException
Error in query: Table or view not found: `bad_namespace`.`mycte`; line 1 pos 49;
'Project [*]
+- 'UnresolvedRelation `bad_namespace`.`mycte`
spark-sql>
Expected failure in 3.0.0:
$ spark-submit --version
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.0.0
/_/
Using Scala version 2.12.10, OpenJDK 64-Bit Server VM, 1.8.0_262
Branch HEAD
Compiled by user ubuntu on 2020-06-06T11:32:25Z
Revision 3fdfce3120f307147244e5eaf46d61419a723d50
Url https://gitbox.apache.org/repos/asf/spark.git
Type --help for more information.
$ echo "WITH mycte AS (SELECT 1 AS column) SELECT * FROM bad_namespace.mycte;" | SPARK_CONF_DIR=spark_conf spark-sql
20/07/17 11:01:40 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/07/17 11:01:42 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
20/07/17 11:01:42 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
20/07/17 11:01:44 ERROR ObjectStore: Version information found in metastore differs 1.2.0 from expected schema version 2.3.0. Schema verififcation is disabled hive.metastore.schema.verification
20/07/17 11:01:44 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore georgeleslie-waksman#10.0.1.178
Spark master: local[*], Application Id: local-1595008901594
spark-sql> WITH mycte AS (SELECT 1 AS column) SELECT * FROM bad_namespace.mycte;
20/07/17 11:01:45 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
20/07/17 11:01:45 WARN ObjectStore: Failed to get database bad_namespace, returning NoSuchObjectException
Error in query: Table or view not found: bad_namespace.mycte; line 1 pos 49;
'Project [*]
+- 'UnresolvedRelation [bad_namespace, mycte]
spark-sql>