I am trying to identify all the tables that contain a certain column (for example, “thenameoffield”) using Hue
In the Hue User Guide under Data Catalogs, it suggests entering type:field name in the search bar, but when I do this there are no results found. There are no clear examples in the documentation, but I assume "name" is the name of the field I want to search for. The documentation does not state if wildcard can be used, such as "name". I did try this but no results were produced.
I also tried running SQL queries, but based on the error messages, Hue does not accept the syntax (assuming it's correct to start with):
I. Entering the syntax above into the Hue query area:
type:field(column): ‘thenameoffield’ AND
type:field 'thenameoffield'
with and without quotes, but it doesn’t work. “AnalysisException: Syntax error in line2:undefined: type:field(column): /thenameoffield’”
II. I also tried:
SELECT * FROM INFORMATION_SCHEMA.COLUMNS;
but not working either- “AnalysisException: Syntax error in line2:undefined: EXPLAIN SELECT * FROM,…Exception: Syntax error”
III. I also attempted to use:
SELECT
sys.columns.name AS ColumnName,
tables.name AS TableName
FROM
sys.columns
JOIN sys.tables ON
sys.columns.object_id = tables.object_id
WHERE
sys.columns.name LIKE ‘%thenameoffield%’
but received the error message: AnalysisException: Syntax error in line 1: undefined: SELECT sys.columns.name as ColumnName,…
Any help would be greatly appreciated.
The Catalog search is currently powered by either Apache Atlas or Cloudera Navigator so you would need to point Hue to those to make it work.
About the DB metadata querying, you might have more chance by tagging the question with Apache Hive instead of Hue (as Hue just sends the query to Hive). Would also help to have the Hive verion.
Related
Using Hive 2.3.7 on AWS EMR (5.33.1) I have created a database which shows correctly when calling show databases;. I then create a table which seems to work correctly (no exceptions). When I call describe <table>; It correctly returns the name and schema of the table. However when I run show tables; the following error is returned:
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask.MetaException(message:Got exception: org.apache.hadoop.hive.metastore.api.MetaException
Exception thrown when executing query :
SELECT A0.TBL_NAME,A0.TBL_NAME AS NUCORDER0 FROM TBLS A0 LEFT OUTER JOIN DBS B0 ON
A0.DB_ID = B0.DB_ID WHERE B0.`NAME` = ? AND LOWER(A0.TBL_NAME) LIKE '_%' ESCAPE '\' ORDER BY NUCORDER0)
If anyone can shed any light on this issue it would be really appreciated.
I have googled around and found nothing of any use.
EDIT: show tables in <schema>; returned the same result
EDIT 2: This issue was solved by updating the EMR to emr-6.4.0. I have no great insight into the issue beyond what is mentioned here.
I think your metadata database has been corrupted/has bad data. I would take a backup. And then see if you can restore some previous backups. I would connect to the database directly and look at the those tables and see if anything looks out of the ordinary. If you find a bad table entry don't delete it. I'd try using "Delete table" commands (via hive) to remove it to keep integrity. If you have to you can delete entries in your database, you have a backup and could restore back the tables.
Hive meta store is using datanucleus, https://www.datanucleus.org/, for all CRUD of metastore database. It's generating \\ to escape backslash itself, but Mariadb interprete \\ as string literal. So it needs to use \\ as escape character.
You can see sql_mode setting here, https://mariadb.com/kb/en/sql-mode/#sql_mode-values.
Get rid of NO_BACKSLASH_ESCAPE from the mode and it should be all right.
Try providing the schema which you want to see the tables:
show tables in schema_name;
I'm writing a Java application to automatically build and run SQL queries. For many tables my code works fine but on a certain table it gets stuck by throwing the following exception:
Exception in thread "main" org.postgresql.util.PSQLException: ERROR: column "continent" does not exist
Hint: Perhaps you meant to reference the column "countries.Continent".
Position: 8
The query that has been run is the following:
SELECT Continent
FROM network.countries
WHERE Continent IS NOT NULL
AND Continent <> ''
LIMIT 5
This essentially returns 5 non-empty values from the column.
I don't understand why I'm getting the "column does not exist" error when it clearly does in pgAdmin 4. I can see that there is a schema with the name Network which contains the table countries and that table has a column called Continent just as expected.
Since all column, schema and table names are retrieved by the application itself I don't think there has been a spelling or semantical error so why does PostgreSQL cause problems regardless? Running the query in pgAdmin4 nor using the suggested countries.Continent is working.
My PostgreSQL version is the newest as of now:
$ psql --version
psql (PostgreSQL) 9.6.1
How can I successfully run the query?
Try to take it into double quotes - like "Continent" in the query:
SELECT "Continent"
FROM network.countries
...
In working with SQLAlchemy environment, i have got this error with the SQL like this,
db.session.execute(
text('SELECT name,type,ST_Area(geom) FROM buildings WHERE type == "plaza" '))
ERROR: column "plaza" does not exist
Well, i changed == by = , Error still persists, then i interchanged the quotes, like follows. It worked. Weird!
....
text("SELECT name,type,ST_Area(geom) FROM buildings WHERE type = 'plaza' "))
This problem occurs in postgres because the table name is not tablename instead it is "tablename".
for eg.
If it shows user as table name,
than table name is "user".
See this:
Such an error can appear when you add a space in the name of a column by mistake (for example "users ").
QUICK FIX (TRICK)
If you have recently added a field which you have already deleted before and now trying to add the same field back then let me share you this simple trick! i did this and the problem was gone!!
so, now just delete the migration folder entirely on the app,then instead of adding that field you need to now add a field but with the name of which you have never declared on this app before, example if you are trying to add title field then create it by the name of heading and now do the migration process separately on the app and runserver, now go to admin page and look for that model and delete all the objects and come to models back and rename the field that you recently made and name it to which you were wishing it with earlier and do the migrations again and now your problem must have been gone!!
this occurs when the objects are there in the db but you added a field which wasn't there when the earlier objs were made, so by this we can delete those objs and make fresh ones again!
I got the same error when I do PIVOT in RedShift.
My code is similar to
SELECT *
INTO output_table
FROM (
SELECT name, year_month, sales
FROM input_table
)
PIVOT
(
SUM(sales)
FOR year_month IN ('nov_2020', 'dec_2020', 'jan_2021', 'feb_2021', 'mar_2021', 'apr_2021', 'may_2021', 'jun_2021', 'jul_2021', 'aug_2021',
'sep_2021', 'oct_2021', 'nov_2021', 'dec_2021', 'jan_2022', 'feb_2022', 'mar_2022', 'apr_2022', 'may_2022', 'jun_2022',
'jul_2022', 'aug_2022', 'sep_2022', 'oct_2022', 'nov_2022')
)
I tried year_month without any quote (got the error), year_month with double quote (got the error), and finally year_month with single quote (it works this time). This may help if someone in the same situation like my example.
I am trying following query in MS Access 2010 database:
Query:
SELECT ID, Title, Priority, Workflow_Instance_Step_ID:ID
FROM Task
Error:
Error Source: Microsoft Office Access Database Engine
Error Message: Syntax error (missing operator) in query expression 'Workflow_Instance_Step_ID:ID'
I know that field "Workflow_Instance_Step_ID:ID" is giving error as it has ':' operator, but I cant change it as it it coming from share point list.
I have tried '[Workflow_Instance_Step_ID:ID]' but still its giving an error.
Please give me your suggestion on the same.
I am unable to recreate your issue, at least with a native Access table in Access 2010. I created a table named [baz] with a single text field named [foo:bar]
foo:bar
-------
this
is
a
test
and the query
SELECT [foo:bar] FROM baz;
works fine, so a field name containing a colon character does not seem to cause problems for Access SQL as such.
An ODBC linked table to a SQL Server table with the same structure does not cause problems either.
In both cases the queries were run from within Access 2010 itself.
So we might be in diagnosis mode, so lets try
SELECT * FROM Task
and then look at the FieldNames that come back.
I'm attempting to write an extension for SQL Developer to better support Postgres. These are just XML configuration files with SQL snippets in them. To display the values for a postgres sequence, I need to run a simple query of the following form:
select * from schema.sequence
The trouble with this is that the Oracle SQL Developer environment provides the correct schema and node (sequence) name as bind variables. This would mean that I should format the query as:
select * from :SCHEMA.:NAME
The trouble with this is that bind variables are only valid in the select clause or the where clause (as far as I'm aware), and using this form of the query returns a "syntax error at or near "$1" error message.
Is there a way to return the values in the sequence object without directly selecting them from the sequence? Perhaps some obtuse joined statement from pg_catalog tables?
Try this:
select *
from information_schema.sequences
where sequence_name = :name
and sequence_schema = :schema;
It's not exactly the same thing as doing a select from the sequence, but the basic information is there.
If you've used Oracle, you've probably gotten the helpful message "ORA-00942: Table or view does not exist". Is there a legitimate technical reason the message doesn't include the name of the missing object?
Arguments about this being due to security sound like they were crafted by the TSA. If I'm an attacker, I'd know what table I just attempted to exploit, and be able to interpret this unhelpful message easily. If I'm a developer working with a complex join through several layers of application code, it's often very difficult to tell.
My guess is that when this error was originally implemented, someone neglected to add the object name, and now, people are afraid it will break compatibility to fix it. (Code doing silly things like parsing the error message will be confused if it changes.)
Is there a developer-friendly (as opposed to recruiting your DBA) way to determine the name of the missing table?
Although I've accepted an answer which is relevant to the topic, it doesn't really answer my question: Why isn't the name part of the error message? If anyone can come up with the real answer, I'll be happy to change my vote.
You can set an EVENT in your parameter file (plain text or spfile) to force Oracle to dump a detailed trace file in the user_dump_dest, the object name might be in there, if not the SQL should be.
EVENT="942 trace name errorstack level 12"
If you are using a plain text file you need to keep all your EVENT settings on consecutive lines. Not sure how that applied to spfile.
SQL*Plus does tell you the table that doesn't exist. For example:
SQL> select
2 *
3 from
4 user_tables a,
5 non_existent_table b
6 where
7 a.table_name = b.table_name;
non_existent_table b
*
ERROR at line 5:
ORA-00942: table or view does not exist
Here it shows that the name of the missing table and the line number in the SQL statement where the error occurs.
Similarly, in a one-line SQL statement you can see the asterisk highlighting the name of the unknown table:
SQL> select * from user_tables a, non_existent_table b where a.table_name = b.table_name;
select * from user_tables a, non_existent_table b where a.table_name = b.table_name
*
ERROR at line 1:
ORA-00942: table or view does not exist
In terms of your question, I guess the reason the error message doesn't include the name of the table is that the error message itself needs to be static text. The line number and location in the line of the error is clearly passed back to SQL*Plus (somehow).
I would disagree with the opinion, that SQL+ lets you understand which table name is unacceptable. True, it helps in direct DML, although parsing it is very hard. But when it comes to dynamic, we get no help:
SQL> begin
2 execute immediate 'insert into blabla values(1)';
3 end;
4 /
begin
*
ERROR at line 1:
ORA-00942: table or view does not exist
ORA-06512: at line 2
If you are using a SQL browsing tool like TOAD or TORA it will help you with ORA errors by highlightling or pointing moving the cursor to where you made your error.
Copy and paste your SQL in to one of these tools to help. You may also find the analyse info available useful too.
If its not a huge statement, then the easiest way is just to check the data dictionary,
SQL> select * from xx,abc;
select * from xx,abc
*
ERROR at line 1:
ORA-00942: table or view does not exist
SQL> select owner,table_name from all_tables where table_name in ('XX','ABC');
OWNER TABLE_NAME
------------------------------ ------------------------------
MWATSON XX
SQL>
This isn't ideal, but short of going and examining trace files, I'm not sure how else to do it.
I've never had a problem with interpreting Oracle error messages. Part of the reason is that every interactive tool I've seen for developing SQL for Oracle helpfully points to the location the query went wrong. That includes SQL*Plus, as others have noted, and the Perl DBI module:
$ exec_sql.pl 'select * from daul'
DBD::Oracle::db prepare failed: ORA-00942: table or view does not exist (DBD ERROR: error possibly near <*> indicator at char 14 in 'select * from <*>daul') [for Statement "select * from daul"] at exec_sql.pl line 68.
Well, that is a bit hard to read since it's all squished on one line. But a GUI tool would be able to point to the token where Oracle started having problems with the query. And given a bit of work on a parser, you could write a tool to pick out the offending table.
To answer the underlying question, Oracle errors don't seem to be designed to work the way you expect. As far as I can tell, none of the the error messages in Oracle support variable text. Instead, Oracle returns two bits of information: an error number and a location where the error occurs. If you have proper tools, it's pretty easy to diagnose an error with those pieces of data. It can be argued that Oracle's system is nicer to tool creators than one which provides variable amounts of diagnostic data depending on the error. Imagine having to write a custom parser for all of Oracle's error messages (including future errors) to highlight the offending location.
Sometimes including the table name would be misleading. Just knowing where things went wrong can be a huge help:
SQL> select * from where dummy = 'X';
select * from where dummy = 'X'
*
ERROR at line 1:
ORA-00903: invalid table name
As for why Oracle chose to do thing this way, I have some speculations:
IBM used this style of error message for System R, which Larry Ellison, Bob Miner and Ed Oates copied to build Oracle V2. (Backward compatibility.)
Error number and location are the smallest possible representation of diagnostic information. (Parsimony.)
As I indicated above, to simplify the creation of tools that connect to Oracle. (Interoperability.)
In any case, I don't think you need to be a DBA to figure out which table doesn't exist. You just need to use the proper tools. (And adjust your expectations, I suppose.)
Reason 1: Multi-lingual interface
There is a language-specific message configuration file for your database instance. Messages are pulled out of there and translated from the pure numeric version to the numeric+text version.
It was probably considered better to have the hardcoded strings, than to run the risk at runtime of having a mysterious failure due to an improperly formatted "%s" string.
(Not that I particularly agree with this POV, btw.)
Reason 2: Security
Right now you don't particularly expose the internal workings of your application if you print a PHP, etc, dump of an Oracle error message to the browser.
Applications would be a bit more exposed if more detail were printed by default... For example, if citibank printed a more explanatory message.
(see disclaimer above, I would be happy to get more information in the error as well.)
#Matthew
Your query's a start, but it might not work when you have multiple schemas. For example, if I log into our instance as myself, I have read access to all our tables. But if I don't qualify the table name with the schema I'll get an ORA-00942 for tables without synonyms:
SQL> select * from tools;
select * from tools
*
ERROR at line 1:
ORA-00942: table or view does not exist
The table still shows up in all_tables though:
SQL> select owner, table_name from all_tables where table_name = 'TOOLS';
OWNER TABLE_NAME
------------------------------ ------------------------------
APPLICATION TOOLS
#erikson
Sorry that doesn't help much. I'm with Mark - I used TOAD.