probably a very trivial question but I'm not sure about this and also don't want to lose the table. How do I rename a table in Athena?
Database name - friends
table name - centralPark
desired table name -centralPerk
you can't!
see the list of unsupported DDL in Athena.
what you can do is to make a new table using select:
CREATE TABLE centralPark
AS SELECT * FROM centralPerk
WITH DATA
and drop the old table:
DROP TABLE IF EXISTS centralPerk
Using a CTAS query is effective, but I found it to be quite slow. It needs to copy all the files.
But you don't need to copy the files. You can create a new table directly in the Glue catalog and point it at the existing files. This works in seconds or less.
If you're using Python, I highly recommend the awswrangler library for this kind of work.
import awswrangler as wr
def wrangler_copy(db, old_name, new_name):
wr.catalog.create_parquet_table(
db,
new_name,
path=wr.catalog.get_table_location(db, old_name),
columns_types=wr.catalog.get_table_types(db, old_name),
# TODO: partitions, etc
)
And then drop the old table if you like.
DROP TABLE IF EXISTS <old_name>
Related
I am trying to rename a database in azure databricks but I am getting the following error:
no viable alternative at input 'ALTER DATABASE inventory
Below is code:
%sql
use inventory;
ALTER DATABASE inventory MODIFY NAME = new_inventory;
Please explain what is meant by this error "no viable alternative at input 'ALTER DATABASE inventory"
and how can I solve it
It's not possible to rename database on Databricks. If you go to the documentation, then you will see that you can only set DBPROPERTIES.
If you really need to rename database, then you have 2 choices:
if you have unmanaged tables (not created via saveAsTable, etc.), then you can produce SQL using SHOW CREATE TABLE, drop your database (be careful anyway), and recreate all tables from saved SQL
if you have managed tables, then the solution would be to create new database, and either use CLONE (only for Delta tables), or CREATE TABLE ... AS SELECT for other file types, and after that drop your database
Alex Ott's answer, to use Clone, is OK if you do not need to maintain the versioning history of your database when you rename it.
However if you wish to time travel on the database of Delta tables after the renaming, this solution works:
Create your new database, specifying its location
Move the file system from the old location to the new location
For each table on the old database, create a table on the new database, based on the location (my code relies on the standard file structure of {database name}/{table name} being observed). No need to specify schema as it's just taken from the files in place
Drop old database
You will then be left with a database with your new name, that has all of the data and all of the history of your old database, i.e. a renamed database of Delta tables.
Pyspark method (on databricks, with "spark" and "dbutils" already defined by default) :
def rename_db(original_db_name, original_db_location, new_db_name, new_db_location):
spark.sql(f"create database if not exists {new_db_name} location '{new_db_location}'")
dbutils.fs.mv(original_db_location,new_db_location,True)
for table in list(map(lambda x: x.tableName, spark.sql(f"SHOW TABLES FROM {original_db_name}").select("tableName").collect())):
spark.sql(f"create table {new_db_name}.{table} location '{new_db_location}/{table}'")
spark.sql(f"drop database {original_db_name} cascade")
return spark.sql(f"SHOW TABLES FROM {new_db_name}")
Is there a way to create the contents of the BigQuery schema into a Table?
I don't want to create a table from the schema instead I want to move the contents of the schema into a BigQuery Table. I couldn't find any trivial method to do this.
Currently I download the table schema as JSON and then create a new table from it
This worked for me!
SELECT *
FROM
mydataset.INFORMATION_SCHEMA.COLUMN_FIELD_PATHS
When creating a table in SQL Server, it is created using dbo.<tableName> format.
I want to change dbo.tableName to source.tableName, as I want to import data into a source table and then cook that data.
Thanks
You are talking about schemas. If the schema source doesn't exist yet, you need to run create schema source. Once the schema exists it's as easy as create table source.tableName (...).
I am moving data around within Impala, not my design, and I have lost some data. I need to copy the data from the parquet tables back to their original non-parquet tables. Originally, the developers had done this with a simple one liner in a script. Since I don't know anything about databases and especially about Impala I was hoping you could help me out. This is the one line that is used to translate to a parquet table that I need to be reversed.
impalaShell -i <ipaddr> use db INVALIDATE METADATA <text_table>;
CREATE TABLE <parquet_table> LIKE <text_table> STORED AS PARQUET TABLE;
INSERT OVERWRITE <parquet_table> SELECT * FROM <text_table>;
Thanks.
Have you tried simply doing
CREATE TABLE <text_table>
AS
SELECT *
FROM <parquet_table>
Per the Cloudera documentation, this should be possible.
NOTE: Ensure that your does not exist or use a table name that does not already exist so that you do not accidentally overwrite other data.
I have default db in hive table which contains 80 tables .
I have created one more database and I want to copy all the tables from default DB to new Databases.
Is there any way I can copy from One DB to Other DB, without creating individual table.
Please let me know if any solution..
Thanks in advance
I can think of couple of options.
Use CTAS.
CREATE TABLE NEWDB.NEW_TABLE1 AS select * from OLDDB.OLD_TABLE1;
CREATE TABLE NEWDB.NEW_TABLE2 AS select * from OLDDB.OLD_TABLE2;
...
Use IMPORT feature of Hive
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport
Hope this helps.
create external table new_db.table like old_db.table location '(path of file in hdfs file)';
if you have partition in table then you have to add partition in new_db.table.
These are probably the fastest and simplest way to copy / move tables from one db to other.
To move table source
Since 0.14, you can use following statement to move table from one database to another in the same metastore:
alter table old_database.table_a rename to new_database.table_a;
The above statements will also move the table data on hdfs if table_a is a managed table.
To copy table
You can always use CREATE TABLE <new_db>.<new_table> AS SELECT * FROM <old_db>.<old_table>; statements. But I believe this alternate method of copying database using hdfs dfs -cp and then creating tables with LIKE can be a little faster if your tables are huge:
hdfs dfs -cp /user/hive/warehouse/<old_database>.db /user/hive/warehouse/<new_database>.db
And then in Hive:
CREATE DATABASE <new_database>;
CREATE TABLE <new_database>.<new_table> LIKE <old_database>.<old_table>;
You can approach one of the following option :
The syntax looks something like this:
EXPORT TABLE table_or_partition TO hdfs_path;
IMPORT [[EXTERNAL] TABLE table_or_partition] FROM hdfs_path [LOCATION [table_location]];
Some sample statements would look like:
EXPORT TABLE TO 'location in hdfs';
Use test_db;
IMPORT FROM 'location in hdfs';
Export Import can be appled on a partition basis as well:
EXPORT TABLE PARTITION (loc="USA") to 'location in hdfs';
The below import commands imports to an external table instead of a managed one
IMPORT EXTERNAL TABLE FROM 'location in hdfs' LOCATION ‘/location/of/external/table’;