I have created a HsqlDB database and inserted a lot of data in it already. The tables were create in default mode, i.e., the table type is MEMORY.
Not I want to switch those tables to CACHED mode since they are getting larger and larger. The HsqlDB 1.8 guide says the cached tables can be created by using command
CREATE CACHED TABEL ...
Is there any way to alter the existing tables into CACHED without losing any data?
For example, when your table name is mytable and you want to change it to cached.
With version 2.x there is a command for this:
SET TABLE mytable TYPE CACHED
With version 1.8.x you need to shutdown the database first. Then edit the .script file and modify the line for the table
CREATE MEMORY TABLE mytable to CREATE CACHED TABLE mytable
After saving the .script file, open your database as normal.
Related
I'm loading data from a named external stage (S3) by using COPY INTO, and this S3 bucket keep all old files.
Here's what I want:
When a new file comes in, truncate the table and load the new file only, if there's no new file coming in, just keep the old data without truncation.
I understand that I can put option like FORCE = False to avoid loading old files again, but how do I only truncate the table when there's new file coming in?
I would likely do this a bit differently, since there isn't a way to truncate/delete records in the target table from the COPY command. This will be a multi-step process, but can be automated via Snowflake:
Create a transient table. For sake of description, I'll just call this STG_TABLE. You will also maintain your existing target table called TABLE.
Modify your COPY command to load to STG_TABLE.
Create a STREAM called STR_STG_TABLE over STG_TABLE.
Create a TASK called TSK_TABLE with the following statement
This statement will execute only if your COPY command actually loaded any new data.
CREATE OR REPLACE TASK TSG_TABLE
WAREHOUSE = warehouse_name
WHEN SYSTEM$STREAM_HAS_DATA('STR_STG_TABLE')
AS
INSERT OVERWRITE INTO TABLE (fields)
SELECT fields FROM STR_STG_TABLE;
The other benefit of using this method is that your transient table will have the full history of your files, which can be nice for debugging issues.
I have a instance of Crate 1.0.2 and I dropped a table from it. Then re-created table with same name and slightly modified schema. Then I imported data using copy from command. File argument to copy from command consists of 10,000 records and copy from command runs ok. When I check table tab in crate web console, it shows many partitions added and each partition having few records. If I add number of records column on this tab, it comes close to 10k but when I fire a command "select count(*) from mytable", it returns around 8000 records only. On further investigation found that there are certain partitions on which data cannot be queried at all. Has any one seen this problem? Does it have anything to do with table drop and creation with same name ? I also observed that when a table is dropped, not all files related to that table are deleted from path.data. Are these directories a reason for those partitions become non-query able? While importing, I saw "Document already exists" exception. I know my data does not have any duplicate value for primary column.
Some questions to clarify the issue:
Have you run refresh table mytable after your copy command has finished?
Are you sure that with the new schema of the table, there are no duplicate records?
Since 1.x versions are not supported anymore, could you try with CrateDB 2.1.6 which is the current stable version to see if the problem persists?
I want to load mysqldump file into a server.
while loading dump , I want to change few column values and update schema.
for example for guid column we gave varchar(100) so now I want to change into binary(16) that means I need change in table schema and table values.
can I do this changes while loading dump file into new server.
Thanks
No, basically you can't do anything WHILE loading dump. As mentioned in comments, you have two options:
Edit SQL in dump
Load dump and after that execute a script
with needed fixes.
If you have access to initial database, you can produce another dump with needed changes.
When I load a (csv)-file to a hive table I can load without overwriting, thus adding the new file to the table.
Internally the file is just copied to the correct folder in HDFS
(e.g. user/warehouse/dbname/tablName/datafile1.csv). And probably some metadata is updated.
After a few loads I want to remove the contents of a specific file from the table.
I am sure I cannot simply delete the file because of the metadata that needs to be adjusted as well. There must be some kind of build-in function for this.
How do I do that?
Why do you need that?I mean Hive was developed to serve as a warehouse where you put lots n lots n lots of data and not to delete data every now and then. Such a need seems to be a poorly thought out schema or a poor use of Hive, at least to me.
And if you really have these kind of needs why don't you create partitioned tables? If you need to delete some specific data just delete that particular partition using either TRUNCATE or ALTER.
TRUNCATE TABLE table_name [PARTITION partition_spec];
ALTER TABLE table_name DROP [IF EXISTS] PARTITION partition_spec, PARTITION partition_spec,...
if this feature is needed more than just once in a while you can use MapR's distribution while allows this kind of operations with no problem (even via NFS). otherwise, if you don't have partition I think you'll have to create and new table using CTAS filterring the data in the bad file or just copy the good files back to os with "hadoop fs -copyToLocal" and move them back to hdfs into new table
I want to save some temporary data in memory, which should be removed after server shuts down.
There is a temporary table in HSQLDB, but the data is removed immediately after transaction committed, which is too short for me. On the other side, the memory table keeps a script log file and resume the data when server new starts. It takes time and place to maintain such script logs, which are useless for my situation.
what I need is just a type of table, only the table structure is persistent in hard disk, the data and the data operations should only be performed in memory. Otherwise why do I need a in-memory DB instead of mysql?
Is there such type of table in HSQLDB?
thanks
Create the file: database, then create the tables. Perform SHUTDOWN. Edit the .properties file for the database, add the setting below and save.
files_readonly=true
When you perform your tests with this database, no data is written to disk.
Alternatively, with the latest versions of HSQLDB 2.2.x, you can can specify this property on the connection URL during the tests. For example
jdbc:hsqldb:file:myfilepath;files_readonly=true