I have a table in BigQuery that is partitioned by ingestion time (_PARTITIONTIME), and I want to move the table to be stored in google storage but have the same structure and partitioned by _PARTITIONTIME, is there a way to do it?
I can proposed you a script that do that in one shot. maybe not optimized, but I tested and it works
DECLARE number_of_entry INT64;
DECLARE i INT64 default 0;
DECLARE current_entry INT64;
SET (number_of_entry) = (
Select count(*)
from `<projectID>.<dataset>.INFORMATION_SCHEMA.PARTITIONS`
where table_name='<Your partitioned table>'
);
LOOP
set i = i + 1;
if i > number_of_entry then
leave;
end if;
# Get the value of the current entry based on the rowid()
set current_entry = (
select entry
FROM (
select entry,row_number() over() as row
from (
select partition_id as entry
from `<projectID>.<dataset>.INFORMATION_SCHEMA.PARTITIONS`
where table_name='<Your partitioned table>
)
where row = i
);
EXPORT DATA OPTIONS(
uri='gs://<YourBucket>/export-splited/' || current_entry || '-*.csv',
format='CSV',
overwrite=true,
header=true,
field_delimiter=',') AS
select ......
from `<projectID>.<dataset>.<Your partitioned table>`
where _PARTITIONTIME=PARSE_DATE("%Y%m%d",current_entry);
END LOOP;
Related
Our IT team loads couple of tables every month. The new load should have more records than the previous load, with at least 2% more records.
It's a truncate and load process, I'm collecting the num of records from each table before the truncate, and I'm checking the difference in excel every month to make sure the data load is correct.
Is there anyway to automate this in Oracle.
eg:
Table_name Before_cnt After_cnt
XX_TEST1 4,606,619,326 4,983,759,822
XX_TEST2 121,973,005 123,161,581
You can apply the steps just like below :
SQL> create table XX_TEST1( id int primary key );
SQL> insert into XX_TEST1 select level from dual connect by level <= 100;
SQL> begin -- if table exists, then drop it!
for c in (select table_name from cat where table_name = 'XX_TEST1_OLD' )
loop
execute immediate 'drop table '||c.table_name;
end loop;
end;
/
SQL> create table XX_TEST1_old as select count(*) as cnt from XX_TEST1;
SQL> begin
execute immediate 'truncate table XX_TEST1';
end;
/
SQL> insert into XX_TEST1 select level from dual connect by level <= 103;
SQL> with xt1_new(cnt_new) as
(
select count(id) from XX_TEST1
)
select case when sign( (100 * ( cnt_new - cnt) / cnt)-2 ) = 1 then 1
else 0 end as "Rate Satisfaction"
from XX_TEST1_old
cross join xt1_new;
If this SELECT statement retuns 1, then we're successful to reach the target, else returns 0 and means we're unsuccessful.
Demo
I am stuck at a place.
There is a procedure that checks for something and inserts into an table type upon successful determination of that condition.
But i can insert only once in the table type. Is there a way to insert again and again into the table type.
PROCEDURE "hello"."helloWorld.db::sampleException" (OUT TRACE_RECORD "hello"."LogTrace" )
LANGUAGE SQLSCRIPT AS
BEGIN
DECLARE i int;
select count(*) into i from "hello"."REGION";
IF :i > 1 then
TRACE_RECORD = SELECT '1' AS "LogID", '1' AS "TraceID" FROM DUMMY;
end if;
IF :i > 2 then
TRACE_RECORD = SELECT '2' AS "LogID", '2' AS "TraceID" FROM DUMMY;
end if;
END;
What i get on executing the procedure is only the last record "2,2".
How can i insert both the records 1,1 and 2,2.
Note: I do not want to use Temporary Tables.
Any help on this..
Thanks.!
Editing the Question a bit:
-I have to use Table TYPE (till the time there is no optimal way better than it)
-I have to insert more than 20-30 records in the table type.
Do you have to write this as a procedure? A table-valued function seems more suitable:
CREATE FUNCTION f_tables4 (in_id INTEGER)
RETURNS TABLE (
"LogID" VARCHAR(400),
"TraceID" VARCHAR(400)
)
LANGUAGE SQLSCRIPT
AS
BEGIN
RETURN
SELECT t."LogID", t."TraceID"
FROM (
SELECT 1 AS i, '1' AS "LogID", '1' AS "TraceID" FROM DUMMY
UNION ALL
SELECT 2 AS i, '2' AS "LogID", '2' AS "TraceID" FROM DUMMY
) t
JOIN (SELECT count(*) AS cnt FROM "hello"."REGION") c
ON c.cnt > t.i
END
I have a table and I want to check the datetime format of records according to YYYYMMDD,HH24MISS. If the datetime format of my records is incorrect, write an error message. How can I make a function or procedure in PL/SQL?
You can write a function like this:
CREATE OR REPLACE FUNCTION CheckDateString(str IN VARCHAR2) RETURN DATE IS
BEGIN
RETURN TO_DATE(str,'YYYYMMDD,HH24MISS');
EXCEPTION
WHEN OTHERS THEN
RETURN NULL;
END CheckDateString;
and use it like this
SELECT *
FROM my_table
WHERE CheckDateString(DATE_STRING) IS NULL
AND DATE_STRING IS NOT NULL;
Of course the next action point would be to correct the wrong values and change the data type of this column to DATE, resp. TIMESTAMP.
In case your column is VARCHAR2 and you need to check that the values inside it can be transformed to date using your desired format, this could be a solution:
declare
v_foo_date date;
begin
for r_date in (
select date_col from my_table
) loop
begin
v_foo_date := to_date(r_date.date_col, 'YYYYMMDD,HH24MİSS');
exception when others then
dbms_output.put_line('error in validating value ' || r_date.date_col);
end;
end loop;
end;
If you would like to get the data which has the desired format, maybe you could do something like this:
Otherwise I would go with Wernfried Domscheit's answer.
Cast the date as datetime from your table
select *, cast(yourDateColumn as datetime) as verifiedDate
into #tempTable
from yourTable
Then check the data from yourTable against the data you have in #tempTable
select * from yourTable yt
inner join #tempTable tt on
yt.yourDateColum = tt.verifiedDate and yt.ID = tt.ID
I got a problem I not sure how to solve it so far...
I have two tables that are related to each other with a 1 x n relation. I will try to describe the more importants fields below:
Table One - company:id PK,companyname varchar;
Table Two - training: course varchar,companyid bigint FK,id PK;
The problem is: I would like to update the information on course field of the table training because there are many courses with the same name. My idea is use something like
for s in 1..n loop
update training set course = course || s;
end loop;
No need for a loop you can do this with plain SQL:
with numbered as (
select id,
row_number() over (order by id) as rn
from training
)
update training
set course = course||n.rn::text
from numbered n
where n.id = training.id;
The common table expression assigns a number for each row in the training table and that number is then used to generate the new course name.
UPDATE training
SET course = course || num
FROM (
SELECT generate_series(1, (
SELECT count(course)
FROM training
)) num
) t
I solved mine doubt creating this function below:
CREATE OR REPLACE FUNCTION changeName()
RETURNS VOID AS
$$
DECLARE
table1id table1.id%TYPE;
counter1 RECORD;
counter2 RECORD;
BEGIN
FOR counter1 IN SELECT repeated_column,foreign_keyid,COUNT(repeated_column) AS contagem
FROM table1 GROUP BY repeated_column,foreign_keyid HAVING COUNT(repeated_column) > 1 LOOP
FOR counter2 IN 1..counter1.contagem LOOP
SELECT id INTO table1id FROM table1 WHERE repeated_column IN(counter1.repeated_column) AND foreign_keyid = counter1.foreign_keyid;
UPDATE table1 SET repeated_column = repeated_column || ' (' || counter2 || ')' WHERE id = table1id;
END LOOP;
END LOOP;
END;
$$
LANGUAGE 'plpgsql'
here is the problem.
I read year numbers from a table and insert them in another one (now static but in the end temp table)
Then I look over those numbers and create tables with them.
BEGIN
INSERT INTO temp_year
( year_column )
(
select extract(year from datum) from datetest
);
FOR counter_id IN ( SELECT * FROM temp_year )
LOOP
EXECUTE IMMEDIATE 'Create table YEAR_' || counter_id || ' (year int, name char(50))'
END LOOP;
END;
/
Temp_year has a column year_column (int) (filled like 2012)
datatest has a date colum with values like 10.02.2012
The result should be a table named YEAR_2012 with columns year int and name char(50)
However this is not working. I quits at the execute immediate part, even when there are years in the temp_year table.
Any ideas??
Thanks in advance.
THeVagabond
Try this one:
EXECUTE IMMEDIATE 'Create table YEAR_' || counter_id.year_column || ' (year integer, name varchar2(50))';
(Do not miss the semicolon at the end)