Is there an option for bq load to specify datetime format to parse? I'm getting an error when using bq load due to a datetime with milliseconds in it.
Sample file below:
ID|Card|Status|ExpiryDate|IssuedDate
1105|9902|Expired|2015-12-31 00:00:00|2014-07-04 14:43:41.963000000
Command used below:
bq load --source_format=CSV --skip_leading_rows 1 --field_delimiter "|" --replace mytable $GSPATH
It is not possible to control/change date or datetime formatting when loading data into BigQuery.
As a solution, I would try to load the datetime field as a STRING and then try to use the PARSE_DATETIME function or something else to postprocess and convert the string to datetime.
An example of the code to parse the string to datetime:
select PARSE_DATETIME('%Y-%m-%d %H:%M:%E*S','2014-07-04 14:43:41.963000000');
Related
I try to change a columns' data type from string to DATETIME (for example '04/12/2016 02:47:30') with the format 'YY/MM/DD HH24:MI:SS' but it shoes an error like :
Failed to parse input timestamp string at 8 with format element ' '
The initial file was a csv which i uploaded from my drive. I tried to convert the column's data type from google sheets and then re'upload it, but the column type still remains as string.
I think when you load your CSV file to the BigQuery table, you use autodetect mode.
Unfortunately with this mode, BigQuery will consider your date as String even if you changed it from Google Sheet.
Instead of using autodetect, I propose you using a Json schema for your BigQuery table.
In your schema you will indicate that the column type for your date field is timestamp.
The format you indicated 04/12/2016 02:47:30 is compatible with a timestamp and BigQuery will convert it for you.
For the loading file to BigQuery, you can directly use the console or gcloud cli with bq command :
bq load \
--source_format=CSV \
mydataset.mytable \
gs://mybucket/mydata.csv \
./myschema.json
For the BigQuery Json schema, the timestamp type is :
{
{
"name": "yourDate",
"type": "TIMESTAMP",
"mode": "NULLABLE",
"description": "Your date"
}
}
getting an error 1206 and 1205 when injesting data from fireshose to redshift using a copy command
Below is the raw data on firehose
{
"Name": "yoyo"
"a_timestamp": "2021-05-11T15:02:02.426729Z",
"a_date": "2021-05-11T00:00:00Z"
}
below is the copy command
COPY pqr_table FROM 's3://xyz/<manifest>' CREDENTIALS 'aws_iam_role=arn:aws:iam::<aws-account-id>:role/<role-name>' MANIFEST json 's3://xyz/abc.json' DATEFORMAT 'YYYY-MM-DD' ;
below is the DDL command
create table events (
Name varchar(8),
a_timestamp timestamp,
a_date date)
It would be great if anyone can please help me with this
Those are errors for bad timestamp and date formats. You need to have "timeformat" specified with that string as it is not Redshift's default format. I'd first try 'auto' for both of these and see if Redshift can work things out.
dateformat as 'auto'
timeformat as 'auto'
Also, having time specified in your date may create some confusion and may need you to manually specify the format or ingest as timestamp and then cast to date. I'd fist see if 'auto' does the trick.
How can i load a csv successfully using bq load (gsutil) where a FLOAT column has few values as #N/A.
I get the below error when i use the below bq load
bq --location=australia-southeast1 load --max_bad_records=20 --allow_jagged_rows --skip_leading_rows=1 --source_format=CSV DATASET1.T1.FILE1 gs://load_files/Test/FILE!.csv
ERROR - Could not parse #N/A as double for field blah blah
Modifying csv file is not an option
You can try the --null-marker flag (cf here), specifying "#N/A" as a special null character.
I am using Apache::Log::Parser to parse Apache log files.
I extracted the date from log file using the following code.
my $parser = Apache::Log::Parser->new(fast=>1);
my $log = $parser->parse($data);
$t = $log->{date};
Now,I tried to use Time::Piece to parse dates, but I'm unable to do it.
print "$t->day_of_month";
But, it's not working. How to use Time::Piece to parse date?
You cannot call methods on objects inside of string interpolation. It will probably output something like this:
Sat Feb 18 12:44:47 2017->day_of_month
Remove the double quotes "" to call the method.
print $t->day_of_month;
Now the output is:
18
Note that you need to create a Time::Piece object with localtime or gmtime if you have an epoch value in your log, or using strptime if the date is some kind of timestamp.
Can someone give a full example of date time functions including the 'register' jar ? I have been trying to get CurrentTime() and ToDate() running without much success. I have the piggybank jar in classpath and registered the same. But it always says the function has to be defined before usage.
I read this question comparing datetime in pig before this.
Datetime functions can be easily implemented using native pig, you no need to go for piggybank jar.
Example:
In this example i will read set of dates from the input file, get the current datetime and calculate the total no of days between previous and current date
input.txt
2014-10-12T10:20:47
2014-08-12T10:20:47
2014-07-12T10:20:47
PigScript:
A = LOAD 'input.txt' AS (mydate:chararray);
B = FOREACH A GENERATE ToDate(mydate) AS prevDate,CurrentTime() AS currentDate,DaysBetween(CurrentTime(),ToDate(mydate)) AS diffDays;
DUMP B;
Output:
(2014-10-12T10:20:47.000+05:30, 2014-12-12T10:39:15.455+05:30, 61)
(2014-08-12T10:20:47.000+05:30, 2014-12-12T10:39:15.455+05:30, 122)
(2014-07-12T10:20:47.000+05:30, 2014-12-12T10:39:15.455+05:30, 153)
You can refer few examples from my old post
Human readable String date converted to date using Pig?
Storing Date and Time In PIG
how to convert UTC time to IST using pig