I'm doing some ETL from a CSV file in GCS to BQ, everything works fine, except for dates. The field name in my table is TEST_TIME and the type is DATE, so in the TableRow I tried passing a java.util.Date, a com.google.api.client.util.DateTime, a String, a Long value with the number of seconds, but none worked.
I got error messages like these:
Could not convert non-string JSON value to DATE type. Field: TEST_TIME; Value: ...
When using DateTime I got this error:
JSON object specified for non-record field: TEST_TIME.
//tableRow.set("TEST_TIME", date);
//tableRow.set("TEST_TIME", new DateTime(date));
//tableRow.set("TEST_TIME", date.getTime()/1000);
//tableRow.set("TEST_TIME", dateFormatter.format(date)); //e.g. 05/06/2016
I think that you're expected to pass a String in the format YYYY-MM-DD, which is similar to if you were using the REST API directly with JSON. Try this:
tableRow.set("TEST_TIME", "2017-04-06");
If that works, then you can convert the actual date that you have to that format and it should also work.
While working with google cloud dataflow, I used a wrapper from Google for timestamp - com.google.api.client.util.DateTime.
This worked for me while inserting rows into Big Query tables. So, instead of
tableRow.set("TEST_TIME" , "2017-04-07");
I would recommend
tableRow.set("TEST_TIME" , new DateTime(new Date()));
I find this to be a lot cleaner than passing timestamp as a string.
Using the Java class com.google.api.services.bigquery.model.TableRow, to set milliseconds since UTC into a BigQuery TIMESTAMP do this:
tableRow.set("timestamp", millisecondsSinceUTC / 1000.0d);
tableRow.set() expects a floating point number representing seconds since UTC with up to microsecond precision.
Very non-standard and undocumented (set() boxes the value in an object, so it's unclear what data types set() accepts. The other proposed solution of using com.google.api.client.util.DateTime did not work for me.)
Related
Snowflake documentation says to use TO_DATE('2022-01-01', 'YYYYMM'), however, when running that I receive the error message:
"Error: too many arguments for function [TO_DATE("policy_effective_date", 'YYYYMM')] expected 1, got 2"
Any help is appreciated.
I was expecting to see 2022-01-01 turn into 202201. Even if I need to bring in DD that's fine too, I can just capture the LEFT 6 digits, but regardless the system is saying it's too many arguments.
TO_DATE() function accepts string/varchar format date and converts to Date Data type. For that we need to pass existing String date type format to parse. To Convert to the required format we need to use to_varchar as given in the documentation. https://docs.snowflake.com/en/sql-reference/functions-conversion.html
I have a bigquery code.
CREATE TEMP FUNCTION to_struct_attributes(input STRING)
RETURNS STRUCT<status_code STRING, created_time TIMESTAMP>
LANGUAGE js AS """
let res = JSON.parse(input);
res['created_time'] = Date(res['created_time'])
return res;
""";
SELECT
5 AS ID,
to_struct_attributes(
TO_JSON_STRING(
STRUCT(
TIMESTAMP(PARSE_TIMESTAMP('%Y%m%d%H%M%S', '20220215175959','America/Los_Angeles')) AS created_time
)
)
) AS ATTRIBUTES;
When I execute this, I'm getting the following error:
Failed to coerce output value "2022-02-16 01:59:59+00" to type TIMESTAMP
I feel this is quite strange, since BigQuery should be able to interpret it correctly and I haven't had this issue with any other datatypes. Also, if I do:
SELECT TIMESTAMP("2022-02-16 01:59:59+00")
It returns:
2022-02-16 01:59:59 UTC
So BigQuery can indeed parse it correctly. I'm not sure why it doesn't happen for the UDF. On searching the internet, I found this question and as the answer suggests, if I change the return statement to:
return Date(res.created_time);
It resolves the issue. But for a project of mine, doing it for every timestamp is not feasible due to the high number of struct columns.
So, I wanted to know if someone has a better alternative to it?
PS : I have removed a lot of non-essential parts from the above example, so this might look a bit abstract. Also, the actual use-case is a bit different and complex that's why I need that JS udf.
The best way to do what you want is to implement the following code.
return Date(res.created_time);
This happens when you pass a TIMESTAMP to a UDF, it is represented as a DATE object, as stated in the documentation. This is like a return of a TIMESTAMP from a JavaScript UDF, where you need to construct and return a DATE object.
I was trying to query the duration of a WWE wrestler in the matches which he won and here I have successfully retrieved the data:
I would like to remove those zeros and I tried the following piece of code:
strftime(%H:%M:%S, R1.time_in_match)
which doesn't work and give the following error:
': near "%": syntax error
The data I imported were from a data frame which stores the time as time object and put them into the database as type DATETIME. The typeof function used on the "time_in_match" field returned type TEXT.
I was wondering if there is any way to format the time.
Thanks!
SQLite provides 2 functions for this case:
strftime('%H:%M:%S', R1.time_in_match)
and simpler:
time(R1.time_in_match)
because:
The time() function returns the time as HH:MM:SS
(from Date And Time Functions)
See the demo.
Your expression works as intended if you surround the format specifier with single quotes:
strftime('%H:%M:%S', time_in_match)
You could also just use string functions here:
substr(time_in_match, 1, 8)
Demo on DB Fiddle
I wanted to use Date.UTC to store dates and datetimes in postgresql 9.2 "json" field, but of course it fails:
hp=> update formapp_record set data='{"dt": Date.UTC(120, 10, 2)}' where id=17;
ERROR: invalid input syntax for type json
LINE 1: update formapp_record set data='{"dt": Date.UTC(120, 10, 2)}...
^
DETAIL: Token "Date" is invalid.
CONTEXT: JSON data, line 1: {"dt": Date...
It is possible to store the UTC timestamp directly, but then how could the decoder know that the value should decode to a date or datetime instead of an int ?
It is also possible to store the Date.UTC call as string as such:
update formapp_record set data='{"dt": "Date.UTC(120, 10, 2)"}' where id=17;
While that works, it requires 0. checking if the string starts with Date.UTC and 1. use eval in plv8
A solution would be to store some metadata like:
update formapp_record set data='{"dt": {"_type": "date", "_value": [120, 10, 2]}}' where id=17;
But that's not very "standard", it's even "hackish".
What's your take on this matter ?
Alas, json doesn't know anything about dates.
I'd store an ISO 8601 date as a string. Yes, it's a pain. Yes, it means there's no nice standard way to tell "this is a date" vs "this is a string". IMO it's less painful than most of the other options, though.
A possible solution is to use Postgres's row_to_json function and just store your dates as timestamps and extract them to json as required. However, Tobe Hede wrote some json functions for Postgres that may help and seem to be alot more complete then the 2 native options that Postgres has made available for 9.2
See post How do I query using fields inside the new PostgreSQL JSON datatype? for the thread.
I'm working in a web application using VB.NET. There is also VisualBasic code mixed in it, in particular the Date variable and the Month function of VB.
The problem is this part:
Month("10/01/2008")
On the servers, I get 10 (October) as the month (which is supposed to be correct). On my machine, I get 1 (January) (which is supposed to be wrong).
Two of my colleagues (on their own machines) get different answers, one got 1, the other got 10.
The question is, why is this so?
On my end, I can solve the problem by using .NET's DateTime's Parse (or ParseExact) function to force everything to be "dd/MM/yyyy" format. This works. I'm just wondering why there's an inconsistency.
Extra info: I know the parameter for Month function is supposed to be a Date variable. The code used a string as parameter, and Option Strict was off, and the developers mainly let VB do its own conversion thing. (Legacy code maintenance has a lot of inertia...)
If it helps, the version of Microsoft.VisualBasic.dll on the servers is 7.10.6310.4 (under the Framework folder v1.1.4322). The version on mine (and my 2 colleagues') machine is 7.10.6001.4.
Edit: Regional settings for all machines already set to dd/MM/yyyy format (short date format).
This normally has to do with the regional settings, and more specifically the date/time formats. If you set these formats so that they are all the same on the machines you're testing on, the results should be consistent.
Your idea of using ParseExact is definitely the better solution to go with, IMHO.
This is because the runtime has to convert your given value "10/01/2008" which is indeed a string implicitly to the DateTime datatype.
When converting strings to dates and the other way round, the string format depends on the locale settings of windows.
See this link on msdn.
In this article a way to specify a date literal which is independent of your locale settings:
Just enclose the date with the sign # and specify it in the form mm/dd/yyyy:
So the code
Month(#10/01/2008#)
should give you the answer 10 on any machine.
Ther a two more worarounds given in that msdn article:
1. Use the Format Function with predifned Date/Time Format
To convert a Date literal to the
format of your locale, or to a custom
format, supply the literal to the
Format Function, specifying either
Predefined Date/Time Formats (Format
Function) or User-Defined Date/Time
Formats (Format Function). The
following example demonstrates this.
MsgBox("The formatted date is " &
Format(#5/31/1993#, "dddd, d MMM
yyyy"))
2. Use the DateTime-Class Constructor to construt the right DateTime value
Alternatively, you can use one of the
overloaded constructors of the
DateTime structure to assemble a date
and time value. The following example
creates a value to represent May 31,
1993 at 12:14 in the afternoon.
Dim dateInMay As New
System.DateTime(1993, 5, 31, 12, 14,
0)