I'm trying to grab a data row in BigQuery by timestamp '2018-12-08 00:00:42.808 America/Los_Angeles'. This works with between clause. For example, timestamp BETWEEN '....00:00:42.808...' AND '....00:00:42.809...'.
However, I'm not able to find anything when I just want to do timestamp = '....00:00:42.808...'. I'm not sure why this is and I can't seem to find much answer on google for this particular case.
The timestamp in Google BigQuery is quite exact, so your query probably does not EXACTLY hit the timestamps in your table. You can use TIMESTAMP_TRUNC -function if you want to hit timestamp at millisecond, second or any rounded level. With this function you can have a where clause like this:
where TIMESTAMP_TRUNC(timestamp, millisecond, 'America/Los_Angeles')='2018-12-08 00:00:42.808 America/Los_Angeles'
This would give you the result at millisecond level you expect. You can find more information on TIMESTAMP_TRUNC and other BigQuery functions from https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators.
Related
I have a column named "date_time" which has date and time stamp of some year, I want to find which part of the day does this time fall into, like morning, noon, evening and night for extracting the features as below :
if date_time.dt.hour >=5 and new_data.current_date_time.dt.hour <12 --> then it's morning
if date_time.dt.hour >=12 and new_data.current_date_time.dt.hour<17 --> then noon
if date_time.dt.hour >=17 and new_data.current_date_time.dt.hour<20 --. then evening
else night.
But I'm unable to filter as above using .dt.hour attribute of the pandas to_datetime datatype, please help me in achieving this.
You should get the datetime from timstamp, there are multiple ways to do this, you can refer this Converting between datetime and Pandas Timestamp objects.
Once you have time object like in HH, MM, SS etc. you can use your logic for getting morning, noon, evening, night.
One point here, standard is to get AM or PM or in 24HH time, if you want to say Morning or Evening you have to match time based on your condition, not searching for direct method.
Again its my opinion.
Specifically if i answer:
Step-1 Parse the timestamp into string, get object something like
(YY-MM-DD HH:MM:SS)
Step-2 Extract HH and MM from string object
Step-3 Perform your logic by casting these string into number
This is an addition to the Shubham's answer. I assume, following what he described, you are able to extract time-stamps properly.
Further, depending on your use-case, you may want to make changes to the column itself, or you could add one more column to store these values. Let me explain the process using the later. To run through an exemplar code, let me call this column as part_of_day. This can be done as
df["C"] = ""
Now, you will have to start add values to the column based on certain conditions. The general syntax for this would be.
df.loc[<mask to generate the labels to index> , <optional column(s)>] = <some value>
For your case, one of the conditions may be look like
df.loc[5 <= df.date_time.hour and df.date_time.hour <=12 , "part_of_day"] = "morning"
Right now I'm trying to check a timestamp column, and find out if the schedule I'm looking at starts at midnight. I don't want to check the date portion.
So far I've tried a few things, but they are all similar to this.
SELECT * FROM schedule_summary
WHERE sche_START_TIME = TIME('00:00:00');
time(Sche_START_TIME) = Time('00:00:00')
Convert both to time data types and compare them.
Though you need to be careful if timezones for either component could be different or if you need to account for variances in geographic location and times.
I could not find any function in snowflake docs that can do this.
If I understand what you mean correctly it appears to be:
TO_TIMESTAMP( epoch_sec )
This is the reference. There's variations for time zone support too.
Unfortunately I don't think there is a perfect solution for this issue. The Snowflake docs do say that the to_timestamp() function supports epoch seconds, microseconds, and nanoseconds, however their own example using the number 31536000000000000 does not even work.
select to_timestamp(31536000000000000); -- returns "Invalid Date" (incorrect)
The number of digits your epoch number has will vary by its origin source. What I found helpful was using a tool like epoch converter to input the full epoch number and determine what your date should be. Then try to get to that date in Snowflake using some manipulation. To get the real date for the example above:
select to_timestamp(left(31536000000000000, 11)); -- returns "1971-01-01" (correct)
You may notice that this is not set in stone. Adding or removing the number of digits you keep in your to_timestamp function will completely change the output, so you may need to add or remove numbers to get the date you are looking for. For example, the number 1418419324000000 should return date "2014-12-12"...
select to_timestamp(1418419324000000); -- returns "Invalid Date" (incorrect)
select to_timestamp(left(1418419324000000, 11)); -- returns "2419-06-24" (incorrect)
select to_timestamp(left(1418419324000000, 10)); -- returns "2014-12-12" (correct)
I had to play around with how many characters I input to get to where I needed to be. It's definitely a hack, but it's a simple solution to get there.
I was wondering if there was a way to store a date (example: 01/01/2013) as datetime without SQL Server CE adding the time (example: 12:00:00 AM).
I could always store it as the string "01/01/2013" but I really want to be able to compare the dates on querying the database.
I realize that as long as I only stored the date part, all of the times in the datetime field would have equal values (i.e. 12:00:00 AM), so comparing them wouldn't be a problem and I could just always ignore the time part, however, it seems ridiculous to have this unnecessary data appended to every entry in the table.
Is there a way to store only the date part of the datetime as datetime so that the dates can still be compared in the SQL query or do I just need to live with this overhead and move on?
Side Note:
I just spent the last 30 minutes searching Google and SO for an answer I was sure was already out there, but to my surprise, I couldn't find anything on this issue.
Update:
The conclusion I have come to is that I will just accept the time in the datetime format and let it always default to 12:00:00 AM by only adding the date part during the INSERT statement (e.g. 01/01/2013). As long as the time part always remains the same throughout, the dates will still be easily comparable and I can just trim it up when I convert it to string for screen display. I believe this will be the easiest way to handle this scenario. After all, I decided to use SQL for the power of its queries, otherwise, I might have just used XML instead of a database, in the first place.
No you really can't get rid of the time component. It is part of the data type defined by sql server. I was very annoyed by it until I found that I could still display the dates without the time using JQuery to reformat them with the date formatter plugi:
https://github.com/phstc/jquery-dateFormat
Good Luck!
select CONVERT(date, GETDATE())
Is there a way to change the default format of a date in Postgres?
Normally when I query a Postgres database, dates come out as yyyy-mm-dd hh:mm:ss+tz, like 2011-02-21 11:30:00-05.
But one particular program the dates come out yyyy-mm-dd hh:mm:ss.s, that is, there is no time zone and it shows tenths of a second.
Apparently something is changing the default date format, but I don't know what or where. I don't think it's a server-side configuration parameter, because I can access the same database with a different program and I get the format with the timezone.
I care because it appears to be ignoring my "set timezone" calls in addition to changing the format. All times come out EST.
Additional info:
If I write "select somedate from sometable" I get the "no timezone" format. But if I write "select to_char(somedate::timestamptz, 'yyyy-mm-dd hh24:mi:ss-tz')" then timezones work as I would expect.
This really sounds to me like something is setting all timestamps to implicitly be "to_char(date::timestamp, 'yyyy-mm-dd hh24:mi:ss.m')". But I can't find anything in the documentation about how I would do this if I wanted to, nor can I find anything in the code that appears to do this. Though as I don't know what to look for, that doesn't prove much.
Never mind :'(
I found my problem. I was thinking that I was looking directly at the string coming back from the database. But I was overlooking that it was reading it as a Timestamp and then converting the Timestamp to a string. This was buried inside a function called "getString", which is what threw me off. I was thinking it was ResultSet.getString, but it was really our own function with the same name. Oops. What idiot wrote that function?! Oh, it was me ...
Thanks to all who tried to help. I'll give you each an upvote for your trouble.
I believe the table columns are specified differently. Try these variants:
timestamp
timestamp(0) no millis
timestamptz with timezone
timestamptz(0) with timezone, no millis
With which client are you running the select statements? Formatting the output is the application's responsibility, so without knowing which application you use to display the data, it's hard to tell.
Assuming you are using psql, you can change the date format using the SET command:
http://www.postgresql.org/docs/current/static/sql-set.html
Which is essentially a way to change the configuration parameters. The ones that are responsible for formatting data are documented here:
http://www.postgresql.org/docs/current/static/runtime-config-client.html#RUNTIME-CONFIG-CLIENT-FORMAT
Daniel tells me to post my findings as an answer and accept it to close the question. Okay.
I found that the date format I was seeing that did not include a time zone was not what was coming directly from Postgres, but that there were a couple of function calls that I was missing that converted the incoming date to a java.util.Timestamp, and then from the java.util.Timestamp to a String. It was in this conversion from the Timestamp to the String that the time zone was defaulting to EST.
In my own humble defense, my mistake was not as dumb as it may sound. :-0 We had the execution of the query in a subclass that read the results into a List, which we do to allow modification of the query results before output. (In this case we are adding a coule of columns that are derived from the stored columns.) Then we have a set of functions that resemble the JDBC functions to pull the data out of the List, so a calling program can easily switch from processing a query directly to processing the List. When I was wrestling with the date format problem, it just didn't register on me that I wasn't looking at "real JDBC", but at "simulated JDBC" calls.