Looking at the functions here. The formatting shows fractional seconds available to the millisecond (0.001) but is it accurate to the millisecond? I have not been able to find the resolution of these calls in any of the documentation.
https://www.sqlite.org/datatype3.html#section_2_2
SQLite does not have a storage class set aside for storing dates
and/or times. Instead, the built-in Date And Time Functions of SQLite
are capable of storing dates and times as TEXT, REAL, or INTEGER
values:
TEXT as ISO8601 strings ("YYYY-MM-DD HH:MM:SS.SSS").
REAL as Julian day numbers, the number of days since noon in Greenwich on November
24, 4714 B.C. according to the proleptic Gregorian calendar.
INTEGER as Unix Time, the number of seconds since 1970-01-01 00:00:00 UTC.
Applications can chose to store dates and times in any of these
formats and freely convert between formats using the built-in date and
time functions.
It appears that you can reach the highest resolution with ISO8601 strings. There should be no problem with accuracy with these strings, as long as you're not mixing storage representations.
This depends on the date format.
INTEGER numbers are accurate to the second.
TEXT values are accurate to the millisecond. You could specify more digits in the fractional seconds fields, but the built-in function will ignore all after the first three.
The resolution of Julian day numbers is better than a millisecond, but when formatting them, the built-in functions will not output more than three fractional digits.
If you want to store time values with higher resolution than is permitted by the date and time functions, you can opt for INTEGER storage, but you're on your own for conversion functions and the like.
Storing unix epoch timestamps with up to nanosecond precision will cover dates from 1970 until minimally the year 2262 (if nanoseconds). You're on your own for conversion to/from ISO date strings and time unit-relative arithmetic, but if you're content with that, then you'll have a fast, high precision and compact storage format.
I have a system that receives sensor values from devices at a rate timed in nanoseconds, so storing and indexing on those timestamps has been very helpful. I do have to provide a query interface that converts from ISO, Python datetime and other formats into ns and back, but that's the deal.
Working with 8-byte timestamps helped keep the record length short, with quick inserts and queries.
Related
I have some SQLite database in which one of the columns has data type as realdate and the column has value as 2453137.5
can anyone please comment on this?
any help is appreciated :)
From SQLlite Docs
SQLite does not have a storage class set aside for storing dates and/or times. Instead, the built-in Date And Time Functions of SQLite are capable of storing dates and times as TEXT, REAL, or INTEGER values:
TEXT as ISO8601 strings ("YYYY-MM-DD HH:MM:SS.SSS").
REAL as Julian day numbers, the number of days since noon in Greenwich on November 24, 4714 B.C. according to the proleptic Gregorian calendar.
INTEGER as Unix Time, the number of seconds since 1970-01-01 00:00:00 UTC.
Applications can chose to store dates and times in any of these formats and freely convert between formats using the built-in date and time functions.
In your example you are using REAL datatype to store Dates. It will give the output which is not human readable.
For eg., If i'm storing current date and time
CREATE TABLE
IF NOT EXISTS DATEREAL (d1 real);
INSERT INTO DATEREAL (d1)
VALUES(julianday('now'));
SELECT * from DATEREAL;
Output : 2458792.7882345
You can read this using built-in date() and time() as shown below
SELECT
date(d1),
time(d1)
FROM
datereal;
Output :
date(d1) time(d1)
2019-11-05 06:55:03
Check demo here
One of the powerful features of SQLite is allowing you to choose the storage type.
Real number has 2 advantages:
High precision regarding fraction seconds
Longest time range
I got this answer from a user named Zso.
Here's the link to the original post How do DATETIME values work in SQLite?.
Hope this might help you to understand better.
I work on an application that stores datetimes in a SQL Server database. Some of these are a point in time stored in UTC (such as log item datetimes), while others are a literal date/time (such as "take medication X at 4pm on 20 July, irrespective of your timezone).
Problem is that these both have a date and time component, so using a datetime2 column type makes sense for both. We're now in a situation where it is often unclear in our app whether a date/time column is a UTC point in time or a literal date/time.
What is the most common practice to distinguish between these 2 cases? I can think of these options:
1) End all UTC columns in ...Utc, while literal date/time columns have no special ending.
2) End all literal columns in ...Literal, while UTC date/time columns have no special ending.
3) Give UTC columns the data type datetime2 and literal date/time columns datetimeoffset.
Always try to use the appropriate type first and then good naming. If datetime2(0) is a good fit, use it.
In my system I add a suffix to the column name, for example: PlaybackStartedLocal datetime2(0), PlaybackStartedUTC datetime2(0). In my case I have to store both local and UTC values for the same event, because some reports need local value, some UTC and it is very difficult to convert between them later.
In general it is a good practice to include units of measurement into the column/variable name.
What do you prefer to see:
PlaybackDurationMSec or PlaybackDuration
LengthMeters / LengthMiles or Length
A well-known example when two teams of programmers didn't notice that they were interpreting metric values as imperial and visa versa: A disaster investigation board reports that NASA’s Mars Climate Orbiter burned up in the Martian atmosphere because engineers failed to convert units from English to metric.
The software calculated the force the thrusters needed to exert in
pounds of force. A separate piece of software took in the data
assuming it was in the metric unit: newtons.
We had this programming discussion on Freenode and this question came up when I was trying to use a VARCHAR(255) to store a Date Variable in this format: D/MM/YYYY. So the question is why is it so bad to use a VARCHAR to store a date. Here are the advantages:
Its faster to code. Previously I used DATE, but date formatting was a real pain.
Its more power hungry to use string than Date? Who cares, we live in the Ghz era.
Its not ethically correct (lolwut?) This is what the other user told me...
So what would you prefer to use to store a date? SQL VARCHAR or SQL DATE?
Why not put screws in with a hammer?
Because it isn't the right tool for the job.
Some of the disadvantages of the VARCHAR version:
You can't easily add / subtract days to the VARCHAR version.
It is harder to extract just month / year.
There is nothing stopping you putting non-date data in the VARCHAR column in the database.
The VARCHAR version is culture specific.
You can't easily sort the dates.
It is difficult to change the format if you want to later.
It is unconventional, which will make it harder for other developers to understand.
In many environments, using VARCHAR will use more storage space. This may not matter for small amounts of data, but in commercial environments with millions of rows of data this might well make a big difference.
Of course, in your hobby projects you can do what you want. In a professional environment I'd insist on using the right tool for the job.
When you'll have database with more than 2-3 million rows you'll know why it's better to use DATETIME than VARCHAR :)
Simple answer is that with databases - processing power isn't a problem anymore. Just the database size is because of HDD's seek time.
Basically with modern harddisks you can read about 100 records / second if they're read in random order (usually the case) so you must do everything you can to minimize DB size, because:
The HDD's heads won't have to "travel" this much
You'll fit more data in RAM
In the end it's always HDD's seek times that will kill you. Eg. some simple GROUP BY query with many rows could take a couple of hours when done on disk compared to couple of seconds when done in RAM => because of seek times.
For VARCHAR's you can't do any searches. If you hate the way how SQL deals with dates so much, just use unix timestamp in 32 bit integer field. You'll have (basically) all advantages of using SQL DATE field, you'll just have to manipulate and format dates using your choosen programming language, not SQL functions.
Two reasons:
Sorting results by the dates
Not sensitive to date formatting changes
So let's take for instance a set of records that looks like this:
5/12/1999 | Frank N Stein
1/22/2005 | Drake U. La
10/4/1962 | Goul Friend
If we were to store the data your way, but sorted on the dates in assending order SQL will respond with the resultset that looks like this:
1/22/2005 | Drake U. La
10/4/1962 | Goul Friend
5/12/1999 | Frank N. Stein
Where if we stored the dates as a DATETIME, SQL will respond correctly ordering them like this:
10/4/1962 | Goul Friend
5/12/1999 | Frank N. Stein
1/22/2005 | Drake U. La
Additionally, if somewhere down the road you needed to display dates in a different format, for example like YYYY-MM-DD, then you would need to transform all your data or deal with mixed content. When it's stored as a SQL DATE, you are forced to make the transform in code, and very likely have one spot to change the format to display all dates--for free.
Between DATE/DATETIME and VARCHAR for dates I would go with DATE/DATETIME everytime. But there is a overlooked third option. Storing it as a INTEGER unsigned!
I decided to go with INTEGER unsigned in my last project, and I am really satisfied with making that choice instead of storing it as a DATE/DATETIME. Because I was passing along dates between client and server it made the ideal type for me to use. Instead of having to store it as DATE and having to convert back every time I select, I just select it and use it however I want it. If you want to select the date as a "human-readable" date you can use the FROM_UNIXTIME() function.
Also a integer takes up 4 bytes while DATETIME takes up 8 bytes. Saving 50% storage.
The sorting problem that Berin proposes is also solved using integer as storage for dates.
I'd vote for using the date/datetime types, just for the sake of simplicity/consistency.
If you do store it as a character string, store it in ISO 8601 format:
http://www.iso.org/iso/date_and_time_format
http://xml.coverpages.org/ISO-FDIS-8601.pdf
http://www.cl.cam.ac.uk/~mgk25/iso-time.html
Among other things, ISO 8601 date/time string (A) collate properly, (B) are human readable, (C) are locale-indepedent, and (D) are readily convertable to other formats. To crib from the ISO blurb, ISO 8601 strings offer
representations for the following:
Date
Time of the day
Coordinated universal time (UTC)
Local time with offset to UTC
Date and time
Time intervals
Recurring time intervals
Representations can be in one of two formats: a basic format
that has a minimal number of characters and an extended format
that adds characters to enhance human readability. For example,
the third of January 2003 can be represented as either 20030103
or 2003-01-03.
[and]
offer the following advantages over many of the locally used
representations:
Easily readable and writeable by systems
Easily comparable and sortable
Language independent
Larger units are written in front of smaller units
For most representations the notation is short and of constant length
One last thing: If all you need to do is store a date, then storing it in the ISO 8601 short form YYYYMMDD in a char(8) column takes no more storage than a datetime value (and you don't need to worry about the 3 millisecond gap between the last tick of the one day and the first tick of the next. But that's a matter for another discussion. If you break it up into 3 columns — YYYY char(4), MM char(2), DD char(2) you'll use up the same amount of storage, and get more options for indexing. Even better, store the fields as a short for yyyy (4 bytes), and a tinyint for each of MM and DD — now you're down to 6 bytes for the date. The drawback, of course, to decomposing the date components into their constituent parts is that conversion to proper date/time data types is complicated.
when they say "we will use unix time", does that mean i should create a column like this ?
date DATE
or
date TIME
or
date DATETIME
?
A unix time is just a number representing the nr of seconds since 1970. If you want to store that directly, use an INTEGER.
You're likely better off storing this as a DATETIME, as that will facilitate easier querying. Though when inserting data you might need to convert it using the from_unixtime
function.
A DATE stores only the date, not the time(hour/min/secs) and a TIME stores only the time not the date so they're not suitable.
UNIX time is expressed as an integer value - number of seconds since the epoch. You'd just want to use an int field for this, not any of the date variants.
Unix-Time is normally seconds from 01-01-1970...means integer.
What is the underlying datastructure of datetime values stored in SQL Server (2000 and 2005 if different)? Ie down to the byte representation?
Presumably the default representation you get when you select a datetime column is a culture specific value / subject to change. That is, some underlying structure that we don't see is getting formatted to YYYY-MM-DD HH:MM:SS.mmm.
Reason I ask is that there's a generally held view in my department that it's stored in memory literally as YYYY-MM-DD HH:MM:SS.mmm but I'm sure this isn't the case.
It's stored as an 8 byte field, capable of a range from 1753-01-01 through 9999-12-31, accurate to 0.00333 seconds.
The details are supposedly opaque, but most resources (1), (2) that I've found on the web state the following:
The first 4 bytes store the number of days since SQL Server's epoch (1st Jan 1900) and that the second 4 bytes stores the number of ticks after midnight, where a "tick" is 3.3 milliseconds.
The first four bytes are signed (can be positive or negative), which explains why dates earlier than the epoch can be represented.