Please help, I need to filter between dates using java mongodb driver
below is is my Filtering operation however, its failed to select between the date
FindIterable<Document> documents = collection
.find(Filters.and("started", gte("2019-01-01T00:00:00.000Z"), lt("2019-03-01T00:00:00.000Z")))
Therefore, I want to be about to filter for Date range.
You are doing your filter operation on String representation of dates.
You can try the following in order for Spring-data to construct your filter with $date operator.
Instant from = Instant.parse("2019-01-01T00:00:00.000Z");
Instant to = Instant.parse("2019-03-01T00:00:00.000Z");
FindIterable<Document> documents = collection.find(Filters.and("started", gte(from),lt(to)));
I want to get the latest date from my database.
Here is my sql statement.
select "RegDate"
from "Dev"
where "RegDate" = (
select max("RegDate") from "Dev")
It works in my database.
But how can I use it in django?
I tried these codes but it return error. These code are in views.py.
Version 1:
lastest_date = Dev.objects.filter(reg_date=max(reg_date))
Error:
'NoneType' object is not iterable
Version 2:
last_activation_date = Dev.objects.filter(regdate='reg_date').order_by('-regdate')[0]
Error:
"'reg_date' value has an invalid format. It must be in YYYY-MM-DD HH:MM[:ss[.uuuuuu]][TZ] format."
I've defined reg_date at beginning of the class.
What should I do for this?
You make things too complicated, you simply order by regdate, that's all:
last_activation_dev = Dev.objects.order_by('-regdate').first()
The .first() will return such Dev object if it is available, or None if there are no Dev objects.
If you only are interested in the regdate column itself, you can use .values_list(..):
last_activation_date = Dev.objects.order_by('-regdate').values_list('regdate', flat=True).first()
By using .filter() you actually were filtering the Dev table by Dev records such that the regdate column had as value 'reg_date', since 'reg_date' is not a valid datetime format, this thus produced an error.
I am writing a perl script which copies data from a table in one db to the same table in other DB. I am using DBI to obtain connection to DBS.
I noticed that when I am copying data,it's not copied properly.
In source table if date is like this-'04/22/1996 13:51:15 PM'
In destination table it's appearing like this-'22 APR 1996'.
Can anyone help me in copying exact date?
Thanks in advance
#!/usr/bin/env perl
use DateTime::Format::Strptime;
my $datetimestr = '04/22/1996 13:51:15 PM';
# 1) parse datetime
my $parser = DateTime::Format::Strptime->new(
pattern=>'%m/%d/%Y %H:%M:%S %p',
on_error => 'croak',
);
my $dt = $parser->parse_datetime($datetimestr,);
# 2) format datetime for your database format
warn $dt->strftime('%Y-%m-%d %H:%M:%S');
In Spark I am trying to execute SQL queries on a temporary table derived from a data frame that I manually built by reading a csv file and converting the columns into the right data type.
Specifically, the table I'm talking about is the LINEITEM table from [TPC-H specification][1]. Unlike stated in the specification I am using TIMESTAMP rather than DATE because I've read that Spark does not support the DATE type.
In my single scala source file, after creating the data frame and registering a temporary table called "lineitem", I am trying to execute the following query:
val res = sqlContext.sql("SELECT * FROM lineitem l WHERE date(l.shipdate) <= date('1998-12-01 00:00:00');")
When I submit the packaged jar using spark-submit, I get the following error:
Exception in thread "main" java.lang.RuntimeException: [1.75] failure: ``union'' expected but but `;' found
When I omit the semicolon and do the same thing, I get the following error:
Exception in thread "main" java.util.NoSuchElementException: key not found: date
Spark version is 1.4.0.
Does anyone have an idea what's the problem with these queries?
[1] http://www.tpc.org/TPC_Documents_Current_Versions/pdf/tpch2.17.1.pdf
SQL queries passed to SQLContext.sql shouldn't be delimited using semicolon - this the source of your first problem
DATE UDF expects date in the YYYY-MM-DD form and DATE('1998-12-01 00:00:00') evaluates to null. As long as timestamp can be casted to DATE correct query string looks like this:
"SELECT * FROM lineitem l WHERE date(l.shipdate) <= date('1998-12-01')"
DATE is a Hive UDF. It means you have to use HiveContext not a standard SQLContext - this is the source of your second problem.
import org.apache.spark.sql.hive.HiveContext
val sqlContext = new HiveContext(sc) // where sc is a SparkContext
In Spark >= 1.5 it is also possible to use to_date function:
import org.apache.spark.sql.functions.{lit, to_date}
df.where(to_date($"shipdate") <= to_date(lit("1998-12-01")))
Please try hive function CAST (expression AS toDatatype)
It changes an expression from one datatype to other
e.g. CAST ('2016-06-17 00.00.000' AS DATE) will convert String to Date
In your case
val res = sqlContext.sql("SELECT * FROM lineitem l WHERE CAST(l.shipdate as DATE) <= CAST('1998-12-01 00:00:00' AS DATE);")
Supported datatype conversions are as listed in Hive Casting Dates
So this is VERY strange. RODBC seems to drop the time portion of DateTime SQL columns if the result set is large enough. (The queries are running against an SQL Server 2012 machine, and, yes, when I run them on the SQL Server side they produce identical and proper results, regardless of how many rows are returned.)
For example, the following works perfectly:
myconn <- odbcConnect(dsnName, uid, pwd)
results <- sqlQuery(myconn, "SELECT TOP 100 MyID, MyDateTimeColumn from MyTable ORDER BY MyDateTimeColumn DESC")
close(myconn)
In R, the following works as expected:
> results$MyDateTimeColumn[3]
[1] "2013-07-01 00:01:22 PDT"
which is a valid POSIXct date time. However, when somewhere between 10,000 and 100,000 rows are returned, suddenly the time portion disappears:
myconn <- odbcConnect(dsnName, uid, pwd)
bigResults <- sqlQuery(myconn, "SELECT TOP 100000 MyID, MyDateTimeColumn from MyTable ORDER BY MyDateTimeColumn DESC")
close(myconn)
(same code, simply a larger number of rows returned; NOTE: the exact same row has now lost its time component), R responds:
> bigResults$MyDateTimeColumn[3]
[1] "2013-07-01 PDT"
Note that the time is now missing (this is not a different row; it's the exact same row as previous), as the following shows:
>strptime(results$TriggerTime[3], "%Y-%m-%d %H:%M:%S")
[1] "2013-07-01 00:01:22"
>strptime(bigResults$TriggerTime[3], "%Y-%m-%d %H:%M:%S")
[1] NA
Obviously the work-around is either incremental query-with-append or export-to-CSV-and-import-to-R, but this seems very odd. Anyone ever seen anything like this?
Config: I'm using the latest version of RODBC (1.3-10) and can duplicate the behavior on both an R installation running on Windows x64 and an R installation running on Mac OS X 10.9 (Mavericks).
EDIT #2 Adding output of dput() to compare the objects, per request:
> dput(results[1:10,]$MyDateTimeColumn)
structure(c(1396909903.347, 1396909894.587, 1396909430.903, 1396907996.9, 1396907590.02, 1396906077.887, 1396906071.99, 1396905537.36, 1396905531.413, 1396905231.787), class = c("POSIXct", "POSIXt"), tzone = "")
> dput(bigResults[1:10,]$MyDateTimeColumn)
structure(c(1396854000, 1396854000, 1396854000, 1396854000, 1396854000, 1396854000, 1396854000, 1396854000, 1396854000, 1396854000), class = c("POSIXct", "POSIXt"), tzone = "")
It would appear that the underlying data are actually changing as a result of the number of rows returned by the query, which is downright strange.
sqlQuery() has an option called as.is. Setting this to TRUE will pull everything as seen in for example Microsoft SQL Management Studio.
I cope with the same problem as well. Even stranger, on a large dataset one column would import both date and time the other column only imported the date.
My advise would be to split the data/time in SQL
myconn <- odbcConnect(dsnName, uid, pwd)
results <- sqlQuery(myconn, "SELECT TOP 100 MyID, format(MyDateTimeColumn,"HH:mm:ss") as MyTimeColumn,format(MyDateTimeColumn,"yyyy-MM-dd") as MyDateColumn from MyTable ORDER BY MyDateTimeColumn DESC")
close(myconn)
Then combine them in R afterwards. Hope it helps.
I had the same issue and concluded that it is due to DST:
This fails:
as.POSIXct(c("2015-03-29 01:59:22", "2015-03-29 02:00:04"))
This works:
as.POSIXct(c("2015-03-29 01:59:22", "2015-03-29 02:00:04"), tz="UTC")
I could not find how to force tz="UTC" in default RODBC behavior, however using as.is = TRUE and converting columns myself does the job.
Note: At first I had the impression that it was due to huge result, but in fact it was due to the fact that in huge results I have more chances to cross DST updates.
This is an older question, but I had similar issues when trying to programmatically read in data from 15 different .accdb. All POSIXct fields were read in correctly for every database except those from the months of March, from which I inferred that it is some sort of daylight-savings time issue.
The solution for me (because I didn't want to have to make multiple queries to a dB and then rbind() everything together) was to alter my function to include the lines
#Get initial tz
current_tz <- Sys.timezone()
Sys.setenv(TZ = "America/Phoenix")
[BODY OF FUNCTION]
Sys.setenv(TZ = curent_tz)
After including these few lines, the day/time fields from the March databases were being read in correctly.
sqlQuery(ch, getSQL(sqlquerypath))
stripped the times off my datetime column.
sqlQuery(ch, getSQL(sqlquerypath), as.is = TRUE)
fixed the issue.
I think this is a case of times being stripped from the dates where the date range includes shifts to/from daylight savings time. If you are selecting periods that don't include daylight savings shifts, the times will be retained (e.g., from 1/1/2007 to 3/1/2007. This could possibly be avoided by changing the system time on your computer to follow a time zone (e.g., Arizona) where there are no daylight savings shifts (sounds bizarre, but it has worked for me).
To overcome this issue, import the DateTimes as characters (using "as.is") and then convert them to POSIXct. You could alternatively use "strptime" which converts to POSIXlt and allows you to specify the format. Here's an example of a SQL query where DateTimes are imported as.is (TRUE), but asociated DataValues are not (FALSE) and then the DateTime is converted to an R date format:
data <- sqlQuery(channel, paste("SELECT LocalDateTime, DataValue FROM DataValues WHERE LocalDateTime >= '1/1/2007 0:00' AND LocalDateTime < '1/1/2008 0:00' ORDER BY LocalDateTime ASC"),as.is=c(TRUE,FALSE))
data$LocalDateTime <- as.POSIXct(totalP$LocalDateTime,tz="MST")
It may be a daylight saving issue. If there is a time that doesnt exist in your timezone (because of daylight saving) it may cause something like this.
Why this happens on large datasets returned from sqlQuery()? I don't know. But was able to workaround it by applying a sql conversion in the sql call:
data <- sqlQuery(channel, "SELECT CONVERT(nvarchar(24), DtTimeField, 21) AS HourDt, * FROM ...
This is your workaround.