Please help, I need to filter between dates using java mongodb driver
below is is my Filtering operation however, its failed to select between the date
FindIterable<Document> documents = collection
.find(Filters.and("started", gte("2019-01-01T00:00:00.000Z"), lt("2019-03-01T00:00:00.000Z")))
Therefore, I want to be about to filter for Date range.
You are doing your filter operation on String representation of dates.
You can try the following in order for Spring-data to construct your filter with $date operator.
Instant from = Instant.parse("2019-01-01T00:00:00.000Z");
Instant to = Instant.parse("2019-03-01T00:00:00.000Z");
FindIterable<Document> documents = collection.find(Filters.and("started", gte(from),lt(to)));
Related
I have this big query GA4 dataset and on Friday I added a new parameter.
Now, I want to query the dataset including this new param but the query is returning an error because it did not exist in the past. Is there a way to make it return an empty string or something similar if the param is not found?
Excuse my poor SQL knowledge but I am unsure of what I should be searching for online to get the correct SQL syntax. This is what I have so far:
SELECT
(PARSE_DATE('%Y%m%d', event_date)) AS eventDate,
event_name,
NULLIF(params1.value.int_value, null) AS paramName
FROM
`myDataset`,
UNNEST(event_params) AS params1
WHERE
event_name = 'eventName'
AND (params1.key = 'paramName')
LIMIT 100
paramName is the new param added on Friday. When I try to run this for yesterday it works fine but when I try to run it for the entire date range, I get This query returned no results.
I am new to Pig scripting but good with SQL. I wanted the pig equivalent for this SQL line :
SELECT * FROM Orders WHERE Date='2008-11-11'.
Basically I want to load data for one id or date how do I do that?
I did this and it worked, used FILTER in pig, and got the desired results.
`ivr_src = LOAD '/raw/prod/...;
info = foreach ivr_src generate timeEpochMillisUTC as time, cSId as id;
Filter_table= FILTER info BY id == '700000';
sorted_filter_table = Order Filter_table BY $1;
store sorted_filter_table into 'sorted_filter_table1' USING PigStorage('\t', '-
schema');`
Azure Diagnostics is pushing Windows Events into a storage table "WADWindowsEventLogsTable".
I would like to query this storage Table using VisualStudio(2015) and CloudExplorer.
As this table has an huge content, I'm indefinitely waiting for the results..
Here is a query sample:
EventId eq 4096 and Timestamp gt datetime'2016-06-24T08:20:00' and Timestamp lt datetime'2016-06-24T10:00:00'
I suppose that this query is correct ?
Does exist a way to improve performance ?
filter result columns ?
return only TOP X results ?
another usefull tips ?
I know that a better way would be to script that; for example using Python, but I would like to use the UI as much as possible..
(Edit) following Gaurav Mantri answer I used this little C# program to build my query. The answer is so quick and that solve my initial performance issue:
static void Main(string[] args)
{
string startDate = "24 June 2016 8:20:00 AM";
string endDate = "24 June 2016 10:00:00 AM";
string startPKey = convertDateToPKey(startDate);
string endPKey = convertDateToPKey(endDate);
Debug.WriteLine("(PartitionKey gt '" + startPKey + "'"
+ " and PartitionKey le '" + endPKey +"')"
+ " and (EventId eq 4096)"
);
}
private static string convertDateToPKey(string myDate)
{
System.DateTime dt = System.Convert.ToDateTime(myDate);
long dt2ticks = dt.Ticks;
string ticks = System.Convert.ToString(dt2ticks);
return "0" + ticks;
}
NB: for those, like me, who are searching so far away how to export results to a CSV file, you should know that this icon is your answer (and it's not a 'undo' ;) ):
In your query, you're filtering on Timestamp attribute which is not indexed (Only PartitionKey and RowKey attributes are indexed). Thus your query is making a full table scan (i.e. going from the 1st record till the time it finds a matching record) and hence not optimized.
In order to avoid full table scan, please use PartitionKey in your query. In case of WADWindowsEventLogsTable, the PartitionKey essentially represents the date/time value in ticks. What you would need to do is convert the date/time range for which you want to get the data into ticks, prepend a 0 in front of it and then use it in the query.
So your query would be something like:
(PartitionKey gt 'from date/time value in ticks prepended with 0' and PartitionKey le 'to date/time value in ticks prepended with 0') and (EventId eq 4096)
I wrote a blog post about it some time ago that you may find useful: http://gauravmantri.com/2012/02/17/effective-way-of-fetching-diagnostics-data-from-windows-azure-diagnostics-table-hint-use-partitionkey/
I am using Kayako Querying Language and trying to get the query to return all closed/resolved, closed/unresolved, and review tickets from a specified month. When I run the query, every ticket ever comes up, and it seems to ignore the date function I am using. What am I doing wrong?
SELECT COUNT(*) AS Total
FROM ‘Tickets'
WHERE ‘Tickets.Creation Date’ = month(June 2016) AND ‘Tickets.Status’ = ‘Closed’ OR ‘Tickets.Status’ = ‘Resolved’ OR ‘Tickets.Status’ = ‘Unresolved'
GROUP BY ‘Tickets.Status'
Thanks.
Replace the date comparison in your WHERE clause for this:
WHERE 'Tickets.Creation Date':Month = 6 and 'Tickets.Creation Date':Year = 2016
Be also careful with the quotes, you sometimes use ‘ and others '
In Spark I am trying to execute SQL queries on a temporary table derived from a data frame that I manually built by reading a csv file and converting the columns into the right data type.
Specifically, the table I'm talking about is the LINEITEM table from [TPC-H specification][1]. Unlike stated in the specification I am using TIMESTAMP rather than DATE because I've read that Spark does not support the DATE type.
In my single scala source file, after creating the data frame and registering a temporary table called "lineitem", I am trying to execute the following query:
val res = sqlContext.sql("SELECT * FROM lineitem l WHERE date(l.shipdate) <= date('1998-12-01 00:00:00');")
When I submit the packaged jar using spark-submit, I get the following error:
Exception in thread "main" java.lang.RuntimeException: [1.75] failure: ``union'' expected but but `;' found
When I omit the semicolon and do the same thing, I get the following error:
Exception in thread "main" java.util.NoSuchElementException: key not found: date
Spark version is 1.4.0.
Does anyone have an idea what's the problem with these queries?
[1] http://www.tpc.org/TPC_Documents_Current_Versions/pdf/tpch2.17.1.pdf
SQL queries passed to SQLContext.sql shouldn't be delimited using semicolon - this the source of your first problem
DATE UDF expects date in the YYYY-MM-DD form and DATE('1998-12-01 00:00:00') evaluates to null. As long as timestamp can be casted to DATE correct query string looks like this:
"SELECT * FROM lineitem l WHERE date(l.shipdate) <= date('1998-12-01')"
DATE is a Hive UDF. It means you have to use HiveContext not a standard SQLContext - this is the source of your second problem.
import org.apache.spark.sql.hive.HiveContext
val sqlContext = new HiveContext(sc) // where sc is a SparkContext
In Spark >= 1.5 it is also possible to use to_date function:
import org.apache.spark.sql.functions.{lit, to_date}
df.where(to_date($"shipdate") <= to_date(lit("1998-12-01")))
Please try hive function CAST (expression AS toDatatype)
It changes an expression from one datatype to other
e.g. CAST ('2016-06-17 00.00.000' AS DATE) will convert String to Date
In your case
val res = sqlContext.sql("SELECT * FROM lineitem l WHERE CAST(l.shipdate as DATE) <= CAST('1998-12-01 00:00:00' AS DATE);")
Supported datatype conversions are as listed in Hive Casting Dates