I am trying to mimic the solution in this answer post: Sqlite convert string to date
The problem I am running into though is that my raw text date field contains dates in the following format:
1/5/2014 0:00:00
instead of:
01/05/2014 0:00:00
How can I account for this single digit month/day variability and adapt the code for my purpose? I cannot think of a decent way. I have thought of using excel to create a conversion table that I could join upon. Is there a more elegant approach?
Edit on 10/2/2014:
I would have thought there would be more like 81 possible combinations to identify. Or Maybe there is a simpler way I am not thinking of?
1/1/
1/2/
1/3/
1/4/
1/5/
1/6/
1/7/
1/8/
1/9/
2/1/
2/2/
2/3/
2/4/
2/5/
2/6/
2/7/
2/8/
2/9/
...
There are eight different combinations of one- or two-digit fields; you just have to check for all of them:
SELECT CASE
WHEN Trans_Date LIKE '_/_/____ _:__:__' THEN substr(Trans_Date, 10, 1)
WHEN Trans_Date LIKE '_/_/____ __:__:__' THEN substr(Trans_Date, 10, 2)
...
END AS Hour,
...
FROM LS2014
Related
I have a table like this in SQLITE3:
I need to query this table by ID|DOC_ID|TRANS_DOC_ID and most importantly by DATE because I need to get the data day by day. ex: TODAY|YESTERDAY|ETC
So far the query is easy, as I can just do this to get the rows by day:
SELECT * FROM CLIENTRECORD WHERE DATE = '2020-12-01'
The problem is when I need to display specific records on other dates:
ex: I have a row with DATE 2020-12-01 but I also want it displayed on DATE 2020-01-01 or maybe 2020-01-02, etc. What do I do in this situation? and so I thought about adding another col as DATES which was supposed to be an array of comma-separated dates BUT I researched that this is a BAD solution, I also thought about adding a separate TABLE just for dates but since the dates aren't fixed (they might contain 1 date or maybe even 10 who knows), I am confused as to what I am supposed to do.
The end goal is that a row may or may not contain more than 1 date, would look something like this if I want to query for the row with or without multiple dates:
SELECT * FROM CLIENTRECORD WHERE DATE = '2020-12-01' OR DATES LIKE '2020-12-01'
something similar to it.
I'm struggling to add a where-clause based on the difference of the current date and the date of the record entry. There are some other simplistic clauses as well, but I have no problem with those, so I'm making the demo data simple to highlight the issue I'm having.
Demo dataset:
Rec_no Rec_date
77 20170606
69 20170605
55 20170601
33 20170520
29 20170501
Date is recorded in format yyyymmdd and I'd like to build a where clause to only show records that are created X number of days ago from current date - lets say 10.
So, in this case, only records no 33 and 29 should be shown.
Unfortunately I'm not sure on what the actual DB engine is, but it should be something from IBM.
How could this be done?
As suggested in the comments updating the schema to store the date at the correct type is the best option before you start, however if this is not possible for whatever reason, you would first need to convert the stored date to the correct format at runtime.
I'll write an example in t-sql as that's what I know. Once you have worked out your dbms i can edit to the relevant functions/syntax
Select *
FROM Demodataset
WHERE Cast(Rec_Date as datetime) >= dateadd(day,-10,getdate())
Here trip 1 involves 2 activity_code in a single day and also concludes in a single day and most other activities are just single day but i have one trip that span over more than one day.
What could be the best possible way to store date range for that column that span more than one days.
Splitting the column into multiple begin date and end date just doesn't make sense as there would be many blank columns?
trip_id(pk,fk) Activity_code(pk,fk) date
1 a1 1st October 2015
1 a2 1st October 2015
2 a3 2nd -5th October 2015
Keep in mind that i need to search the activity_code on basis of month. such as list all the activity code that occur in October ?
Is it possible to insert a range of date in a single column or any other design solution ?
Is there any datatype that can represent the date range in single value ?
PS: oracle 11g e
Store the date ranges as FirstDate/LastDate or FirstDate/Duration.
This allows you to store the values in the native format for dates. Storing dates as strings is a bad, bad idea, because strings don't have all the built-in functionality provided for native date types.
Don't worry about the additional storage for a second date or duration. In fact, the two columns together are probably smaller than storing the value as a string.
Splitting the date into start date and end date would be ideal. Storing dates as strings is not recommended. If you store your dates as strings then there is a possibility of malformed data being stored in the column since a VARCHAR2 column will allow any value. You will have to build strong validations in your script while inserting the data which is unnecessary.
Secondly, you will not be able to perform simple operations like calculating the duration/length of the trip easily if both the start_date and end_date are stored in the same column. If they are stored in different columns it would be as simple as
SELECT trip_id, activity_code, end_date - start_date FROM trips;
suppose I have a table MyTable with a column some_date (date type of course) and I want to select the newest 3 months data (or x days).
What is the best way to achieve this?
Please notice that the date should not be measured from today but rather from the date range in the table (which might be older then today)
I need to find the maximum date and compare it to each row - if the difference is less than x days, return it.
All of this should be done with sqlalchemy and without loading the entire table.
What is the best way of doing it? must I have a subquery to find the maximum date? How do I select last X days?
Any help is appreciated.
EDIT:
The following query works in Oracle but seems inefficient (is max calculated for each row?) and I don't think that it'll work for all dialects:
select * from my_table where (select max(some_date) from my_table) - some_date < 10
You can do this in a single query and without resorting to creating datediff.
Here is an example I used for getting everything in the past day:
one_day = timedelta(hours=24)
one_day_ago = datetime.now() - one_day
Message.query.filter(Message.created > one_day_ago).all()
You can adapt the timedelta to whatever time range you are interested in.
UPDATE
Upon re-reading your question it looks like I failed to take into account the fact that you want to compare two dates which are in the database rather than today's day. I'm pretty sure that this sort of behavior is going to be database specific. In Postgres, you can use straightforward arithmetic.
Operations with DATEs
1. The difference between two DATES is always an INTEGER, representing the number of DAYS difference
DATE '1999-12-30' - DATE '1999-12-11' = INTEGER 19
You may add or subtract an INTEGER to a DATE to produce another DATE
DATE '1999-12-11' + INTEGER 19 = DATE '1999-12-30'
You're probably using timestamps if you are storing dates in postgres. Doing math with timestamps produces an interval object. Sqlalachemy works with timedeltas as a representation of intervals. So you could do something like:
one_day = timedelta(hours=24)
Model.query.join(ModelB, Model.created - ModelB.created < interval)
I haven't tested this exactly, but I've done things like this and they have worked.
I ended up doing two selects - one to get the max date and another to get the data
using the datediff recipe from this thread I added a datediff function and using the query q = session.query(MyTable).filter(datediff(max_date, some_date) < 10)
I still don't think this is the best way, but untill someone proves me wrong, it will have to do...
I'm trying to make a graph in Rails, for example the avg sales amount per day for each day in a given date range
Say I have a products_sold model which has a "sales_price" float attribute. But if a specific day has no sales (e.g none in the model/db), I want to return simply 0.
What's the best way in MySQL/Rails to get this done? I know I can do something like this:
(This SQL query might be the completely wrong way to get what I'm wanting too)
SELECT avg(sales_price) AS avg, DATE_FORMAT(created_at, '%m-%d-%Y') AS date
FROM products_sold WHERE merchant_id = 1 GROUP BY date;
And get results like this:
| avg | date |
23 01-03-2009
50 01-05-2009
34 01-07-2009
... ...
What I'd like to get is this:
| avg | date |
23 01-03-2009
0 01-04-2009
50 01-05-2009
0 01-06-2009
34 01-07-2009
0 01-08-2009
... ...
Can I do this with SQL or will I have to post-process the results to find what dates in the daterange aren't in the SQL result set? Perhaps I need some sub-selects or IF statements?
Thanks for any help everyone.
Is there a reason (other than the date one already mentioned) why you wouldn't use the built-in group function capabilities in ActiveRecord? You seem to be concerned about "post-processing", which I don't think is really something to worry about.
You're in Rails, so you should probably be looking for a Rails solution first[1]. My first thought would be to do something like
Product.average(:sales_price, :group => "DATE(created_at)", :conditions => ["merchant_id=?", 1])
which ActiveRecord turned into pretty much the SQL you described. Assuming there's a declared has_many association between Merchant and Product, then you'd probably be better using that, so something like:
ave_prices = Merchant.find(1).products.average(:sales_price, :group => "DATE(created_at)")
(I'm hoping that your description of the model as "products_sold" is some kind of transcription error, btw - if not, you're somewhat off-message with your class naming!)
After all that, you're back where you started, but you got there in a more conventional Rails way (and Rails really values conventions!). Now we need to fill in the gaps.
I'll assume you know your date range, let's say it's defined as all dates from from_date to to_date.
date_aves = (from_date..to_date).map{|dt| [dt, 0]}
That builds the complete list of dates as an array. We don't need the dates where we got an average:
ave_price_dates = ave_prices.collect{|ave_price| ave_price[0]} # build an array of dates
date_aves.delete_if { |dt| ave_price.dates.index(dt[0]) } # remove zero entries for dates retrieved from DB
date_aves.concat(ave_prices) # add the query results
date_aves.sort_by{|ave| ave[0] } # sort by date
That lot looks a bit cluttered to me: I think it could be terser and cleaner. I'd investigate building a Hash or Struct rather than staying in arrays.
[1] I'm not saying don't use SQL - situations do occur where ActiveRecord can't generate the most efficient query and you fall back on find_by_sql. That's fine, it's supposed to be like that, but I think you should try to use it only as a last resort.
For any such query, you will need to find a mechanism to generate a table with one row for each date that you want to report on. Then you will do an outer join of that table with the data table you are analyzing. You may also have to play with NVL or COALESCE to convert nulls into zeroes.
The hard part is working out how to generate the (temporary) table that contains the list of dates for the range you need to analyze. That is DBMS-specific.
Your idea of mapping date/time values to a single date is spot on, though. You'd need to pull a similar trick - mapping all the dates to an ISO 8601 date format like 2009-W01 for week 01 - if you wanted to analyze weekly sales.
Also, you would do better to map your DATE format to 2009-01-08 notation because then you can sort in date order using a plain character sort.
To dry up a bit:
ave_prices = Merchant.find(1).products.average(:sales_price, :group => "DATE(created_at)")
date_aves = (from_date..to_date).map{|dt| [dt, ave_prices[dt.strftime "%Y-%m-%d"] || 0]}
Does MySQL have set-returning functions? I.e. functions that return different values on each row of a query? As an example from PostgreSQL, you can do:
select 'foo', generate_series(3, 5);
This will produce a result set consisting of 2 columns and 3 rows, where the left column contains 'foo' on each row and the right column contains 3, 4 and 5.
So, assuming you have an equivalent of generate_series() in MySQL, and subqueries: What you need is a LEFT OUTER JOIN from this function to the query that you already have. That will ensure you see each date appear in the output:
SELECT
avg(sales_price) as avg,
DATE_FORMAT(the_date, '%m-%d-%Y') as date
FROM (select cast('2008-JAN-01' as date) + generate_series(0, 364) as the_date) date_range
LEFT OUTER JOIN products_sold on (the_date = created_at)
WHERE merchant_id = 1
GROUP BY date;
You may need to fiddle with this a bit to get the syntax right for MySQL.