I have a csv, and one column is date, format dd/mm/yyyy.
I read it using z=pd.read_csv('property_scrape.csv')
My raw data is:
After I read it in, some of the values are kept in the format I downloaded (dd/mm/yyyy), while somewhere in the middle, the dates are converted to yyyy-mm-dd:
27 01/10/2019
28 01/10/2019
29 01/10/2019
...
21092 2020-08-22
21093 2020-08-22
21094 2020-08-22
Name: Date, Length: 21122, dtype: object
Does anyone know why this happens?
Also, is there a way to ensure that this date column is always read the correct/constant way?
The problem is Pandas samples the first row and thinks it is MM/DD/YYYY instead of DD/MM -- there isn't a real way to know this. Then later when it finds 22, which not a valid MM it defaults to object/string.
You can add flag infer_datetime_format=False and it should read as strings then you can parse it -- you can pass a lambda to read_csv as well, not sure if an easier way to just pass a format string -- see article article dsexchange 34357
I have a csv file, the one from https://www.kaggle.com/jolasa/waves-measuring-buoys-data-mooloolaba/downloads/waves-measuring-buoys-data-mooloolaba.zip/1. The first entries look like this:
The first column has dates which I'm trying to read with this command:
matrix = dlmread ('waves-measuring-buoys-data/WavesMooloolabaJan2017toJun2019.csv',',',1,0);
(If referring to file on Kaggle, note that I slightly modified the directory and file names for ease of reading)
Then when I check a date by printing matrix(2,1), I get 1 instead of 01/01/2017 00:00.
How do I get the correct format?
csvread is only for numeric inputs.
Use csv2cell from the io package instead, to obtain your data as a string, and then perform any necessary string operatios and conversions accordingly.
I'm trying to parse an XML file where the users (in their infinite wisdom) type a key value into a free-form field, <Description>. The values are normally typed in with returns (BR's?) between them. For instance:
<Description>
% Increase: 27%
Completion Date: 10-Aug-2015
</Description>
I need to look for an extract the date following the string "Completion Date:" Looking around here on SO I found something similar and adapted it to:
compdate = deal.SelectSingleNode("./Terms/Description[substring-before(substring-after(.,'Completion Date:'),'/')]")
The problem is that in the original question there was a trailing character that could be used to delimit the text, a /. In my case, there might be a BR of some sort, or it might be the last (as in this case) or only item on the line and thus there's no delimiter.
So... suggestions on how to extract the date? I can do it on the VB side, but I'd like to remain in the XPath world for code clarity - unless of course the resulting XPath is unreadable.
If XPath 2.0 solution is acceptable, try
./Terms/Description/tokenize(substring-after(.,'Completion Date: '), '\n')[1]
If not and date format is always DD-mon-YYYY (e.g. 01-Dec-2018), try
./Terms/Description/substring(substring-after(.,'Completion Date: '), 1, 11)
I've csv file containing date in ISO format like below.
id,x1,x2,x3
AIR,Partner,2015-10-20T04:00:00.000Z,2015-10-20T04:00:00.000Z,2016-02-12T05:00:00.000Z
CMX,Partner,Tier,2017-03-23T04:00:00.000Z
WKA,Partner,Tier,2017-05-22T04:00:00.000Z
APP,Partner,2017-10-04T04:00:00.000Z,Tier
BUN,2017-09-27T04:00:00.000Z,Partner,,2017-09-27T04:00:00.000Z
There is no fix column for date it can appear in any column except 1st column.
I want to convert all occurrence of ISO date into DD-MON-YYYY or DD/MM/YYYY format.
Please help.
You can use the following through a UNIX pipe:
sed -E 's#\b([0-9]{4})-([0-9]{2})-([0-9]{2})T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}Z\b#\3/\2/\1#g'
Another option would be the following PCRE, which explicitly enforces the , separator at the start (as you mention this field cannot be the first one), and either , or EOL at the end of the matched expression:
cat | perl -pe 's#,\K([0-9]{4})-([0-9]{2})-([0-9]{2})T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}Z(?=,|$)#$3/$2/$1#g'
You can combine the previous answer with How to Print Spelled out month names rather than numbers to get the following two options, depending on which of the two answers you like best:
The first option is to not use any libraries, but just use the direct embedding of the names of the months:
cat 48941818.txt | perl -pe '#m = qw(Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec); s#,\K([0-9]{4})-([01][0-9])-([0-3][0-9])T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}Z(?=,|$)#$3-$m[$2]-$1#g'
The second option is to use http://perldoc.perl.org/POSIX.html#strftime as in C, and treat the replace part as perl code with the e modifier, as per Embedding evaluations in Perl regex, and, ultimately, as documented at http://perldoc.perl.org/perlrequick.html#Search-and-replace, which makes it extra flexible as you can do further improvements to the date format (although on my system %v didn't work, and %e had extra space in one-digit days).
% cat 48941818.txt | perl -pe 'use POSIX qw(strftime); s#,\K([0-9]{4})-([01][0-9])-([0-3][0-9])T([0-9]{2}):[0-9]{2}:[0-9]{2}.[0-9]{3}Z(?=,|$)#POSIX::strftime(qw/%e-%b-%Y/,0,0,$4,$3,$2-1,$1-1900)#ge'
id,x1,x2,x3
AIR,Partner,20-Oct-2015,20-Oct-2015,12-Feb-2016
CMX,Partner,Tier,23-Mar-2017
WKA,Partner,Tier,22-May-2017
APP,Partner, 4-Oct-2017,Tier
BUN,27-Sep-2017,Partner,,27-Sep-2017
%
I have banged my head against this for too long now!
I have a string: 2012-09-27T18:00:00.000-04:00
I have a format: [dateFormatter setDateFormat:#"yyyy-MM-dd'T'HH:mm:ss'Z'"];
I get a null result converting the string.
Can someone help with the correct date format?
According to the documentation for NSDateFormatter:
The format string uses the format patterns from the Unicode Technical Standard #35. The version of the standard varies with release of the operating system:
Formatters in OS X v10.8 and iOS 6.0 use [version tr35-25].
Following that link:
s 1..2 12 Second. Use one or two for zero padding.
In other words, s for seconds means only the integral part.
But right underneath that, there's:
S 1..n 3456 Fractional Second - truncates (like other time fields) to the count of letters. (example shows display using pattern SSSS for seconds value 12.34567)
Then, the reason you aren't matching the timezone is that you're quoting the Z, so it matches a literal Z rather than a timezone format.
To match the ISO8601 timezone format you're seeing, the same documentation says you want ZZZZZ.
So, it looks like, at least for 10.8/iOS 6, you want:
[dateFormatter setDateFormat:#"yyyy-MM-dd'T'HH:mm:ss'.'SSSZZZZZ"]
For earlier versions, you've got the links to the docs; you should be able to figure it out now.
Testing this (in Python, to save a few lines of code):
>>> import Cocoa
>>> df = Cocoa.NSDateFormatter.alloc().init()
>>> df.setDateFormat_("yyyy-MM-dd'T'HH:mm:ss'.'SSSZZZZZ")
>>> print df.dateFromString_(2012-09-27T18:00:00.000-04:00')
2012-09-27 22:00:00 +0000