Calculate age in months - optimize date transformations in pandas - pandas

Question

I am trying a very simple thing - to calculate age in months between two columns and save it to a new column
df['AGE'] = (df.apply(lambda x: (x['DAX'].year - int(x['BIRTH_DATE'][:4])) * 12 +
x['DAX'].month - int(x['BIRTH_DATE'][5:7])
if x['BIRTH_DATE'] is not None
and int(x['BIRTH_DATE'][:4]) > 1900
else -1 # data quality
, axis=1).astype(np.int8))
I am doing this while loading a pretty big 2Gb csv file. DAX is parsed directly in the reader while BIRTH_DATE is left as string.
And this simple calculation bumps up the load time by a factor of x10. Is there any smarter way to calculate age in months on big data frames?
Here is a sample of data:
DAX BIRTH_DATE
2015-01-01 1931-12-03
2015-01-01 1991-04-19
2015-01-01 1992-10-11
2015-01-01 1982-05-20
2015-01-01 1987-12-20
2015-01-01 1976-07-30
2015-01-01 1951-05-11
2015-01-01 1993-05-06
2015-01-01 1989-02-27
I am trying to get another column 'AGE' as number of months since birthday.

Answer 1

first convert BIRTH_DATE to datetime dtype:
In [257]: df['BIRTH_DATE'] = pd.to_datetime(df['BIRTH_DATE'], errors='coerce')
check:
In [258]: df.dtypes
Out[258]:
DAX datetime64[ns]
BIRTH_DATE datetime64[ns]
dtype: object
now we can do this simple math:
In [259]: df['AGE'] = df.DAX.dt.year*12 + df.DAX.dt.month - \
(df.BIRTH_DATE.dt.year*12 + df.BIRTH_DATE.dt.month)
In [260]: df
Out[260]:
DAX BIRTH_DATE AGE
0 2015-01-01 1931-12-03 997
1 2015-01-01 1991-04-19 285
2 2015-01-01 1992-10-11 267
3 2015-01-01 1982-05-20 392
4 2015-01-01 1987-12-20 325
5 2015-01-01 1976-07-30 462
6 2015-01-01 1951-05-11 764
7 2015-01-01 1993-05-06 260
8 2015-01-01 1989-02-27 311

Answer 2

As you have not provided any sample data I'm not completely sure what format your data is in. Something like this should work though and be considerably faster than using apply():
df['AGE'] = (df.DAX - df.BIRTH_DATE.astype('datetime64[ns]')).dt.days / 30
Again without the data I'm not sure what you're data quality steps needs to do, but it will probably be fixed after the above line like so:
df.loc[df['AGE'].isnull(), 'AGE'] = -1