Select oldest column for grouped spark dataframe - dataframe

Question

Given a dataframe (df) with the following columns:
id,
created_date,
name
I need to ensure that all rows with the same name have the same id. I can create a mapping from old id to new id (selected at 'random' using max).
df.groupBy('name')\
.agg(
func.max('id').alias('new_id'),
func.collect_set(id).alias('grouped_ids'))\
.filter(func.size('grouped_ids') > 1)\
.select(func.explode("grouped_ids").alias('old_id'), "new_id")\
.filter("new_id != old_id")
I can the leftouter join this to the original df (on id = old_id) and swap the ids if there is a new_id available.
However, I need to ensure that the new_id selected is the one with the oldest created_date in the dataframe (rather than just selecting the max).
How best to go about this?
e.g. Given the data
id, created_date, name
---
17a, 2019-01-05, Jeff
17a, 2019-01-03, Jeremy
d21, 2019-01-04, Jeremy
u45, 2019-01-04, Jeremy
d21, 2019-01-02, Scott
x22, 2019-01-01, Julian
Rows 2, 3 and 4 group on Jeremy so should have the same id. The oldest id in the dataframe for the grouped ids is d21 as on row 5 the created_date is 2019-01-02, so that should be selected and applied to all rows in the dataframe with the other grouped ids, and we end up with:
id, created_date, name
---
d21, 2019-01-05, Jeff
d21, 2019-01-03, Jeremy
d21, 2019-01-04, Jeremy
d21, 2019-01-04, Jeremy
d21, 2019-01-02, Scott
x22, 2019-01-01, Julian
UPDATE:
#Charles Du - Cheers, I tried your code but it didn't work out, the oldest id was selected from the grouped names, not the df as a whole and the new_id was not applied throughout the df.
Result:
0 = {Row} Row(name='Scott', created_date='2019-01-02', new_ID='d21', id='d21', created_date='2019-01-02')
1 = {Row} Row(name='Julian', created_date='2019-01-01', new_ID='x22', id='x22', created_date='2019-01-01')
2 = {Row} Row(name='Jeremy', created_date='2019-01-03', new_ID='17a', id='17a', created_date='2019-01-03')
3 = {Row} Row(name='Jeremy', created_date='2019-01-03', new_ID='17a', id='d21', created_date='2019-01-04')
4 = {Row} Row(name='Jeremy', created_date='2019-01-03', new_ID='17a', id='u45', created_date='2019-01-04')
5 = {Row} Row(name='Jeff', created_date='2019-01-05', new_ID='17a', id='17a', created_date='2019-01-05')

Answer 1

My spitball here
from pyspark.sql import functions as F
new_df = df.groupBy('name').agg(F.min('date'))
new_df = new_df.join(df, on=['name', 'date'], how='inner')
# This should give you a df with a single record for each name with the oldest ID.
new_df = new_df.withColumnRenamed('id', 'new_ID')
#you'll need to decide on a naming convention for your date column since you'll have two if you don't rename
res = new_df.join(df, on='name', how='inner)
that should match up your id with the oldest date.