Automate PySpark SQL script with parameters - apache-spark-sql

Question

I have a PySpark Sql script that need to run daily and would like to pass the required parameters to the script and use them in my SQL queries inside the script.
For ex: below are the parameters that I would like to pass to the script and use them inside my script
my_st_dt='2019-02-04'
my_end_dt='2019-02-10'
mth_yyyymm='201902'
my_partition_dt='20190204'
my_table_name='table_1'
my_path='hdfs:///abcd/efgh/ijkl/mnop'
my_query1='''
SELECT * FROM parquet.`{my_file_path}/{my_table}/data`
WHERE {my_partition_name} = {partition} AND (my_date >= '{partition_st_dt}' AND
my_date <= '{partition_end_dt}')
'''.format(my_file_path=my_path,my_table=my_table_name,my_partition_name='my_yyyymm', \
partition=mth_yyyymm,partition_st_dt=my_st_dt, partition_end_dt=my_end_dt)
I have several queries like above in my script. Can someone please show me efficient way of writing this code such that I don't have to edit the script everytime I run? If there is any option other than Python ".format", then please do let me know as well. Thanks a lot in advance.

Answer 1

There are two ways we can do that:
Method 1: Use argsparse and pass all the required variables while executing the script.
parser = argparse.ArgumentParser(
description="Usage: spark2-submit ens_load_pb_sdps.py <sdp_name>")
parser.add_argument(
"-start_date", "--start_date"
type=date,
help='start date for query')
parser.add_argument(
"end_date", "--end_date"
type=date,
help='start date for query')
parser.add_argument(
"-table_name", "--table_name"
type=date,
help='start date for query')
parser.add_argument(
"-file_path", "--file_path"
type=date,
help='start date for query')
parser.add_argument(
"partition_name", "--partition_name"
type=date,
help='start date for query')
parser.add_argument(
"-partition_value", "--partition_value"
type=date,
help='start date for query')
args = parser.parse_args()
start_date, end_date, table_name, file_path, partition_name, partition_value = args.start_date, args.end_date, args.table_name, args.file_path, args.partition_name, args.partition_value
query_stmt="""
SELECT * FROM parquet.{file_path}/{table_name}/data
WHERE {partition_name} = {partition_value} AND (my_date >= '{start_date}' AND
my_date <= '{end_date}')
""".format(file_path=file_path, table_name=table_name, partition_name=partition_name, \
partition_value=partition_value, start_date=start_date, end_date=end_date)
Method2: Use a conf file
Also instead of passing the value while executing the scripts, we can keep the value in a conf json file and then load that json file into a python dict and then use it while forming the query.
Inside Json File - conf.json
{
"start_date": "2019-01-01",
"end_date": "2020-01-01",
"partition_name": "name",
"partition_value": "John Doe",
"file_path": "C://files/dummyPathh/",
"table_name": "dummyTable"
}
Python code to load the conf file -
file_path = "C://path/to/config" // path to conf file
json_file = glob.glob(file_path)
with open(json_file, 'r', encoding="utf8") as file:
conf_object = json.load(file)
start_date, end_date, table_name, file_path, partition_name, partition_value = conf_object.start_date, conf_object.end_date, conf_object.table_name, conf_object.file_path, conf_object.partition_name, conf_object.partition_value
query_stmt="""
SELECT * FROM parquet.{file_path}/{table_name}/data
WHERE {partition_name} = {partition_value} AND (my_date >= '{start_date}' AND
my_date <= '{end_date}')
""".format(file_path=file_path, table_name=table_name, partition_name=partition_name, \
partition_value=partition_value, start_date=start_date, end_date=end_date)
You can change the value in the conf file for running different queries.
To take it one step more furthur,
You can make a nested dict json conf file and then use it so that you don't have to edit the conf file each time we run the query.