How to get apache airflow to render Hive HQL ${variables} with Jinja - hive

Question

It seems like this is supported where you pass in a HQL script with ${xxx} vars and it gets preprocessed to convert them to {{xxx}} Jinja-style before the stage that actually does the template rendering to then replace those with values from a user-supplied dictionary. I believe this because there is a function like this in the HiveOperator class:
def prepare_template(self):
if self.hiveconf_jinja_translate:
self.hql = re.sub(
"(\$\{([ a-zA-Z0-9_]*)\})", "{{ \g<2> }}", self.hql)
if self.script_begin_tag and self.script_begin_tag in self.hql:
self.hql = "\n".join(self.hql.split(self.script_begin_tag)[1:])
The problem is I cannot figure out how to trigger this piece of code to get called before the template rendering stage. I have a basic DAG script like this:
from airflow import DAG
from airflow.operators.hive_operator import HiveOperator
from datetime import datetime, timedelta
default_args = dict(
owner='mpetronic',
depends_on_past=False,
start_date=datetime(2017, 5, 2),
verbose=True,
retries=1,
retry_delay=timedelta(minutes=5)
)
dag = DAG(
dag_id='report',
schedule_interval='* * * * *',
user_defined_macros=dict(a=1, b=2),
default_args=default_args)
hql = open('/home/mpetronic/repos/airflow/resources/hql/report.hql').read()
task = HiveOperator(
task_id='report_builder',
hive_cli_conn_id='hive_dv',
schema='default',
mapred_job_name='report_builder',
hiveconf_jinja_translate=True,
dag=dag,
hql=hql)
I can see that my user_defined_macros dictionary makes it to the code where it gets merged with a global jinja context dictionary that then is applied to my HQL script to render it as a template. However, because my HQL is native HQL, all my variables that I want to update are in the form of ${xxx} and jinja just skips over them. I need airflow to call prepare_template() first but just don't see how to make that happen.
I realize I could just manually alter my HQL ${xxx} to {{xxx}} as that works but that seems like an anti-pattern. I want the script to be able to work natively or via airflow. This is the function, in the TaskInstance class, the does render my manually altered {{xxx}} values:
def render_templates(self):
task = self.task
jinja_context = self.get_template_context()
if hasattr(self, 'task') and hasattr(self.task, 'dag'):
if self.task.dag.user_defined_macros:
jinja_context.update(
self.task.dag.user_defined_macros)
rt = self.task.render_template # shortcut to method
for attr in task.__class__.template_fields:
content = getattr(task, attr)
if content:
rendered_content = rt(attr, content, jinja_context)
setattr(task, attr, rendered_content)

Answer 1

I figured out my issue. This is the regex used in the noted method:
(\$\{([ a-zA-Z0-9_]*)\})
It does not account for beeline variables in the form of:
${hivevar:var_name}
It does not consider a colon in the pattern. That form is the more standard way of defining Hive variables within a namespace using beeline. To make this Jinja substitution work, you have to reference variables in the HQL using just ${var_name} but you can only define variables in beeline using:
set hivevar:var_name=123;
I think that Airflow should fully support the hivevar:var_name style of namespaced variables when you are running with beeline given beeline is the preferred client to use with Hive.