SQL Server 2012 job scheduling - sql

I have a situation where I have 1 package (Package A) that take about 3 min to complete and another package (Package B) that takes about 4 min to complete. I need package A to run every 5 min via a job and package B to run every 15 min. However package B can only run after package A has been completed.
Example
5- Package A runs
10- Package A runs
15- Package A runs and then Package B
20- Package A runs
25- Package A runs
30- Package A runs and then Package B
Is there any way to schedule jobs to meet the requirements above?
Really appreciate your help. Thanks in advance!

for package B i would just put in logic to check if package A has stopped running
the wait script would be
WHILE EXISTS (
SELECT
*
FROM
msdb.dbo.sysjobs_view job
JOIN
msdb.dbo.sysjobactivity activity
ON
job.job_id = activity.job_id
JOIN
msdb.dbo.syssessions sess
ON
sess.session_id = activity.session_id
JOIN
(
SELECT
MAX( agent_start_date ) AS max_agent_start_date
FROM
msdb.dbo.syssessions
) sess_max
ON
sess.agent_start_date = sess_max.max_agent_start_date
WHERE
run_requested_date IS NOT NULL AND stop_execution_date IS NULL
AND job.name = 'package A')
BEGIN
WAITFOR DELAY '00:00:30';
END
This would check every 30 seconds if package A has stopped running.
Alternatively, you create a table that would keep track of job status and have A write to it to keep track of the status.

Related

SQL Server - keep restarting EXEC as long as dependent job completes

I need to automate element in my backups schedule for differential copies.
DIFF must be performed only when job_a is not running.
The only moment when below code clashes with job_a is when job_a may be still running in scheduled time for job_b to start. I need some pattern to keep re-running job_b till job_a won't be returned by first NOT EXISTS
I did a test where job_a starts run one second before job_b. While job_b should be trying to restart as long as job_a disappears from system logs, it abandons the retries.
IF NOT EXISTS
(SELECT job.name FROM msdb.dbo.sysjobs_view job
INNER JOIN msdb.dbo.sysjobactivity activity ON job.job_id = activity.job_id
INNER JOIN msdb.dbo.syssessions sess ON sess.session_id = activity.session_id
INNER JOIN (SELECT MAX(agent_start_date) AS max_agent_start_date FROM msdb.dbo.syssessions) sess_max ON sess.agent_start_date = sess_max.max_agent_start_date
WHERE run_requested_date IS NOT NULL
AND stop_execution_date IS NULL
AND job.name = 'job_a')
BEGIN
DECLARE #path_diff varchar(max)
SET #path_diff = 'H:\MSSQL13.MSSQLSERVER\MSSQL\Backup\dev\db_name' + CONVERT(varchar, GETDATE(), 112) + REPLACE(CONVERT(varchar, GETDATE(), 114),':','') + '_DIFF_COMP.BAK'
BACKUP DATABASE [db_name] TO DISK = #path_diff WITH COMPRESSION, DIFFERENTIAL , STATS = 10
END
--need a step that will re-start the job once job_a doesn't appear in NOT EXISTS
ELSE
EXEC msdb.dbo.sp_stop_job N'job_b'
PRINT 'Full backup is taking longer then usual, please inspect!'
EXEC msdb.dbo.sp_start_job N'job_b'

How to iterate many Hive scripts over spark

I have many hive scripts (somewhat 20-25 scripts), each scripts having multiple queries. I want to run these scripts using spark so that the process can run faster. As map reduce job in hive takes long time to execute from spark it will be much faster. Below is the code I have written but its working for 3-4 files but when given multiple files with multiple queries its getting failed.
Below is the code for the same. Please help me if possible to optimize the same.
val spark = SparkSession.builder.master("yarn").appName("my app").enableHiveSupport().getOrCreate()
val filename = new java.io.File("/mapr/tmp/validation_script/").listFiles.filter(_.getName.endsWith(".hql")).toList
for ( i <- 0 to filename.length - 1)
{
val filename1 = filename(i)
scala.io.Source.fromFile(filename1).getLines()
.filterNot(_.isEmpty) // filter out empty lines
.foreach(query =>
spark.sql(query))
}
some of the error I cam getting is like
ERROR SparkSubmit: Job aborted.
org.apache.spark.SparkException: Job aborted.
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:224)
ERROR FileFormatWriter: Aborting job null.
org.apache.spark.SparkException: Job aborted due to stage failure: ShuffleMapStage 12 (sql at validationtest.scala:67) has failed the maximum allowable number of times: 4. Most recent failure reason: org.apache.spark.shuffle.FetchFailedException: failed to allocate 16777216 byte(s) of direct memory (used: 1023410176, max: 1029177344) at org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:528)
many different types of error I get when run the same code multiple times.
Below is how one of the HQL file looks like. its name is xyz.hql and has
drop table pontis_analyst.daydiff_log_sms_distribution
create table pontis_analyst.daydiff_log_sms_distribution as select round(datediff(date_sub(current_date(),cast(date_format(CURRENT_DATE ,'u') as int) ),cast(subscriberActivationDate as date))/7,4) as daydiff,subscriberkey as key from pontis_analytics.prepaidsubscriptionauditlog
drop table pontis_analyst.weekly_sms_usage_distribution
create table pontis_analyst.weekly_sms_usage_distribution as select sum(event_count_ge) as eventsum,subscriber_key from pontis_analytics.factadhprepaidsubscriptionsmsevent where effective_date_ge_prt < date_sub(current_date(),cast(date_format(CURRENT_DATE ,'u') as int) - 1 ) and effective_date_ge_prt >= date_sub(date_sub(current_date(),cast(date_format(CURRENT_DATE ,'u') as int) ),84) group by subscriber_key;
drop table pontis_analyst.daydiff_sms_distribution
create table pontis_analyst.daydiff_sms_distribution as select day.daydiff,sms.subscriber_key,sms.eventsum from pontis_analyst.daydiff_log_sms_distribution day inner join pontis_analyst.weekly_sms_usage_distribution sms on day.key=sms.subscriber_key
drop table pontis_analyst.weekly_sms_usage_final_distribution
create table pontis_analyst.weekly_sms_usage_final_distribution as select spp.subscriberkey as key, case when spp.tenure < 3 then round((lb.eventsum )/dayDiff,4) when spp.tenure >= 3 then round(lb.eventsum/12,4)end as result from pontis_analyst.daydiff_sms_distribution lb inner join pontis_analytics.prepaidsubscriptionsubscriberprofilepanel spp on spp.subscriberkey = lb.subscriber_key
INSERT INTO TABLE pontis_analyst.validatedfinalResult select 'prepaidsubscriptionsubscriberprofilepanel' as fileName, 'average_weekly_sms_last_12_weeks' as attributeName, tbl1_1.isEqual as isEqual, tbl1_1.isEqualCount as isEqualCount, tbl1_2.countAll as countAll, (tbl1_1.isEqualCount/tbl1_2.countAll)* 100 as percentage from (select tbl1_0.isEqual as isEqual, count(isEqual) as isEqualCount from (select case when round(aal.result) = round(srctbl.average_weekly_sms_last_12_weeks) then 1 when aal.result is null then 1 when aal.result = 'NULL' and srctbl.average_weekly_sms_last_12_weeks = '' then 1 when aal.result = '' and srctbl.average_weekly_sms_last_12_weeks = '' then 1 when aal.result is null and srctbl.average_weekly_sms_last_12_weeks = '' then 1 when aal.result is null and srctbl.average_weekly_sms_last_12_weeks is null then 1 else 0 end as isEqual from pontis_analytics.prepaidsubscriptionsubscriberprofilepanel srctbl left join pontis_analyst.weekly_sms_usage_final_distribution aal on srctbl.subscriberkey = aal.key) tbl1_0 group by tbl1_0.isEqual) tbl1_1 inner join (select count(*) as countAll from pontis_analytics.prepaidsubscriptionsubscriberprofilepanel) tbl1_2 on 1=1
Your issue is your code is running out of memory as shown below
failed to allocate 16777216 byte(s) of direct memory (used: 1023410176, max: 1029177344)
Though what you are trying to do is not optimal way of doing things in Spark but I would recommend that you remove the memory serialization as it will not help in anyways. You should cache data only if it is going to be used in multiple transformations. If it is going to be used once there is no reason to put the data in cache.

How do I get the user who invoked a SQL Agent job?

I'm working on a job step which needs to reference the Login Name value for the user who started the job. It'll be passed as a value into a table field, which is then referenced by other processes.
I haven't been able to find if or where the starting username is stored in system tables. It must be stored somewhere as the (job outcome) step in msdb.dbo.sysjobhistory references is thusly:
The job succeeded. The Job was invoked by User [domain]\[username]. The last step to run was...
I'd like to capture that [domain]\[username] value for my job step. I know it can be done as a subsequent job, referencing the original and doing some string manipulation, but I'd prefer to do it in the same job, if possible.
Thanks in advance!
That is logged in the message column of the msdb.dbo.sysjobhistory table.
You may want to do some joins to the sysjobs table as well. I.E.
select
j.name as 'JobName',
run_date,
run_time,
msdb.dbo.agent_datetime(run_date, run_time) as 'RunDateTime',
case
when left(h.message,16) = 'Executed as user'
then substring(h.message,19,charindex('.',h.message,1) - 18)
when h.message like '%invoked by%'
then substring(h.message,charindex('invoked by ',h.message) + 11,charindex('.',substring(h.message,charindex('invoked by ',h.message) + 11,99)) - 1)
else h.message
end
From
msdb.dbo.sysjobs j
INNER JOIN
msdb.dbo.sysjobhistory h
ON j.job_id = h.job_id
where
j.enabled = 1 --Only Enabled Jobs
order by
JobName,
RunDateTime desc
REF

Running two sql server jobs into a single job not working

we have a sql server agent job which is running two sql servant jobs.The job is designed in the way that the second job will wait till the first job completed. But sometimes the wait is not working and it causes both jobs ends up running at same time. This causes failure of both jobs. We uses below code for waiting mechanism.
WHILE
(
SELECT COUNT(*) AS cnt
FROM msdb.dbo.sysjobs J
INNER JOIN msdb.dbo.sysjobactivity JA
ON J.job_id = JA.job_id
WHERE start_execution_date IS NOT NULL
AND stop_execution_date IS NULL
AND J.name = "first job name"
AND last_executed_step_date > DATEADD(DD,-1,GETDATE())--//Added to bypass historical corrupt data
) > 0
BEGIN
--WAIT
WAITFOR DELAY '00:01';
END
any suggestion why this is not working sometimes?

Long running job

I am not sure what is going on? I still get an email even though the job stopped running at 11:50 pm which clearly is less than thean 7 hours being that it started at 7 pm last night..It is suppose to email me if the job runs longer than 7 hours and for some reason yesterday it worked in test but today its not..Any ideas???
SELECT *
FROM msdb..sysjobactivity aj
JOIN msdb..sysjobs sj on sj.job_id = aj.job_id
WHERE DATEDIFF(HOUR,aj.start_execution_date,GetDate())> 7
AND aj.start_execution_date IS NOT NULL
AND sj.name = 'Nightly_Job'
and not exists ( -- make sure this is the most recent run
select 1
from msdb..sysjobactivity new
where new.job_id = aj.job_id
and new.start_execution_date > aj.start_execution_date)
if ##ROWCOUNT > 0
BEGIN
USE msdb
EXEC sp_send_dbmail
#profile_name = 'DB_Mail',
#recipients = 'xxx#yyy.com',
#subject = 'T-SQL Query Result',
#body = 'The Nightly_Job has been running 7
The problem with your query is that it only checks how long has passed since the job was started... it doesn't take into account when the job completed.
If there are no other executions of the job within 7 hours, then your check that limits the query to the most recent run doesn't exclude the previously completed job... your query then returns the job that was started over 7 hours ago, but in this case completed within 4 hours.
To fix your query to exclude completed jobs, modify your query as follows:
SELECT *
FROM
msdb..sysjobactivity aj
JOIN msdb..sysjobs sj on sj.job_id = aj.job_id
WHERE
DATEDIFF(HOUR, aj.start_execution_date,
ISNULL(aj.stop_execution_date, GetDate()) )> 7
AND sj.name = 'Nightly_Job'
AND NOT EXISTS ( -- make sure this is the most recent run
SELECT 1
FROM msdb..sysjobactivity new
WHERE new.job_id = aj.job_id
AND new.start_execution_date > aj.start_execution_date)