I have some maintenance job, here is structure:
Step 1 and 3 just checking where are
Step 1 and 3 just checking where is specific database is currently resides - is it primary or secondary replica
If sys.fn_hadr_is_primary_replica ( 'DATABASENAME') <> 1
BEGIN
RAISERROR ('Not PRIMARY REPLICA FOR DATABASE NAME', -- Message text.
16, -- Severity.
1 -- State.
);
END
STEP 2 and 4 just performs backup of specific database using Ola script
EXECUTE [dbo].[DatabaseBackup]
#Databases = 'SOMEDB',
#Directory = 'SOMESHARE',
#BackupType = 'FULL',
#Verify = 'Y',
#CleanupTime = 336,
#CheckSum = 'Y',
#LogToTable = 'Y',
#Compress = 'Y'
Step 1 moves job execution to Step 3 in case of database is not on primary replica now
Step 3 quit with success in case of failure
This works well on secondary replica and job quit with success with failed first and third step
But on primary replica there absolutely strange things happening and SQL agent can mix step or doing logic that should not happen anyway. Looks like following
Just rebooted the server and now everything working fine , its probably happened because of installing multiple windows and sql server patches since last 5 days without reboot whole server.
Related
I have many hive scripts (somewhat 20-25 scripts), each scripts having multiple queries. I want to run these scripts using spark so that the process can run faster. As map reduce job in hive takes long time to execute from spark it will be much faster. Below is the code I have written but its working for 3-4 files but when given multiple files with multiple queries its getting failed.
Below is the code for the same. Please help me if possible to optimize the same.
val spark = SparkSession.builder.master("yarn").appName("my app").enableHiveSupport().getOrCreate()
val filename = new java.io.File("/mapr/tmp/validation_script/").listFiles.filter(_.getName.endsWith(".hql")).toList
for ( i <- 0 to filename.length - 1)
{
val filename1 = filename(i)
scala.io.Source.fromFile(filename1).getLines()
.filterNot(_.isEmpty) // filter out empty lines
.foreach(query =>
spark.sql(query))
}
some of the error I cam getting is like
ERROR SparkSubmit: Job aborted.
org.apache.spark.SparkException: Job aborted.
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:224)
ERROR FileFormatWriter: Aborting job null.
org.apache.spark.SparkException: Job aborted due to stage failure: ShuffleMapStage 12 (sql at validationtest.scala:67) has failed the maximum allowable number of times: 4. Most recent failure reason: org.apache.spark.shuffle.FetchFailedException: failed to allocate 16777216 byte(s) of direct memory (used: 1023410176, max: 1029177344) at org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:528)
many different types of error I get when run the same code multiple times.
Below is how one of the HQL file looks like. its name is xyz.hql and has
drop table pontis_analyst.daydiff_log_sms_distribution
create table pontis_analyst.daydiff_log_sms_distribution as select round(datediff(date_sub(current_date(),cast(date_format(CURRENT_DATE ,'u') as int) ),cast(subscriberActivationDate as date))/7,4) as daydiff,subscriberkey as key from pontis_analytics.prepaidsubscriptionauditlog
drop table pontis_analyst.weekly_sms_usage_distribution
create table pontis_analyst.weekly_sms_usage_distribution as select sum(event_count_ge) as eventsum,subscriber_key from pontis_analytics.factadhprepaidsubscriptionsmsevent where effective_date_ge_prt < date_sub(current_date(),cast(date_format(CURRENT_DATE ,'u') as int) - 1 ) and effective_date_ge_prt >= date_sub(date_sub(current_date(),cast(date_format(CURRENT_DATE ,'u') as int) ),84) group by subscriber_key;
drop table pontis_analyst.daydiff_sms_distribution
create table pontis_analyst.daydiff_sms_distribution as select day.daydiff,sms.subscriber_key,sms.eventsum from pontis_analyst.daydiff_log_sms_distribution day inner join pontis_analyst.weekly_sms_usage_distribution sms on day.key=sms.subscriber_key
drop table pontis_analyst.weekly_sms_usage_final_distribution
create table pontis_analyst.weekly_sms_usage_final_distribution as select spp.subscriberkey as key, case when spp.tenure < 3 then round((lb.eventsum )/dayDiff,4) when spp.tenure >= 3 then round(lb.eventsum/12,4)end as result from pontis_analyst.daydiff_sms_distribution lb inner join pontis_analytics.prepaidsubscriptionsubscriberprofilepanel spp on spp.subscriberkey = lb.subscriber_key
INSERT INTO TABLE pontis_analyst.validatedfinalResult select 'prepaidsubscriptionsubscriberprofilepanel' as fileName, 'average_weekly_sms_last_12_weeks' as attributeName, tbl1_1.isEqual as isEqual, tbl1_1.isEqualCount as isEqualCount, tbl1_2.countAll as countAll, (tbl1_1.isEqualCount/tbl1_2.countAll)* 100 as percentage from (select tbl1_0.isEqual as isEqual, count(isEqual) as isEqualCount from (select case when round(aal.result) = round(srctbl.average_weekly_sms_last_12_weeks) then 1 when aal.result is null then 1 when aal.result = 'NULL' and srctbl.average_weekly_sms_last_12_weeks = '' then 1 when aal.result = '' and srctbl.average_weekly_sms_last_12_weeks = '' then 1 when aal.result is null and srctbl.average_weekly_sms_last_12_weeks = '' then 1 when aal.result is null and srctbl.average_weekly_sms_last_12_weeks is null then 1 else 0 end as isEqual from pontis_analytics.prepaidsubscriptionsubscriberprofilepanel srctbl left join pontis_analyst.weekly_sms_usage_final_distribution aal on srctbl.subscriberkey = aal.key) tbl1_0 group by tbl1_0.isEqual) tbl1_1 inner join (select count(*) as countAll from pontis_analytics.prepaidsubscriptionsubscriberprofilepanel) tbl1_2 on 1=1
Your issue is your code is running out of memory as shown below
failed to allocate 16777216 byte(s) of direct memory (used: 1023410176, max: 1029177344)
Though what you are trying to do is not optimal way of doing things in Spark but I would recommend that you remove the memory serialization as it will not help in anyways. You should cache data only if it is going to be used in multiple transformations. If it is going to be used once there is no reason to put the data in cache.
I have a table LOGS with this attributes:
ID(int)
Date(datetime)
Core(int)
Source(string)
Message(string)
That table contains log entries from multiple jobs.
A job has multiple log entries, but the start/end entries always have the same message.
Example:
->1, 07.12.2016 10:49:00, 2, Some DLL, Calling Execute on CleanUp // Start
2, 07.12.2016 10:49:01, 3, Other DLL, BLABLABLA
3, 07.12.2016 10:49:10, 1, Other DLL, BLABLABLA
->4, 07.12.2016 10:50:15, 2, Other DLL, BLABLABLA // Job does sth.
->5, 07.12.2016 10:50:50, 2, Other DLL, Execution completed // End
The rows marked with an arrow belonging to the same job.
As you can see, a job starts with 'Calling Execute...' and ends with 'Execution completed'.
What I want to achieve:
My task is to get the average job running times. The initial aproach was to filter with
WHERE Message LIKE '%JOBNAME%' OR Message LIKE 'Execution completed'
and comparing the dateTimes. This worked for some jobs, but some jobs run rarely so I only get "Execution completed" and the precision is not that great doing this manually.
At the end I want a list with following attributes:
ID(start log),
Start-Date,
End-Date,
Core,
Source-Start,
Source-End,
Message-Start,
Message-End
So later it's easy to calculate the difference and do the avg on it.
My idea
-> Get jobs by searching for a message.
-> Get a list with the message "Executing completed" having:
a higher ID (end log is always after start log)
a later datetime
the same core
For example:
Having a job with the attributes
1, 07.12.2016 11:33:00, 2, Source 1, Calling Execute on job Cleanup
Then searching for all logs with
ID>1,
dateTime>07.12.2016 11:33:00,
Core=2,
Message="Execution completed"
Picking the first item of that list should be the end log of the job
How can I do this with a sql query?
PS: I cannot change anything in the database, I can only read data.
You can identify the jobs using a correlated subquery to get the next end record. The following shows how to get these fields:
select l.*, lend.*
from (select l.*,
(select min(l2.date)
from logs l2
where l2.core = l.core and
l2.message like '% End'
l2.date > l.date
) as end_date
from logs l
where l.message like '% Start'
) l join
logs lend
on lend.core = l.core and lend.date = l.end_date;
This assumes that the date/time values are unique for a given "core".
I want to demonstrate a deadlock situation:
In my first transaction:
UPDATE POSITION SET EXTRA = EXTRA || 'yes' WHERE NAME="JOHN"
UPDATE POSITION SET EXTRA = 'HI' WHERE EXTRA = 'EXTRA';
So second transaction:
UPDATE POSITION SET BONUS = BONUS * 1.05;
UPDATE POSITION SET BONUS = 0 IF BONUS IS NULL;
So isn't possible to occur deadlock here just want to try and understand for it
for my knowledge. deadlock occur if update at different row but not different column and transaction occur same with each other, but for this 4 updates. i don't know how to make it become deadlock situation
Deadlocks occur when two processes block each other by trying to obtain the same resources in a different order. I've seen Oracle deadlocks happen for three reasons, there are probably more:
Concurrent sessions update the same rows in different order because explain plans retrieve the rows differently. For example, one session uses an index and another uses a full table scan.
Un-indexed foreign keys cause table locks.
Bitmap indexes and any type of concurrent DML on a table.
The code below demonstrates the first case. It generates a deadlock by looping through two of your update statements. The index causes the first session to use an INDEX RANGE SCAN and the second session uses a FULL TABLE SCAN. The results are not deterministic but it only took about a second for this to fail on my PC.
Sample schema and data
create table position(name varchar2(100), extra varchar2(4000), bonus number);
insert into position select 'JOHN', null, 1 from dual connect by level <= 100;
insert into position select level , null, 1 from dual connect by level <= 100000;
create index position_index on position(name);
Session 1 (run at the same time as session 2)
begin
for i in 1 .. 1000 loop
UPDATE POSITION SET EXTRA = EXTRA || 'yes' WHERE NAME='JOHN';
commit;
end loop;
end;
/
Session 2 (run at the same time as session 1)
begin
for i in 1 .. 1000 loop
UPDATE POSITION SET BONUS = BONUS * 1.05;
commit;
end loop;
end;
/
Error message
ERROR at line 1:
ORA-00060: deadlock detected while waiting for resource
ORA-06512: at line 3
Find the location of the trace file generated for each deadlock:
select value from v$parameter where name like 'background_dump_dest';
Example of a trace:
...
Deadlock graph:
---------Blocker(s)-------- ---------Waiter(s)---------
Resource Name process session holds waits process session holds waits
TX-0009000F-00004ACC-00000000-00000000 37 129 X 55 375 X
TX-0008001B-0000489C-00000000-00000000 55 375 X 37 129 X
session 129: DID 0001-0025-00000281 session 375: DID 0001-0037-00012A2C
session 375: DID 0001-0037-00012A2C session 129: DID 0001-0025-00000281
Rows waited on:
Session 129: obj - rowid = 0001AC1C - AAAawcAAGAAAudMAAQ
(dictionary objn - 109596, file - 6, block - 190284, slot - 16)
Session 375: obj - rowid = 0001AC1C - AAAawcAAGAAAudMAAL
(dictionary objn - 109596, file - 6, block - 190284, slot - 11)
----- Information for the OTHER waiting sessions -----
Session 375:
sid: 375 ser: 10033 audsid: 56764801 user: 111/xxxxxxxxxxxx
flags: (0x45) USR/- flags_idl: (0x1) BSY/-/-/-/-/-
flags2: (0x40009) -/-/INC
pid: 55 O/S info: user: oracle, term: xxxxxxxxxx, ospid: 7820
image: ORACLE.EXE (SHAD)
client details:
O/S info: user: xxxxxxxxxx\xxxxxxxxxx, term: xxxxxxxxxx, ospid: 11848:10888
machine: xxxxxxxxxx\xxxxxxxxxx program: sqlplus.exe
application name: SQL*Plus, hash value=3669949024
current SQL:
UPDATE POSITION SET BONUS = BONUS * 1.05
----- End of information for the OTHER waiting sessions -----
Information for THIS session:
----- Current SQL Statement for this session (sql_id=cp515bpfsjd07) -----
UPDATE POSITION SET EXTRA = EXTRA || 'yes' WHERE NAME='JOHN'
...
The locked object is not always the table directly modified. Check which object caused the problem:
select * from dba_objects where object_id = 109596;
I need some help for a problem that's driving me crazy!
I've moved an ASP + SQL Server application from an old server to a new one.
The old one was a Windows 2000 server with MSDE, and the new one is a Windows 2008 with SQL Server 2008 Express.
Everything is ok, even a little faster, except just one damned function whose asp page gives a time out.
I've tried the query within that page in a management query windows and it never ends, while in the old server it took about 1 minute to be completed.
The query is this one:
SELECT DISTINCT
TBL1.TBL1_ID,
REPLACE(TBL1_TITOLO, CHAR(13) + CHAR(10), ’ ’),
COALESCE(TBL1_DURATA, 0), TBL1_NUMERO,
FLAG_AUDIO
FROM
SPOT AS TBL1
INNER JOIN
CROSS_SPOT AS CRS ON CRS.TBL1_ID = TBL1.TBL1_ID
INNER JOIN
DESTINATARI_SPOT AS DSP ON DSP.TBL1_ID = TBL1.TBL1_ID
WHERE
DSP.PTD_ID_PUNTO = 1044
AND DSP.DSP_FLAG_OK = 1
AND TBL1.FLAG_AUDIO_TESTO = 1
AND TBL1.FLAG_AUDIO_GRAFICO = ’A’
AND CRS.CRS_STATO > 2
OR TBL1.TBL1_ID IN (SELECT ID
FROM V_VIEW1
WHERE ID IS NOT NULL AND V_VIEW1.ID_MODULO = 403721)
OR TBL1.TBL1_ID IN (SELECT TBL1_ID
FROM V_VIEW2
WHERE V_VIEW2.ID_PUNTO = 1044)
ORDER BY
TBL1_NUMERO
I've tried to transform the 2 views in last lines into tables and the query works, even if a little slower than before.
I've migrated the db with it's backup/restore function. Could it be and index problem?
Any suggestions?
Thanks in advance!
Alessandro
Run:
--Defrag all indexes
sp_msForEachTable 'DBCC DBREINDEX (''?'')'
--Update all statistics
sp_msForEachTable 'UPDATE STATISTICS ? WITH FULLSCAN'
If that doesn't "just fix it", it's going to some subtle "improvement" in the SQL Server optimizer that made things worse.
Try the index tuning wizard (or whatever its SSMS2008 equivalent).
After that, you'll have to start picking the query apart, removing things until it runs fast. Since you have 2 OR clauses, you basically have 3 separate queries:
SELECT ... FROM ...
WHERE DSP.PTD_ID_PUNTO = 1044
AND DSP.DSP_FLAG_OK = 1
AND TBL1.FLAG_AUDIO_TESTO=1
AND TBL1.FLAG_AUDIO_GRAFICO=’A’
AND CRS.CRS_STATO>2
--UNION
SELECT ... FROM ...
WHERE TBL1.TBL1_ID IN (
SELECT ID
FROM V_VIEW1
WHERE ID IS NOT NULL
AND V_VIEW1.ID_MODULO = 403721
)
--UNION
SELECT ... FROM ...
WHERE TBL1.TBL1_ID IN (
SELECT TBL1_ID
FROM V_VIEW2
WHERE V_VIEW2.ID_PUNTO = 1044
)
See which one of those is the slowest.
p.s. A query taking a minute is pretty bad. My opinion is that queries should return instantly (within the limits of human observation)
I have a central suscriber with 5 publishers. After replication, I want to get the last synchronization date and the server that made that synchronization. Is it possible to find this information from existing tables that SQL Server uses for replication?
I use something simmiler to this to check last replication times. However this is on a database level and not a table level. this is ran from the distribution database of the publisher.
SELECT MAX(DISTINCT h.[time]) AS RunTime
FROM MSmerge_history h INNER JOIN
MSmerge_agents a ON a.id = h.agent_id
WHERE (a.publisher_db = 'PublishedDbName')
AND (h.runstatus <> 1)
AND Left(h.comments,2) = 'Up'
OR (a.publisher_db = 'PublishedDbName')
AND (h.runstatus <> 1)
AND Left(h.comments,2) = 'No'
OR (a.publisher_db = 'PublishedDbName')
AND (h.runstatus <> 1)
AND Left(h.comments,2) = 'Me'
And Publication Like 'PublictionName%'
GO