I'm try to execute the below hive query on Azure HDInsight cluster but it's taking unprecedented amount of time to finish. Did implemented hive settings but of no use. Below are the details:
Table
CREATE TABLE DB_MYDB.TABLE1(
MSTR_KEY STRING,
SDNT_ID STRING,
CLSS_CD STRING,
BRNCH_CD STRING,
SECT_CD STRING,
GRP_CD STRING,
GRP_NM STRING,
SUBJ_DES STRING,
GRP_DESC STRING,
DTL_DESC STRING,
ACTV_FLAG STRING,
CMP_NM STRING)
STORED AS ORC
TBLPROPERTIES ('ORC.COMPRESS'='SNAPPY');
Hive Query
INSERT OVERWRITE TABLE DB_MYDB.TABLE1
SELECT
CURR.MSTR_KEY,
CURR.SDNT_ID,
CURR.CLSS_CD,
CURR.BRNCH_CD,
CURR.SECT_CD,
CURR.GRP_CD,
CURR.GRP_NM,
CURR.SUBJ_DES,
CURR.GRP_DESC,
CURR.DTL_DESC,
'Y',
CURR.CMP_NM
FROM DB_MYDB.TABLE2 CURR
LEFT OUTER JOIN DB_MYDB.TABLE3 PREV
ON (CURR.SDNT_ID=PREV.SDNT_ID
AND CURR.CLSS_CD=PREV.CLSS_CD
AND CURR.BRNCH_CD=PREV.BRNCH_CD
AND CURR.SECT_CD=PREV.SECT_CD
AND CURR.GRP_CD=PREV.GRP_CD
AND CURR.GRP_NM=PREV.GRP_NM)
WHERE PREV.SDNT_ID IS NULL;
But the query is running for hours. Below is the detail:
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 .......... SUCCEEDED 46 46 0 0 0 0
Map 3 .......... SUCCEEDED 169 169 0 0 0 0
Reducer 2 .... RUNNING 1009 825 184 0 0 0
--------------------------------------------------------------------------------
VERTICES: 02/03 [======================>>----] 84% ELAPSED TIME: 13622.73 s
--------------------------------------------------------------------------------
I did set some hive properties
SET hive.execution.engine=tez;
SET hive.tez.container.size=10240;
SET tez.am.resource.memory.mb=10240;
SET tez.task.resource.memory.mb=10240;
SET hive.auto.convert.join.noconditionaltask.size=3470;
SET hive.vectorized.execution.enabled = true;
SET hive.vectorized.execution.reduce.enabled=true;
SET hive.vectorized.execution.reduce.groupby.enabled=true;
SET hive.cbo.enable=true;
SET hive.exec.compress.output=true;
SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
SET mapred.output.compression.type=BLOCK;
SET hive.compute.query.using.stats=true;
SET hive.merge.mapfiles = true;
SET hive.merge.mapredfiles = true;
SET hive.merge.tezfiles = true;
SET hive.merge.size.per.task=268435456;
SET hive.merge.smallfiles.avgsize=16777216;
SET hive.merge.orcfile.stripe.level=true;
Records in Tables:
DB_MYDB.TABLE2= 337319653
DB_MYDB.TABLE3= 1946526625
There doesn't seem to be any impact on the query. Can anyone help me to:
Understand that why this query is not completing and taking indefinite time?
How can I optimize it to work faster and complete?
Using the versions:
Hadoop 2.7.3.2.6.5.3033-1
Hive 1.2.1000.2.6.5.3033-1
Azure HDInsight 3.6
Attempt_1:
As suggested by #leftjoin tried to set the set hive.exec.reducers.bytes.per.reducer=32000000;. This worked until the second last step of the hive script but at the last it failed with Caused by: java.io.IOException: Map_1: Shuffle failed with too many fetch failures and insufficient progress!
Last Query:
INSERT OVERWRITE TABLE DB_MYDB.TABLE3
SELECT
CURR_FULL.MSTR_KEY,
CURR_FULL.SDNT_ID,
CURR_FULL.CLSS_CD,
CURR_FULL.BRNCH_CD,
CURR_FULL.GRP_CD,
CURR_FULL.CHNL_CD,
CURR_FULL.GRP_NM,
CURR_FULL.GRP_DESC,
CURR_FULL.SUBJ_DES,
CURR_FULL.DTL_DESC,
(CASE WHEN CURR_FULL.SDNT_ID = SND_DELTA.SDNT_ID THEN 'Y' ELSE
CURR_FULL.SDNT_ID_FLAG END) AS SDNT_ID_FLAG,
CURR_FULL.CMP_NM
FROM
DB_MYDB.TABLE2 CURR_FULL
LEFT OUTER JOIN DB_MYDB.TABLE1 SND_DELTA
ON (CURR_FULL.SDNT_ID = SND_DELTA.SDNT_ID);
-----------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
-----------------------------------------------------------------
Map 1 ......... RUNNING 1066 1060 6 0 0 0
Map 4 .......... SUCCEEDED 3 3 0 0 0 0
Reducer 2 RUNNING 1009 0 22 987 0 0
Reducer 3 INITED 1 0 0 1 0 0
-----------------------------------------------------------------
VERTICES: 01/04 [================>>--] 99% ELAPSED TIME: 18187.78 s
Error:
Caused by: java.io.IOException: Map_1: Shuffle failed with too many fetch failures and insufficient progress!failureCounts=8, pendingInputs=1058, fetcherHealthy=false, reducerProgressedEnough=false, reducerStalled=false
If it is reducer vertex which is runnong slow, you can increase reducer parallelism by reducing bytes per reducer configuration. Check your current setting and reduce figure accordingly untill you get 2x or more reducers running:
set hive.exec.reducers.bytes.per.reducer=67108864; --example only, check your current settings
--and reduce accordingly to get twice more reducers on Reducer 2 vertex
Change setting, start query, check the number of containers on Reducer 2 vertex, terminate and change again if the number of containers has not increased.
If you want to increase parallelism on mappers also, read this answer: https://stackoverflow.com/a/48487306/2700344
if you don't have index on your fk columns , you should add them for sure , here is my suggestion:
create index idx_TABLE2 on table DB_MYDB.TABLE2 (SDNT_ID,CLSS_CD,BRNCH_CD,SECT_CD,GRP_CD,GRP_NM) AS 'COMPACT' WITH DEFERRED REBUILD;
create index idx_TABLE3 on table DB_MYDB.TABLE3(SDNT_ID,CLSS_CD,BRNCH_CD,SECT_CD,GRP_CD,GRP_NM) AS 'COMPACT' WITH DEFERRED REBUILD;
be noticed from hive version 3.0 , indexing has been removed from hive and alternatively you can use materialized views (supported from Hive 2.3.0 and above) that gives you the same performance.
Related
I have created table as follows:
create table emp (
> eid int,
> fname string,
> lname string,
> salary double,
> city string,
> dept string )
> row format delimited fields terminated by ',';
then to enable partitioning i have set following properties:
set hive.exec.dynamic.partition=true;
set hive.exec.dynamic.partition.mode=nonstrict;
i created partition table as follows:
create table part_emp (
> eid int,
> fname string,
> lname string,
> salary double,
> dept string )
> partitioned by ( city string )
> row format delimited fields terminated by ',';
After creating table i issued insert query as
insert into table part_emp partition(city)
select eid,fname,lname,salary,dept,city from emp;
But it not works..
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Query ID = max_20180311015337_5a67813d-dcc5-46c0-ac4b-a54c11ffb912
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1520757649534_0004, Tracking URL = http://ubuntu:8088/proxy/application_1520757649534_0004/
Kill Command = /home/max/bigdata/hadoop-3.0.0/bin/hadoop job -kill job_1520757649534_0004
Hadoop job information for Stage-1: number of mappers: 0; number of reducers: 0
2018-03-11 01:53:44,996 Stage-1 map = 0%, reduce = 0%
Ended Job = job_1520757649534_0004 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-1: HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
Same Successfully Works on Hive 1.x
I have the same problem, and set hive.exec.max.dynamic.partitions.pernode=1000;(default 100) solves my problem. You may try.
PS：This setting means：Maximum number of dynamic partitions allowed to be created in each mapper/reducer node.
I have a table which is having 3 columns-PID,LOCID,ISMGR. Now in existing scenario, for some person, based on the location ID, he is set as ISMGR=true.
But as per the new requirement, we have to make all the ISMGR=true for any person who is having at least one ISMGR=true(means if he is mangager for any one location, he should be manager for all the locations).
Table Data before running the script:
PID|LOCID|ISMGR
1 1 1
1 2 0
1 3 0
2 1 0
2 2 1
Table Data after running the script:
PID|LOCID|ISMGR
1 1 1
1 2 1
1 3 1
2 1 1
2 2 1
Any help will be highly appreciated..
Thanks in advance.
I would be inclined to write this using exists:
update t
set ismgr = 1
where ismgr = 0 and
exists (select 1 from t t2 where t2.pid = t.pid and t2.ismgr = 1);
exists should be more efficient than doing a subquery with an aggregation.
This will work best with indexes on t(pid, ismgr) and t(ismgr).
This is not an answer but a test of the two solutions offered so far - I will call them the "EXISTS" and the "AGGREGATE" solutions or approaches.
Details of the tests are below, but here are two overall conclusions:
Both approaches have comparable execution times; on average the AGGREGATE approach worked a little faster than the EXISTS approach, but by a very small margin (smaller than the differences between running times from one trial to the next). Without indexes on any columns, the run times were: (first number is for the EXISTS approach and the second for AGGREGATE). Trial 1: 8.19s 8.08s Trial 2: 8.98s 8.22s Trial 3: 9.46s 9.55s Note - Estimated optimizer costs should be used only to compare different execution plans for the same statement, not for different solutions using different approaches. Even so, someone will inevitably ask; so - for the EXISTS approach the lowest cost the Optimizer found was 4766; for AGGREGATE, 2665. Again, though, this is completely meaningless.
If a lot of rows need to be updated, indexes will hurt performance much more than they help it. Indeed, when rows are updated, the indexes must be updated as well. If only a small number of rows must be updated, then the indexes will help, because most of the time is spent finding the rows that must be updated and only little time is spent in the updates themselves. In my example almost 25% of rows had to be updated... so the AGGREGATE solution took 51.2 seconds and the EXISTS solution took 59.3 seconds! RECOMMENDATION: If you expect that a large number of rows may need to be updated, and you already have indexes on the table, you may be better off DROPPING them and re-creating them after the updates! Or, perhaps there are other solutions to this problem; I am not an expert (keep that in mind!)
To test properly, after I created the test table and committed, I ran each solution by itself, then I rolled back and, logged in as SYS (in a different session), I ran alter system flush buffer_cache to make sure performance is not randomly helped by cache hits or hurt by misses. In all cases everything is done from disk storage.
I created a table with id's from 1 to 1.2 million and a random integer between 1 and 3, with probabilities 40%, 40% and 20% respectively (see the use of dbms_random below). Then from this prep data I created the test table: each pid was included one, two or three times based on this random integer; and a random 0 or 1 was added as ismgr (with 50-50 probability) in each row. I also added a random integer between 1 and 4 as locid just to simulate the actual data; I didn't worry about duplicate locid since that column plays no role in the problem.
Of the 1.2 million pids, approximately 480,000 (40%) appear just once in the test table, another ~480,000 appear twice and ~240,000 three times. Total rows should be about 2,160,000. That's the cardinality of the base table (in reality it ended up being 2,160,546). Then: none of the ~480,000 rows with unique pid need to be changed; half of the 480,000 pids with a count of 2 will have the same ismgr (so no change) and the other half will be split, so we will need to change 240,000 rows from these; and a simple combinatorial argument shows that 3/8, or 270,000 rows, of the 720,000 rows for pids that appear three times in the table must be changed. So we should expect that 510,000 rows should be changed. In fact the update statements resulted in 510,132 rows updated (same for both solutions). These sanity checks show that the test was probably set up correctly. Below I show also a small sample from the base table, also as a sanity check.
CREATE TABLE statement:
create table tbl as
with prep ( pid, dup ) as (
select level,
round( dbms_random.value(0.5, 3) ) as dup
from dual
connect by level <= 1200000
)
select pid,
round( dbms_random.value(0.5, 4.5) ) as locid,
round( dbms_random.value(0, 1) ) as ismgr
from prep
connect by level <= dup
and prior pid = pid
and prior sys_guid() is not null
;
commit;
Sanity checks:
select count(*) from tbl;
COUNT(*)
----------
2160546
select * from tbl where pid between 324720 and 324730;
PID LOCID ISMGR
---------- ---------- ----------
324720 4 1
324721 1 0
324721 4 1
324722 3 0
324723 1 0
324723 3 0
324723 3 1
324724 3 1
324724 2 0
324725 4 1
324725 2 0
324726 2 0
324726 1 0
324727 3 0
324728 4 1
324729 1 0
324730 3 1
324730 3 1
324730 2 0
19 rows selected
UPDATE statements:
update tbl t
set ismgr = 1
where ismgr = 0 and
exists (select 1 from tbl t2 where t2.pid = t.pid and t2.ismgr = 1);
rollback;
update tbl
set ismgr = 1
where ismgr = 0
and pid in ( select pid
from tbl
group by pid
having max(ismgr) = 1);
rollback;
-- statements to create indexes, used in separate testing:
create index pid_ismgr_idx on tbl(pid, ismgr);
create index ismgr_ids on tbl(ismgr);
Why PL/SQL? All you need is a plain SQL statement. For example:
update your_table t -- enter your actual table name here
set ismgr = 1
where ismgr = 0
and pid in ( select pid
from your_table
group by pid
having max(ismgr) = 1)
;
The existing solutions are perfectly fine, but I prefer to use merge any time I'm updating rows from a correlated sub-query. I find it to be more readable and the performance is typically commensurate with the exists method.
MERGE INTO t
USING (SELECT DISTINCT pid
FROM t
WHERE ismgr = 1) src
ON (t.pid = src.pid)
WHEN MATCHED THEN
UPDATE SET ismgr = 1
WHERE ismgr = 0;
As #mathguy pointed out, in this case using group by and having is more efficient than distinct. To use that with merge is just a matter of changing the sub-query:
MERGE INTO t
USING (SELECT pid
FROM t
GROUP BY pid
HAVING MAX(ismgr) = 1) src
ON (t.pid = src.pid)
WHEN MATCHED THEN
UPDATE SET ismgr = 1
WHERE ismgr = 0;
In a project we have several SSIS packages (around 200), all the package names are stored in a control table. We need to create a master package which can run all the 200 packages.
Since the max concurrent executable setting was set to 8. So planning to create 8 execute package tasks in a container and was thinking of generating the connection string(Execute package task- File connection String) dynamically using the package names stored in the table.
The control table is in the below format
Id PackageName
---------------
1 Package1
2 Package2
Ideas on how should be implemented helps.
I covered this pattern on https://stackoverflow.com/a/34868545/181965 but you're looking for a package that looks something like this
A sequence container that contains everything that one of those 8 discrete buckets of work would require. In your case, a Variable for
CurrentPackage String
rsObject Object
ContainerId Int32
The containerId will be the values 0 through 7 (since you have 8 buckets of work). As outlined in the other answer, we must scope the variables to the Sequence Container. The default in 2012+ is to create them at the Control Flow level, whereas 2005/2008 would create them at the level of the selected object.
Set up
I created a table and loaded it with 200 rows
CREATE TABLE dbo.so_35415549
(
id int IDENTITY(1,1) NOT NULL
, PackageName sysname
);
INSERT INTO
dbo.so_35415549
(
PackageName
)
SELECT TOP 200
'Package' + CAST(ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) AS varchar(3))
FROM
sys.all_columns AS AC;
Get My Bucket's data
The modulus, modulo, mod whatever you call it operator is our friend here. The mod operator will return the remainder after division. e.g. 10 mod 3 is 1 because 3*3 + 1 = 10
In your case, you'll be modding via 8 so you know the remainder will be bounded between 0 and 7.
SQL Server implements the mod operator as % and you can test the correctness via the following query
SELECT
S.id
, S.PackageName
, S.id % 8 AS ModValue
FROM
dbo.so_35415549 AS S
ORDER BY
1;
Sample output
id PackageName ModValue
1 Package1 1
2 Package2 2
3 Package3 3
4 Package4 4
5 Package5 5
6 Package6 6
7 Package7 7
8 Package8 0
9 Package9 1
10 Package10 2
...
199 Package199 7
200 Package200 0
SQL Get Work List
Using the above query as a template, we will use the following query. Notice the ? in there. That is the placeholder for an Execute SQL Tasks parameterization for an OLE DB Connection Manager.
SELECT
S.PackageName
FROM
dbo.so_35415549 AS S
WHERE
S.id % 8 = ?
ORDER BY
1;
The Parameter we pass in will be #[User::ContainerId]
The Result Set option will be updated from None to Full ResultSet and we push the value into rsObject
FELC Shred Work List
This is a standard shredding of a recordset. We got our variable populated in the previous step so let's enumerate through the results. There will be one column in our result set and you will map that to User::CurrentPackageName
EPT Run Package
This is your Execute Package Task. Use the value of CurrentPackageName and you're set.
I have an update statement which is designed to manage "jobs". I process one job first whilst the others wait in line until the first one is complete. After which they will be sent to process (far faster as I can leverage calcs from the first run).
To quote my own comment:
-- Set job status to 2 for where there is saved results file
-- Where there is no saved results file:
-- --Set job status to 2 for one instance of each unique param combination
-- --Set job status to 8 for all others
PseudoCode:
UPDATE #Jobs
SET [JobStatusId] = CASE
WHEN ISNULL([PreCalculated].[FileCount], 0) > 0 THEN 2
WHEN ISNULL([PreCalculated].[FileCount], 0) = 0 AND [GroupedOrder] = 1 THEN 2
ELSE 8 END
FROM #NewJobsGrouped [NewJobsGrouped]
LEFT JOIN (
SELECT COUNT([Id]) [FileCount],
[ResultsFile].[ParamsId]
FROM #ResultsFile [ResultsFile]
WHERE [ResultsFile].[IsActive] = 1
GROUP BY [ResultsFile].[ParamsId]
) [PreCalculated]
ON [PreCalculated].[ParamsId] = [NewJobsGrouped].[ParamsId]
where #NewJobsGrouped looks like:
Job ID || GroupedOrder || ParamsId
1460 1 807
1461 2 807
1462 3 807
This does not work. Every job is being set to status 2. However:
SELECT CASE
WHEN ISNULL([PreCalculated].[FileCount], 0) > 0 THEN 2
WHEN ISNULL([PreCalculated].[FileCount], 0) = 0 AND [GroupedOrder] = 1 THEN 2
ELSE 8 END [JobStatusId]
etc
Works exactly as I am expecting.
Why would these two case statements give different results? Is there something obvious I am missing? I honestly can't explain what I'm seeing and whilst I can probably use another temp table to hold the output from the select and have a simpler update - but I'd like to understand what's going on?
I came across a schema and an upsert stored procedure that was causing deadlock issues. I have a general idea about why this is causing deadlock and how to fix it. I can reproduce it but I don't have a clear understanding of the sequence of steps that is causing it. It would be great if someone can explain clearly why this is causing deadlock.
Here is the schema and the stored procedures. This code is being executed on PostgreSQL 9.2.2.
CREATE TABLE counters (
count_type INTEGER NOT NULL,
count_id INTEGER NOT NULL,
count INTEGER NOT NULL
);
CREATE TABLE primary_relation (
id INTEGER PRIMARY KEY,
a_counter INTEGER NOT NULL DEFAULT 0
);
INSERT INTO primary_relation
SELECT i FROM generate_series(1,5) AS i;
CREATE OR REPLACE FUNCTION increment_count(ctype integer, cid integer, i integer) RETURNS VOID
AS $$
BEGIN
LOOP
UPDATE counters
SET count = count + i
WHERE count_type = ctype AND count_id = cid;
IF FOUND THEN
RETURN;
END IF;
BEGIN
INSERT INTO counters (count_type, count_id, count)
VALUES (ctype, cid, i);
RETURN;
EXCEPTION WHEN OTHERS THEN
END;
END LOOP;
END;
$$
LANGUAGE PLPGSQL;
CREATE OR REPLACE FUNCTION update_primary_a_count(ctype integer) RETURNS VOID
AS $$
WITH deleted_counts_cte AS (
DELETE
FROM counters
WHERE count_type = ctype
RETURNING *
), rollup_cte AS (
SELECT count_id, SUM(count) AS count
FROM deleted_counts_cte
GROUP BY count_id
HAVING SUM(count) <> 0
)
UPDATE primary_relation
SET a_counter = a_counter + rollup_cte.count
FROM rollup_cte
WHERE primary_relation.id = rollup_cte.count_id
$$ LANGUAGE SQL;
And here is a python script to reproduce the deadlock.
import os
import random
import time
import psycopg2
COUNTERS = 5
THREADS = 10
ITERATIONS = 500
def increment():
outf = open('synctest.out.%d' % os.getpid(), 'w')
conn = psycopg2.connect(database="test")
cur = conn.cursor()
for i in range(0,ITERATIONS):
time.sleep(random.random())
start = time.time()
cur.execute("SELECT increment_count(0, %s, 1)", [random.randint(1,COUNTERS)])
conn.commit()
outf.write("%f\n" % (time.time() - start))
conn.close()
outf.close()
def update(n):
outf = open('synctest.update', 'w')
conn = psycopg2.connect(database="test")
cur = conn.cursor()
for i in range(0,n):
time.sleep(random.random())
start = time.time()
cur.execute("SELECT update_primary_a_count(0)")
conn.commit()
outf.write("%f\n" % (time.time() - start))
conn.close()
pids = []
for i in range(THREADS):
pid = os.fork()
if pid != 0:
print 'Process %d spawned' % pid
pids.append(pid)
else:
print 'Starting child %d' % os.getpid()
increment()
print 'Exiting child %d' % os.getpid()
os._exit(0)
update(ITERATIONS)
for pid in pids:
print "waiting on %d" % pid
os.waitpid(pid, 0)
# cleanup
update(1)
I recognize that one issue with this is that the upsert will can produce duplicate rows (with multiple writers) which will likely result in some double counting. But why will this result in deadlock?
The error I get from PostgreSQL is something like the following:
process 91924 detected deadlock while waiting for ShareLock on transaction 4683083 after 100.559 ms",,,,,"SQL statement ""UPDATE counters
And the client spews something like this:
psycopg2.extensions.TransactionRollbackError: deadlock detected
DETAIL: Process 91924 waits for ShareLock on transaction 4683083; blocked by process 91933.
Process 91933 waits for ShareLock on transaction 4683079; blocked by process 91924.
HINT: See server log for query details.CONTEXT: SQL statement "UPDATE counters
SET count = count + i
WHERE count_type = ctype AND count_id = cid"
PL/pgSQL function increment_count(integer,integer,integer) line 4 at SQL statement
To fix the issue, you need to add a primary key like so:
ALTER TABLE counters ADD PRIMARY KEY (count_type, count_id);
Any insight would be greatly appreciated. Thanks!
because of the primary key, the number of rows in this table is always <= # threads, and the primary key ensures that no row is repeated.
When you remove the primary key, some of the threads get lagged behind and the number of rows increases, and at the same time rows get repeated. When the rows get repeated, then the update time is longer and 2 or more threads will try to update the same row(s).
Open a new terminal and type:
watch --interval 1 "psql -tc \"select count(*) from counters\" test"
Try this with and without the primary key. When you get the first deadlock, look at the results of the query above. In my case this is what I am left with in the table counters:
test=# select * from counters order by 2;
count_type | count_id | count
------------+----------+-------
0 | 1 | 735
0 | 1 | 733
0 | 1 | 735
0 | 1 | 735
0 | 2 | 916
0 | 2 | 914
0 | 2 | 914
0 | 3 | 882
0 | 4 | 999
0 | 5 | 691
0 | 5 | 692
(11 rows)
Your code is the perfect recipe for a race condition (multiple threads, random sleeps).
The problem is most probably due to locking issues, since you don't mention the locking mode i'm going to assume that is a page based lock so, you get the following scenario:
Thread 1 starts, it begins to insert records, lets say that it locks page n° 1, and should lock page 2.
Thread 2 starts, at the same time as 1, but it locks first page 2, and should lock page 1 next.
Both threads are now waiting on each other to complete, so you have a deadlock.
Now, why a PK fixes it?
Because locking is done via index at first, you the race condition is mitigated because the PK is unique on the inserts, so all threads wait for the index, and in updates access is done via index so the record is locked based on its PK.
At some point one user is waiting on a lock another user has, while the first user owns a lock that the second user wants. This is what causes a deadlock.
At a guess, it's because without a primary key (or in fact any key) when you UPDATE counters in your increment sp it is having to read the whole table. The same with the primary_relation table. This is going to leave locks strewn about, and open the way for a deadlock. I'm not a Postgres user so I don't know the details of exactly when it will place locks, but I'm pretty sure that this is what is happening.
Putting a PK on counters is making it possible for the DB to target the rows it reads accurately and put on the minimum number of locks. You should really have a PK on primary_relation too!