Generate random data in Oracle based on ranks - sql

Question

It is given the following scenario. I have a list of 3000 first names and a list of 2500 last names. Each one of them has a "ranking" that represents the position in a name's top. Two or more names can have the same ranking. Also, a table with 1500 cities is given, each with 4 census values in certain years.
From the tables above I must generate 5 million random entries containing the first name, last name, birth date and place of birth of one person, that should follow the rules given by ranking of the names and population number of the cities.
This have to be generated using just Oracle (stored functions, stored procedures and so on). How can I do this?

Answer 1

Disclaimer: I'm not a statistics expert, and there are probably way more efficient means to do that.
The most challenging task seems to be the creation of 5 million names according to ranks. In real world, those would be distributed unevenly among the population: difference between second last and last would be 1-2 persons, and the difference between the first and second rank could be thousands of people. That said, I have no idea how to achieve that, so we'll model it in other way. Suppose we have total population of 100 and list of four ranked names:
Alice: 1
Bob: 2
Betty: 2
Claire: 3
We can make the distribution "even", so that rank 3 has X people, rank 2 has twice as many, and rank 1 thrice as many. If the ranks were unique, the formula would be as simple as X + 2X + 3X = 100, but we have two names in rank 2, so it should be X + 2*2X + 3X = 100, so X = 12.5. We can truncate it to integer and get people counts for all ranks except the first (12, 24 and 24) and first rank would get what remains: 40. Seems good enough, though it will not work for edge case when you have multiple first ranks.
There's a little problem, though. For 3000 different names, the sum of coefficients would be 4501500. So, truncated X would be 1, making rank 3000 to rank 2 have 1 to 2999 people respectively, and rank 1 have a little under 500000. That's not quite good enough. To illustrate with four names above, assume total count of 15. With current algorithm, X will be 1 as well, and distribution will be 1-2-2-10. Luckily, we'll be processing ranks one by one in procedure, so we can remove processed people from equation and recalculate X. E.G. first it's X + 2*2X + 3X = 15 with X=1, then 2*2X + 3X = 14 with X=2. This way, distribution will be 1-4-4-6, which is far from ideal, but better.
Now, this can already be expressed as PL/SQL. I suggest to create the table with following columns: LAST_NAME, FIRST_NAME, BIRTHDAY, CITY, RAND_ROWNO.
First of all, let's fill it with 5M last names. Assuming your table for them is last_names(name, name_rank), you'll need the following:
declare
cursor cur_last_name_ranks is
select name_rank, count(*) cnt, row_number() over (order by name_rank desc) coeff
from last_names l
group by name_rank;
cursor cur_last_names (c_rank number) is
select name from last_names
where name_rank = c_rank;
v_coeff_sum number;
v_total_people_count number:= 5000000;
v_remaining_people number;
v_x number;
v_insert_cnt number;
begin
--Get a sum of all coefficients for our formula
select sum(coeff) into v_coeff_sum
from
(
select count(*) * row_number() over (order by name_rank desc) coeff
from last_names l
group by name_rank
);
v_remaining_people := v_total_people_count;
--Now, loop for all coefficients
for r in cur_last_name_ranks loop
--Recalculate X
v_x := trunc(v_remaining_people / v_coeff_sum);
--First, determine how many rows should be inserted per last name with such rank
if r.name_rank = 1 then
if r.cnt > 1 then
--raise an exception here, we don't allow multiple first ranks
raise TOO_MANY_ROWS;
end if;
v_insert_cnt := v_remaining_people;
else
v_insert_cnt := v_x*r.coeff;
end if;
--Insert last names N times.
--Instead of multiple INSERT statements, use select from dual with connect trick.
for n in cur_last_names(r.name_rank) loop
insert into result_table(last_name)
select n.name from dual connect by level <= v_insert_cnt;
end loop;
commit;
--Calculate remaining people count
v_remaining_people := v_remaining_people - v_x*r.cnt*r.coeff;
--Recalculate remmaining coefficients
v_coeff_sum := v_coeff_sum - r.cnt*r.coeff;
end loop;
end;
Now you have 5 million rows with last names filled according to ranks. Now, we'll need to assign random number from 1 to 5000000 for each row - you'll see why. This is done with a single query using merge on self:
merge into result_table t1
using (select rowid rid, row_number() over (ORDER BY DBMS_RANDOM.VALUE) rnk from result_table) t2
on (t1.rowid = t2.rid)
when matched then update set t1.rand_rowno = t2.rnk
Note that it will take some time because of large size.
Now you must repeat the same procedure for first names. It'll be very similar to last names, except you'll be updating existing records, not inserting new. If you keep track of how many rows you've updated already, it'll be as simple putting this in the inner loop:
update result_table
set first_name = n.name
where rand_rowno between
(v_processed_rows+1) and
(v_processed_rows+v_insert_cnt);
v_processed_rows := v_processed_rows+v_insert_cnt;
That does it - you now have a decent sample of 5M names according to your ranking, last names randomly matched with first names.
Now, for census. I don't really understand your format, but that's relatively simple. If you get data to the form of "N people were born in city C between DATE1 and DATE2", you can update the table in a loop, setting N rows to have CITY = C and BIRTHDAY = a random date between DATE1 and DATE2. You'll need a function to return a random date from a time period, see this. Also, don't forget to assign random row numbers again before doing that.
I'll leave the census part for you to implement, I've spent too much time on writing this already. Thanks for a good brain exercise!