how can I improve performance for this query - sql

Question

I have been battling for ages on how to complete a data task that I have, I have tried doing in c# as I am proficient at this, but its was taking forever!!! so I have decided to do in SQL Server, however it looks like it is still going to take forever, maybe as long as 20 days to complete.
Does anyone know any more efficient ways to write my stored procedure?
ALTER PROCEDURE [dbo].[TotalWins]
AS
BEGIN
SET NOCOUNT ON
DECLARE #meeting_date DATE;
DECLARE #idStore INT;
DECLARE #race_idStore INT;
DECLARE #runner_id INT;
SET #idStore = 0;
SET #race_idStore = -1;
SET #runner_id = 0;
WHILE(#idStore IS NOT NULL)
BEGIN
SET #race_idStore = -1;
SET #runner_id = 0;
SELECT
#idStore = MIN(runners.id)
FROM
dbHorseRacing.dbo.historic_runners AS runners
WHERE
runners.id > #idStore;
IF #idStore IS NOT NULL
BEGIN
SELECT
#runner_id = runners.runner_id, #meeting_date = races.meeting_date
FROM
dbHorseRacing.dbo.historic_runners AS runners
INNER JOIN
dbHorseRacing.dbo.historic_races AS races ON races.race_id = runners.race_id
WHERE
runners.id > #idStore;
INSERT INTO dbHorseRacing.dbo.total_wins
SELECT
#idStore, COUNT(*) AS total_wins
FROM
dbHorseRacing.dbo.historic_runners AS runners
INNER JOIN
dbHorseRacing.dbo.historic_races AS races ON races.race_id = runners.race_id
WHERE
runners.runner_id = #runner_id
AND races.meeting_date < #meeting_date
AND runners.finish_position = 1;
END
END
END
I am updating this with question with the ddl and a data sample for the races and runners table. Sorry that they are quite large...
race sample date:
race_id meeting_id meeting_date course conditions race_name race_abbrev_name race_type_id race_type race_num going direction class draw_advantage num_fences handicap all_weather seller claimer apprentice maiden amateur num_runners num_finishers rating group_race min_age max_age distance_yards added_money official_rating speed_rating private_handicap scheduled_time off_time winning_time_disp winning_time_secs standard_time_disp standard_time_secs loaded_at
-1 2941 2003-07-03 Newbury Arab Race The Emirates Arabian International Arab Race 12 Flat 1 Good Left Handed 1 High numbers best in large fields, especially in very soft ground. NULL 0 0 0 0 0 0 0 8 8 NULL NULL NULL NULL 1320 0 NULL NULL NULL 2003-07-03 18:10:00.000 2003-07-03 00:00:00.000 0:00.00 0 1:14.38 74.379997253418 0x00000000000007DB
race ddl:
[dbo].[historic_races]
(
[race_id] [int] NOT NULL,
[meeting_id] [int] NOT NULL,
[meeting_date] [date] NOT NULL,
[course] [varchar](255) NOT NULL,
[conditions] [varchar](255) NOT NULL,
[race_name] [varchar](255) NOT NULL,
[race_abbrev_name] [varchar](80) NOT NULL,
[race_type_id] [int] NOT NULL,
[race_type] [varchar](80) NOT NULL,
[race_num] [tinyint] NOT NULL,
[going] [varchar](80) NULL,
[direction] [varchar](80) NULL,
[class] [tinyint] NULL,
[draw_advantage] [varchar](255) NULL,
[num_fences] [tinyint] NULL,
[handicap] [tinyint] NULL,
[all_weather] [tinyint] NULL,
[seller] [tinyint] NULL,
[claimer] [tinyint] NULL,
[apprentice] [tinyint] NULL,
[maiden] [tinyint] NULL,
[amateur] [tinyint] NULL,
[num_runners] [tinyint] NULL,
[num_finishers] [tinyint] NULL,
[rating] [int] NULL,
[group_race] [int] NULL,
[min_age] [tinyint] NULL,
[max_age] [tinyint] NULL,
[distance_yards] [int] NULL,
[added_money] [float] NULL,
[official_rating] [int] NULL,
[speed_rating] [int] NULL,
[private_handicap] [int] NULL,
[scheduled_time] [datetime] NULL,
[off_time] [datetime] NULL,
[winning_time_disp] [varchar](20) NULL,
[winning_time_secs] [float] NULL,
[standard_time_disp] [varchar](20) NULL,
[standard_time_secs] [float] NULL,
[loaded_at] [timestamp] NULL,
PRIMARY KEY CLUSTERED
(
[race_id] ASC
)WITH (STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY]
Sample date for runners:
runner_id race_id name foaling_date colour distance_travelled form_figures gender age bred cloth_number stall_number num_fences_jumped long_handicap how_easy_won in_race_comment official_rating official_rating_type speed_rating speed_rating_type private_handicap private_handicap_type trainer_name trainer_id owner_name owner_id jockey_name jockey_id jockey_claim dam_name dam_id sire_name sire_id dam_sire_name dam_sire_id forecast_price forecast_price_decimal starting_price starting_price_decimal betting_text position_in_betting finish_position amended_position unfinished distance_beaten distance_won distance_behind_winner prize_money tote_win tote_place days_since_ran last_race_type_id last_race_type last_race_beaten_fav weight_pounds penalty_weight over_weight tack_hood tack_visor tack_blinkers tack_eye_shield tack_eye_cover tack_cheek_piece tack_pacifiers tack_tongue_strap id total_wins
1 82 401251 David Jack 2010-03-21 CH 143 NULL C 2 UK 4 3 NULL NULL NULL slowly into stride, took keen hold and soon in touch, pushed along and kept on same pace inside final furlong NULL NULL 32 Flat 18 Flat B J Meehan 9262 Roldvale Limited 2311 T E Durcan 18761 NULL NULL NULL NULL NULL NULL NULL 8/1 9 9/1 10 op 8/1 tchd 10/1 5 4 NULL NULL 2 NULL 3.5 216.449996948242 NULL NULL NULL NULL NULL NULL 129 NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL 1267937 NULL
runners ddl:
[dbo].[historic_runners]
(
[runner_id] [int] NOT NULL,
[race_id] [int] NOT NULL,
[name] [varchar](255) NOT NULL,
[foaling_date] [date] NULL,
[colour] [varchar](20) NOT NULL,
[distance_travelled] [int] NULL,
[form_figures] [varchar](80) NULL,
[gender] [varchar](20) NULL,
[age] [int] NULL,
[bred] [varchar](4) NULL,
[cloth_number] [int] NULL,
[stall_number] [int] NULL,
[num_fences_jumped] [int] NULL,
[long_handicap] [int] NULL,
[how_easy_won] [int] NULL,
[in_race_comment] [text] NULL,
[official_rating] [int] NULL,
[official_rating_type] [varchar](80) NULL,
[speed_rating] [int] NULL,
[speed_rating_type] [varchar](80) NULL,
[private_handicap] [int] NULL,
[private_handicap_type] [varchar](80) NULL,
[trainer_name] [varchar](80) NULL,
[trainer_id] [int] NULL,
[owner_name] [varchar](255) NULL,
[owner_id] [int] NULL,
[jockey_name] [varchar](80) NULL,
[jockey_id] [int] NULL,
[jockey_claim] [int] NULL,
[dam_name] [varchar](80) NULL,
[dam_id] [int] NULL,
[sire_name] [varchar](80) NULL,
[sire_id] [int] NULL,
[dam_sire_name] [varchar](80) NULL,
[dam_sire_id] [int] NULL,
[forecast_price] [varchar](20) NULL,
[forecast_price_decimal] [float] NULL,
[starting_price] [varchar](20) NULL,
[starting_price_decimal] [float] NULL,
[betting_text] [text] NULL,
[position_in_betting] [int] NULL,
[finish_position] [int] NULL,
[amended_position] [int] NULL,
[unfinished] [varchar](30) NULL,
[distance_beaten] [float] NULL,
[distance_won] [float] NULL,
[distance_behind_winner] [float] NULL,
[prize_money] [float] NULL,
[tote_win] [float] NULL,
[tote_place] [float] NULL,
[days_since_ran] [int] NULL,
[last_race_type_id] [int] NULL,
[last_race_type] [varchar](80) NULL,
[last_race_beaten_fav] [int] NULL,
[weight_pounds] [int] NULL,
[penalty_weight] [int] NULL,
[over_weight] [int] NULL,
[tack_hood] [int] NULL,
[tack_visor] [int] NULL,
[tack_blinkers] [int] NULL,
[tack_eye_shield] [int] NULL,
[tack_eye_cover] [int] NULL,
[tack_cheek_piece] [int] NULL,
[tack_pacifiers] [int] NULL,
[tack_tongue_strap] [int] NULL,
[id] [int] NOT NULL,
[total_wins] [int] NULL,
CONSTRAINT [PK_RunnerRaceID] PRIMARY KEY CLUSTERED
(
[runner_id] ASC,
[race_id] ASC
)WITH (STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
Desired results - total_wins table
[dbo].[total_wins]
(
[id] [int] NOT NULL,
[total_wins] [int] NULL
)
The "id" on the total wins table corresponds to the id of the runners table, so I have 2mill rows in the runners table, with a unique indicator called id (not to be confused with the runner_id column, which contains duplicate entries as 1 runner can run in lots of races). So I would hope to end up with 2 million rows in the total_wins table, the total wins reflects how many races, the runner has won,prior to the date of the particular race the row relates to.
Any help would be really appreciated!! I have been struggling with this, I even considered flattening down the data and using a big data solution like hadoop, or mongodb.
Thanks
Laura

Answer 1

Thanks to Davids suggestion about using the group by, and avoiding the loop through, I think this is the potential solution...
SELECT runners.id, count(*) as total_wins
FROM dbo.historic_runners as runners
inner join dbo.historic_races as races on races.race_id = runners.race_id
where races.meeting_date <
(
select meeting_date
FROM dbo.historic_runners as ru
inner join dbo.historic_races as ra on ra.race_id = ru.race_id
where ru.id = runners.id
)
and runners.finish_position = 1
group by runners.id
Thanks for you answers on this problem, I appreciate it :)

Answer 2

Laura, I do not know the exact properties of your database, so I will only give you general ideas of what could be improved.
Detecting what is slow
You will need to test what is slow. Make a copy of your database and try to run the queries without the inserts. And then try to run a lot of inserts without custom selections. This way you will detect whether the writes or the reads are slow in your case. If none of these makes it slow, then something else happens to the table slowing down your process.
Take a look at the schema
Take a look whether the schema is adequate, like whether your database is in normal form and if so, in which one. If it is not in normal form, it might be a good idea to convert it to normal form.
Indexing
Take a look at indexes. If reads are slow, then you will need to add indexes for the columns which are involved in your query, but make sure you read an article before you do that about indexing if you are not well versed in the area. If writes are slow, then consider removing unnecessary indexes, like those involving columns which are not used in queries.
Larger batches
I understand that you are iterating the set by each user, but there are probably not so many races that you need to iterate them one-by-one. You could iterate them by batches of 100, by first getting the minimum, then getting the maximum in a select top 100 query ordering by runners.id. This will probably quicken up your process. Note that in later steps you will put the maximum into the minimum, so you will need only one query to determine limits after the very first iteration.
Last but not least
If writes are slow, then you can make a copy of the main table with a lot of indexes, so everything will be quick there and copy only relevant subsets there periodically and use that as a source for the stored procedure, so queries will be quick. This would increase performance, but avoid it if you are not forced to do it, since this adds a lot of maintenance work and a lot of additional possibilities to err.