How can I optimise this SQL query - sql

Question

I'm writing a piece of software that is meant to identify files that have been put onto the web server (CMS) but are no longer needed and should/could be deleted.
To start with I'm trying to reproduce all required steps manually.
I'm using a batch script executed in the webroot to identify all (relevant) files on the server. Then, I'm importing the list to SQL Server and the table looks like this:
id filename
1 filename1.docx
2 files/file.pdf
3 files/filename2.docx
4 files/filename3.docx
5 files/file1.pdf
6 file2.pdf
7 file4.pdf
I also have a CMS database (Alterian/Immediacy CMC 6.X) which has 2 tables storing page content: page_data and PageXMLArchive.
I would like to scan the database to see if the files from the first table are referenced anywhere in the content of the site - p_content column from page_data table and PageXML column from PageXMLArchive table.
So I have a loop which gets each filename and checks if it's referenced in any of those tables, if it is it skips it, if it ain't it adds it to a temporary table.
At the end of the query the temporary table is displayed.
Query below:
DECLARE #t as table (_fileName nvarchar(255))
DECLARE #row as int
DECLARE #result as nvarchar(255)
SET #row = 1
WHILE(#row <= (SELECT COUNT(*) FROM ListFileReport))
BEGIN
SET #result = (SELECT [FileName] FROM ListFileReport WHERE id = #row)
IF ((SELECT TOP(1) p_content FROM page_data WHERE p_content LIKE '%' + LTRIM(RTRIM(#result)) + '%') IS NULL) OR ((SELECT TOP(1) PageXML FROM PageXMLArchive WHERE PageXML LIKE '%' + LTRIM(RTRIM(#result)) + '%') IS NULL)
BEGIN
INSERT INTO #t (_fileName) VALUES(#result)
END
SET #row = #row + 1
END
select * from #t
Unfortunately due to my poor SQL skills the query takes over 2 hours to execute and times out.
How can I imporve that query, or change it to achieve a similar thing without having to run 1000s of WHERE x LIKE statements on ntext fields? I can't make any changes to the database, it has to stay untouched (or it won't be supported - big deal for our customers).
Thanks
EDIT:
Currently I'm working around the issue by batching the results few hundred at a time. It works but takes forever.
EDIT:
Can I possibly utilise Full-Text search to achieve this? I am willing to take a snapshot of the database and work on the copy if there is a way of changing the schema to achieve the desired results.
page_data table:
USE [TD-VMB-01-STG]
GO
/****** Object: Table [dbo].[page_data] Script Date: 12/13/2011 13:19:15 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[page_data](
[p_page_id] [int] NOT NULL,
[p_title] [nvarchar](120) NULL,
[p_link] [nvarchar](250) NULL,
[p_content] [ntext] NULL,
[p_parent_id] [int] NULL,
[p_top_id] [int] NULL,
[p_stylesheet] [nvarchar](50) NULL,
[p_author] [nvarchar](50) NULL,
[p_last_update] [datetime] NULL,
[p_order] [smallint] NULL,
[p_window] [nvarchar](10) NULL,
[p_meta_keywords] [nvarchar](1000) NULL,
[p_meta_desc] [nvarchar](2000) NULL,
[p_type] [nvarchar](1) NULL,
[p_confirmed] [int] NOT NULL,
[p_changed] [int] NOT NULL,
[p_access] [int] NULL,
[p_errorlink] [nvarchar](255) NULL,
[p_noshow] [int] NOT NULL,
[p_edit_parent] [int] NULL,
[p_hidemenu] [int] NOT NULL,
[p_subscribe] [int] NOT NULL,
[p_StartDate] [datetime] NULL,
[p_EndDate] [datetime] NULL,
[p_pageEnSDate] [int] NOT NULL,
[p_pageEnEDate] [int] NOT NULL,
[p_hideexpiredPage] [int] NOT NULL,
[p_version] [float] NULL,
[p_edit_order] [float] NULL,
[p_order_change] [datetime] NOT NULL,
[p_created_date] [datetime] NOT NULL,
[p_short_title] [nvarchar](30) NULL,
[p_authentication] [tinyint] NOT NULL,
CONSTRAINT [aaaaapage_data_PK] PRIMARY KEY NONCLUSTERED
(
[p_page_id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
GO
ALTER TABLE [dbo].[page_data] ADD CONSTRAINT [DF_page_data_p_order] DEFAULT (0) FOR [p_order]
GO
ALTER TABLE [dbo].[page_data] ADD CONSTRAINT [DF__Temporary__p_con__1CF15040] DEFAULT (0) FOR [p_confirmed]
GO
ALTER TABLE [dbo].[page_data] ADD CONSTRAINT [DF__Temporary__p_cha__1DE57479] DEFAULT (0) FOR [p_changed]
GO
ALTER TABLE [dbo].[page_data] ADD CONSTRAINT [DF__Temporary__p_acc__1ED998B2] DEFAULT (1) FOR [p_access]
GO
ALTER TABLE [dbo].[page_data] ADD CONSTRAINT [DF__Temporary__p_nos__1FCDBCEB] DEFAULT (0) FOR [p_noshow]
GO
ALTER TABLE [dbo].[page_data] ADD CONSTRAINT [DF__Temporary__p_edi__20C1E124] DEFAULT (0) FOR [p_edit_parent]
GO
ALTER TABLE [dbo].[page_data] ADD CONSTRAINT [DF__Temporary__p_hid__21B6055D] DEFAULT (0) FOR [p_hidemenu]
GO
ALTER TABLE [dbo].[page_data] ADD CONSTRAINT [DF_page_data_p_subscribe] DEFAULT (0) FOR [p_subscribe]
GO
ALTER TABLE [dbo].[page_data] ADD CONSTRAINT [DF_page_data_p_pageEnSDate] DEFAULT (0) FOR [p_pageEnSDate]
GO
ALTER TABLE [dbo].[page_data] ADD CONSTRAINT [DF_page_data_p_pageEnEDate] DEFAULT (0) FOR [p_pageEnEDate]
GO
ALTER TABLE [dbo].[page_data] ADD CONSTRAINT [DF_page_data_p_hideexpiredPage] DEFAULT (1) FOR [p_hideexpiredPage]
GO
ALTER TABLE [dbo].[page_data] ADD CONSTRAINT [DF_page_data_p_version] DEFAULT (0) FOR [p_version]
GO
ALTER TABLE [dbo].[page_data] ADD CONSTRAINT [DF_page_data_p_edit_order] DEFAULT (0) FOR [p_edit_order]
GO
ALTER TABLE [dbo].[page_data] ADD CONSTRAINT [DF_page_data_p_order_change] DEFAULT (getdate()) FOR [p_order_change]
GO
ALTER TABLE [dbo].[page_data] ADD CONSTRAINT [DF_page_data_p_created_date] DEFAULT (getdate()) FOR [p_created_date]
GO
ALTER TABLE [dbo].[page_data] ADD CONSTRAINT [DF_page_data_p_authentication] DEFAULT ((0)) FOR [p_authentication]
GO
PageXMLArchive table:
USE [TD-VMB-01-STG]
GO
/****** Object: Table [dbo].[PageXMLArchive] Script Date: 12/13/2011 13:20:00 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[PageXMLArchive](
[ArchiveID] [bigint] IDENTITY(1,1) NOT NULL,
[P_Page_ID] [int] NOT NULL,
[p_author] [nvarchar](100) NULL,
[p_title] [nvarchar](400) NULL,
[Version] [int] NOT NULL,
[PageXML] [ntext] NULL,
[ArchiveDate] [datetime] NOT NULL,
CONSTRAINT [PK_PageXMLArchive] PRIMARY KEY CLUSTERED
(
[ArchiveID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
GO
ALTER TABLE [dbo].[PageXMLArchive] ADD CONSTRAINT [DF_PageXMLArchive_ArchiveDate] DEFAULT (getdate()) FOR [ArchiveDate]
GO

Answer 1

You can avoid the loop in many ways, and here is an example...
SELECT
*
FROM
ListFileReport
WHERE
NOT EXISTS (
SELECT *
FROM page_data
WHERE p_content LIKE '%' + LTRIM(RTRIM(ListFileReport.FileName)) + '%'
)
AND
NOT EXISTS (
SELECT *
FROM PageXMLArchive
WHERE PageXML LIKE '%' + LTRIM(RTRIM(ListFileReport.FileName)) + '%'
)
Note: This removes the loop, and will yield a massive improvement because of that. But it still has to parse the whole of both lookup tables for every entry in ListFileReport, without any clever algorithmics, s their could be no useful indexing. So it will still be slow as a dog, it'll just have one broken leg instead of two.
The only way to avoid using LIKE is to parse all of the fields in the page_data and PageXMLArchive tables and create a list of referenced files. As HTML and XML are very structured, this can be done, but I'd look for a library or something to do it for you.
Then, you can create a another table with all of the files, without duplication, and with an appropriate index. Querying against that instead of using LIKE will be massively faster. I have no doubts at all. But writing or finding the code will be a chore.

Answer 2

A stored procedure especially has a loop with select and insert mixed will defiantly slow down the query.
ideally if you could insert into #table select a, b from table it will millions time faster than insert each row separately.
For your example, could do something like:
insert into #t (_fileName) select ... from p_content join ...on .. where sth like %sth
let me know if it is not applicable.