Need re-factoring / indexing advice for a very large table - sql

Question

Ok, so I have a table that's just become a monster. And querying on it has become insanely slow for some of our customers. Here's the table in question:
CREATE TABLE [EventTime](
[Id] [bigint] IDENTITY(1,1) NOT NULL,
[EventId] [bigint] NOT NULL,
[Time] [datetime] NOT NULL,
CONSTRAINT [PK_EventTime] PRIMARY KEY CLUSTERED
(
[Id] ASC
)
)
CREATE NONCLUSTERED INDEX [IX_EventTime_Main] ON [EventTime]
(
[Time] ASC,
[EventId] ASC
)
It has a FK to the Events table. An event is action taken from a certian user, ip, service, and accountId. This EventTime table tells us what events happened at what time. An event can happen today at 3am and also 12pm last week. The idea is to not duplicate event rows.
Now this EventTime table has become massive for some customers; our biggest being 240mill rows and growing. And querying it has become insanely slow when looking at a time set > a few days. Here's the query we're executing today (Note: I'm running queries locally from a rip of the DB to minimize network latency or TO's caused by collectors hitting the DB):
SELECT
a.TrailId, a.[NameId], a.[ResourceId], a.[AccountId], a.[ServiceId]
FROM [EventTime] b WITH (NOLOCK) INNER JOIN [Event] a WITH (NOLOCK) ON a.Id = b.EventId
WHERE
a.TrailId IN (1, 2, 3, 4, 5) AND
a.NameId IN (6) AND
b.[Time] >= '2014-10-29 00:00:00.000' AND
b.[Time] <= '2014-11-12 23:59:59.000'
ORDER BY b.[Time] ASC
Note, trailId is a column in the Event table that tells us what customer to filter down to in the query. We have the list of TrailIds before we execute this query. Now this query very slow, about 45mins to execute. Here's some queries I've tried:
SELECT
a.EventId, a.[NameId], a.[ResourceId], a.[AccountId], a.[ServiceId]
FROM [EventTime] b WITH(NOLOCK)
JOIN [Event] a WITH(NOLOCK) on a.Id = b.EventId
WHERE
b.EventId IN (SELECT Id from [Event] where TrailId IN (1, 2, 3, 4, 5) AND NameId IN (6) ) AND
b.[Time] >= '2014-08-01 00:00:00.000' AND
b.[Time] <= '2014-11-12 23:59:59.000' AND
ORDER BY b.[Time] ASC
subquery worked well for small queries but for larger date ranges the performance suffered greatly. Next I tried
DECLARE #ListofIDs TABLE(Ids bigint)
INSERT INTO #ListofIDs (Ids)
SELECT Id from Event where TrailId IN (140, 629, 630, 631, 632) AND NameId IN (468)
SELECT
a.EventId, a.[NameId], a.[ResourceId], a.[AccountId], a.[ServiceId]
FROM [EventTime] b WITH(NOLOCK)
JOIN [Event] a WITH(NOLOCK) on a.Id = b.EventId
WHERE
b.EventId IN (SELECT Ids FROM #ListofIDs) AND
b.[Time] >= '2014-08-01 00:00:00.000' AND
b.[Time] <= '2014-11-12 23:59:59.000' AND
ORDER BY b.[Time] ASC
Casting my subquery into a table array for my main query to reference did help a bit. The query took about 33mins. But's it's still way way too slow =/
Next I tried playing with indexes. I figured I might have been putting too much into one index. So I dropped the existing and broke it out into two.
CREATE NONCLUSTERED INDEX [IX_EventTime_Main] ON [EventTime]
(
[Time] ASC,
)
GO
CREATE NONCLUSTERED INDEX [IX_EventTime_Event] ON [EventTime]
(
[EventId] ASC
)
This didn't seem to do anything. Same query times.
I think the core issue is, this table is just very unorganized. The Time column has very specific time values and none of them are in order. For example, customer 8's collector might be saving EventTimes for 2014-11-12 04:12:01.000 and customer 10 is saving 2015-03-15 13:59:21.000. So the query has to process and sort all these dates prior to filtering down. So indexing [Time] probably isn't effective at all.
Anyone have any ideas on how I can speed this up?

Answer 1

This is your query:
SELECT e.TrailId, e.[NameId], e.[ResourceId], e.[AccountId], e.[ServiceId]
FROM [EventTime] et WITH (NOLOCK) INNER JOIN
[Event] e WITH (NOLOCK)
ON e.Id = et.EventId
WHERE e.TrailId IN (1, 2, 3, 4, 5) AND
e.NameId = 6 AND
et.[Time] >= '2014-10-29 00:00:00.000' AND
et.[Time] <= '2014-11-12 23:59:59.000'
ORDER BY et.[Time] ASC
The best indexes for this query are probably: Event(NameId, TrailId), EventTime(EventId, Time). This assume that the result set is not humongous (tens of millions of rows), in which case an optimization to get rid of the order by would be desirable.

Answer 2

I would ditch the ID column and make the primary key a composite clustered one on EventId and Time:
CREATE TABLE [EventTime](
[EventId] [bigint] NOT NULL,
[Time] [datetime] NOT NULL,
CONSTRAINT [PK_EventTime] PRIMARY KEY CLUSTERED
(
[EventId] ASC
, [Time] ASC
)
)
CREATE NONCLUSTERED INDEX [IX_EventTime_Main] ON [EventTime]
(
[Time] ASC,
[EventId] ASC
);
Check the execution plans to see of the non-clustered index is used and drop it of not needed.