Store hourly data efficient way - sql

There is a requirement to store hourly data in SQL Server 2016 and retrieve. It's an OLTP database.
I will explain with an example: we need to capture temperature of each city of a country and store on hourly basis. What would be the best and efficient design to do this. The data would be stored for a year and then archived
This is my plan. Can some one review and let me know if this approach is fine?
CREATE TABLE [dbo].[CityMaster]
(
[CityId] [int] NULL,
[CityName] [varchar](300) NULL
) ON [PRIMARY]
--OPTION 1
CREATE TABLE [dbo].[WeatherData]
(
[Id] [bigint] NULL,
[CityId] [int] NULL,
[HrlyTemp] [decimal](18, 1) NULL,
[CapturedTIme] [datetime] NULL
) ON [PRIMARY]
GO
--OPTION2
CREATE TABLE [dbo].[WeatherData_JSon]
(
[Id] [bigint] NULL,
[CityId] [int] NULL,
[Month] [varchar](50) NULL,
[Hrlytemp] [nvarchar](max) NULL
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
GO

To some extent, this depends on how the data is going to be used. The most natural solution is to tweak the first option and use a partitioned table:
CREATE TABLE [dbo].CityHourlyTemperatures (
Id bigint NULL,
CityId int NULL,
HrlyTemp decimal(6, 1) NULL,
CapturedTIme datetime NULL
) ;
Note I changed the name to something that seems to better capture the name.
Even with global warming, I think that 4 or 5 digits of precision in the temperature is quite sufficient -- 18 is way overkill.
Each row here has 8 + 4 + 5 + 8 bytes = 25 bytes (it may be rounded up if there are alignment restrictions). A year has about 8,766 hours. So, if you have 100 cities, this is less than a million rows per year and just a few tens of megabytes per year.
That is quite feasible, but you might want to consider partitioning the table -- the older partitions can act like an "archive".
Your second option stores the temperatures as a blob. This would make sense under only one circumstance: you don't care about the temperatures but you need to return the data to an application that does.
The implication from the name is that you want to store the value as JSON. This usually requires more overhead than storing the data using native types -- and is often less efficient. JSON is very powerful and useful for some types of data, particularly sparse data. However, your data is quite relational and can be stored in a relational format.
If you wanted to save space, you could consider the following:
Replacing the datetime value with an hourId column. This could possibly be a shortint if you only want a few years of data.
Removing the id column and defining the cityid/hourid as the primary key.
However, your volume of data does not seem to suggest that such an approach is necessary.

Option 2 is not feasible.
Option 1 is better.
I think HrlyTemp column's size should be [decimal](4, 1) or [decimal](5, 1) max.
If you deal with all the world's cities data, based on an approximation of total cities in the world ~1000.
Then you need to store 365*24*1000 = 8,760,000 ~ 9M rows per year. To stay on the safe side, we can assume that we have to store 10M data.
Which is OK for the SQL Server.

CREATE TABLE [dbo].[WeatherData]
(
[Id] [bigint] NULL,
[CityId] [int] NULL,
[HrlyTemp] [decimal](18, 1) NULL,
[CapturedTIme] [datetime] NULL,
[DailyTempRepo] nvarchar(max) * record create and update per day
) ON [PRIMARY]
GO
* you can normalize more your table.
how to store data in DailyTempRepo column as json :
[{"TempDate":"2021-03-06","CityId":"2","CapturedTIme":"09:30","HrlyTemp":"70"},
{"TempDate":"2021-03-06","CityId":"2","CapturedTIme":"10:30","HrlyTemp":"78"},
{"TempDate":"2021-03-06","CityId":"2","CapturedTIme":"11:30","HrlyTemp":"81"}]

Related

How to deal with large amount of XML data in a SQL Server database

In a table there are 10 columns and 2 of those columns store huge amounts of data. One column (XML datatype) stores XML data, another column (NVARCHAR(MAX)) stores JSON data. Each rows size around 1.4 MB. Moreover, when I am using a SELECT command it takes a lot of time (22 seconds) to load only one record
TABLE - [dbo].[CampaignConfiguration](
[CampaignId] [uniqueidentifier] NOT NULL,
[ConfigurationXml] [xml] NOT NULL,
[ConfigurationRules] [nvarchar](max) NOT NULL,
[RulesVersion] [nvarchar](50) NULL,
[Created] [datetime] NOT NULL,
[CampaignCalculationParameters] [nvarchar](max) NULL
Select *
From CampaignConfiguration
Where campaignid = 'EB5C2CDB-C076-5174-61D1-D9EA0E04975A';
Is there any better way to deal with to improve SELECT query performance?

Dealing with huge table - 100M+ rows

I have table with around 100 million rows and it is only getting larger, as table is queried pretty frequently I have to come up with some solution to optimise this.
Firstly here is the model:
CREATE TABLE [dbo].[TreningExercises](
[TreningExerciseId] [uniqueidentifier] NOT NULL,
[NumberOfRepsForExercise] [int] NOT NULL,
[CycleNumber] [int] NOT NULL,
[TreningId] [uniqueidentifier] NOT NULL,
[ExerciseId] [int] NOT NULL,
[RoutineExerciseId] [uniqueidentifier] NULL)
Here is Trening table:
CREATE TABLE [dbo].[Trenings](
[TreningId] [uniqueidentifier] NOT NULL,
[DateTimeWhenTreningCreated] [datetime] NOT NULL,
[Score] [int] NOT NULL,
[NumberOfFinishedCycles] [int] NOT NULL,
[PercentageOfCompleteness] [int] NOT NULL,
[IsFake] [bit] NOT NULL,
[IsPrivate] [bit] NOT NULL,
[UserId] [nvarchar](128) NOT NULL,
[AllRoutinesId] [bigint] NOT NULL,
[Name] [nvarchar](max) NULL,
)
Indexes (other than PK which are clustered):
TreningExercises:
TreningId (also FK)
ExerciseId (also FK)
Trenings:
UserId (also FK)
AllRoutinesId (also FK)
Score
DateTimeWhenTreningCreated (ordered by DateTimeWhenTreningCreated DESC)
And here is the example of the most commonly executed query:
DECLARE #userId VARCHAR(40)
,#exerciseId INT;
SELECT TOP (1) R.[TreningExerciseId] AS [TreningExerciseId]
,R.[NumberOfRepsForExercise] AS [NumberOfRepsForExercise]
,R.[TreningId] AS [TreningId]
,R.[ExerciseId] AS [ExerciseId]
,R.[RoutineExerciseId] AS [RoutineExerciseId]
,R.[DateTimeWhenTreningCreated] AS [DateTimeWhenTreningCreated]
FROM (
SELECT TE.[TreningExerciseId] AS [TreningExerciseId]
,TE.[NumberOfRepsForExercise] AS [NumberOfRepsForExercise]
,TE.[TreningId] AS [TreningId]
,TE.[ExerciseId] AS [ExerciseId]
,TE.[RoutineExerciseId] AS [RoutineExerciseId]
,T.[DateTimeWhenTreningCreated] AS [DateTimeWhenTreningCreated]
FROM [dbo].[TreningExercises] AS TE
INNER JOIN [dbo].[Trenings] AS T ON TE.[TreningId] = T.[TreningId]
WHERE (T.[UserId] = #userId)
AND (TE.[ExerciseId] = #exerciseId)
) AS R
ORDER BY R.[DateTimeWhenTreningCreated] DESC
Execution plan: link
Please accept my apologies if it is bit unreadable or unoptimised, it was generated by ORM (Entity Framework), I just edited it a bit.
According to Azure's SQL Analytics tool this query has the most impact on my DB and even though it usually doesn't take too long to execute, from time to time there are spikes in DB I/O due to it.
Also there is a bit business logic involved in this, to simplify it: 99% of the time I need data which is less then a year old.
What are my best options regarding querying and table size?
My thoughts on querying, either:
Create indexed view OR
Add Date and UserId fields to the TreningExerciseId table OR
Some option that I haven't thought of :)
Regarding table size, either:
Partition table (probably by date) OR
Move most of the data (or all of it) to some NoSQL key-value store OR
Some option that I haven't thought of :)
What are your thoughts about these problems, how should I approach solving them?
If you add the following columns to the index "ix_TreninID":
NoOfRepsForExecercise
ExerciseID
RoutineExerciseID
That will make the index a "covering index" and eliminate the need for the lookup which is taking 95% of the plan.
Give it a go, and post back.

SQL Store varchar or int when I have only a set of values

I would like to know if it is better (subjective i know) to store an integer of values or a string of values when the field only has a set of possible values. E.g.
Person Table
1.
Name Age Category
Joe 25 0
Jane 28 2
John 22 1
2.
Name Age Category
Joe 25 Student
Jane 28 Teacher
John 22 Staff
Which method is advisable? Method 1 is probably faster and better for querying, however, there is more programming cost when displaying data.
Method 2 is probably slower, more expressive and less programming cost.
Any advise will be useful.
Thanks in advance
You would generally do this using a reference table, with the category, and an integer for linking the tables.
A reference table has multiple advantages:
The list of possible values is available in one place. This is handy, for instance, for generating a list in an application.
There are no misspellings.
You can store additional information, such as a short name, a long description, honorific, etc.
If you need multi-lingual support, you have all the values in a single place.
The same values can be shared across multiple tables.
Sometimes, a reference table isn't appropriate. For instance, you might have just two values, ON and OFF. You can validate the values using a CHECK CONSTRAINT in most databases. That is a reasonable alternative. But I suspect that the category has more information than just a handful of values.
May be you are looking for a simple respons but, I woud like share my method,
I have a Table named CustomCaptions.
You can find here the structure of the table.
CREATE TABLE [dbo].[CustomCaptions](
[Capt_ID] [int] IDENTITY(1,1) NOT NULL,
[Capt_Code] [varchar](50) NULL,
[Capt_Family] [varchar](50) NULL,
[Capt_FR] [nvarchar](100) NULL,
[Capt_EN] [nvarchar](100) NULL,
[Capt_ES] [nvarchar](100) NULL,
[Capt_IT] [nvarchar](100) NULL,
[Capt_TR] [nvarchar](100) NULL,
[Capt_CS] [nvarchar](100) NULL,
[Capt_DE] [nvarchar](100) NULL,
[Capt_Deleted] [bit] NULL,
[Capt_Order] [smallint] NULL,
CONSTRAINT [PK_CustomCaptions] PRIMARY KEY CLUSTERED
(
[Capt_ID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
The Capt_Family column is the name of your foreign column.
In your case, in the CustomCaptions Table, I keep the code and the family.
For exemple,
Capt_Family Capt_Code Capt_EN ... Capt_Order
Category 0 Student 0
Category 1 Teacher 1
Category 2 Staff 2
Such, for all little tables like category, status, type etc... I use only one table, this reduces the total count of tables in my db.
Also, I have only one methode to fill comboboxes or listboxes by giving only the family name. And also only one screen to edit the content of any list.
And also as you can see in the table structure, you can manage multi language easily in your application.
Depending your need, you can use Capt_EN or another column for another language.
Finaly, if you wish, you can create views which will reduce the programming cost.
I hope this helps.

SQL - Joining to Nonexistent Records

After doing a bit of looking, I thought maybe I'd found a solution here: sql join including null and non existing records. Cross Joining my tables seems like a great way to solve my problem, but now I've hit a snag:
Below are the tables I’m using:
CREATE TABLE [dbo].[DCRSales](
[WorkingDate] [smalldatetime] NOT NULL,
[Store] [int] NOT NULL,
[Department] [int] NOT NULL,
[NetSales] [money] NOT NULL,
[DSID] [int] IDENTITY(1,1) NOT NULL)
CREATE TABLE [dbo].[Stores](
[Number] [int] NOT NULL,
[Has_Deli] [bit] NOT NULL,
[Alcohol_Register] [int] NULL,
[Is_Cost_Saver] [bit] NOT NULL,
[Store_Status] [nchar](10) NOT NULL,
[Supervisor_Number] [int] NOT NULL,
[Email_Address] [nchar](20) NOT NULL,
[Sales_Area] [int] NULL,
[PZ_Store_Number] [int] NULL,
[Has_SCO] [bit] NULL,
[SCO_Reg] [nchar](25) NULL,
[Has_Ace] [bit] NULL,
[Ace_Sq_Ft] [int] NULL,
[Open_Date] [datetime] NULL,
[Specialist] [nchar](2) NULL,
[StateID] [int] NOT NULL)
CREATE TABLE [dbo].[DepartmentMap](
[Department_Number] [int] NOT NULL,
[Description] [nvarchar](max) NOT NULL,
[Parent_Department] [int] NOT NULL)
CREATE TABLE [dbo].[ParentDepartments](
[Parent_Department] [int] NOT NULL,
[Description] [varchar](50) NULL
DCRSales is a table holding new and archived data. The archived data is not perfect, meaning that there are of course certain missing date gaps and some stores which currently have a department they didn't have or no longer have a department they used to have. My goal is to join this table to our department list, list the child departments and parent departments and SUM up the netsales for a given date range. In cases where a store does not have a department whatsoever in that date range, I still need to display it as 0.00.
A more robust solution would probably be to just store all departments for each store regardless of whether they have that department or not (with sales set to 0.00 of course). However I imagine doing that and/or solving my problem here would require very similar queries anyway.
The query I have tried is as follows:
WITH CTE AS (
SELECT S.Number as Store, DepartmentMap.Department_Number as Department, ParentDepartments.Parent_Department as Parent, ParentDepartments.Description as ParentDescription, DepartmentMap.Description as ChildDescription
FROM Stores as S CROSS JOIN dbo.DepartmentMap INNER JOIN ParentDepartments ON DepartmentMap.Parent_Department = ParentDepartments.Parent_Department
WHERE S.Number IN(<STORES>) AND Department_Number IN(<DEPTS>)
)
SELECT CTE.Store, CTE.Department, SUM(ISNULL(DCRSales.NetSales, 0.00)) as Sales, CTE.Parent, CTE.ParentDescription, CTE.ChildDescription
FROM CTE LEFT JOIN DCRSales ON DCRSales.Department = CTE.Department AND DCRSales.Store = CTE.Store
WHERE DCRSales.WorkingDate BETWEEN '<FIRSTDAY>' AND '<LASTDAY>' OR DCRSales.WorkingDate IS NULL
GROUP BY CTE.Store, CTE.Department, CTE.Parent, CTE.ParentDescription, CTE.ChildDescription
ORDER BY CTE.Store ASC, CTE.Department ASC
In this query I try to CROSS JOIN each department to a store from the Stores table so that I can get a combination of every store and every department. I also include the Parent Departments of each department along with the child department's description and the parent department's description. I filter this first portion based on store and department, but this does not change the general concept.
With this result set, I then try to join this table to all of the sales in DCRSales that are within a certain date range. I also include the date if it’s null because the results that have a NULL sales also have a NULL WorkingDate.
This query seemed to work, until I noticed that not all departments are being used with all stores. The stores in particular that do not get combined with all departments are the ones that have no data for the given date range (meaning they have been closed). If there is no data for the department, it should still be listed with its department number, parent number, department description and parent description (with Sales as 0.00). Any help is greatly appreciated.
Your WHERE clause is filtering out records that do have sales at some point in time, but not for the desired period of time, those records don't meet either criteria and are therefore excluded.
I might be under-thinking it, but might just need to move:
DCRSales.WorkingDate BETWEEN '<FIRSTDAY>' AND '<LASTDAY>'
To your LEFT JOIN criteria and get rid of WHERE clause. If that's not right, you could filter sales by date in a 2nd cte prior to the join.
What you want is an OUTER JOIN.
See this: https://technet.microsoft.com/en-us/library/ms187518(v=sql.105).aspx
I suggest that this process is probably much too complicated to be done using a single query. I think that you need to perform several queries: to extract the transactions-of-interest into a separate table, then to modify the results one-or-more times in that table before using it to produce your final statistics. A stored procedure, which drives several separately stored queries, could be used to drive the process, which, in several "stages," makes several "passes" over the initially-extracted data.
One piece of important information, for instance, would be to know when a particular store had a particular department. (For instance: store, department, starting_date, ending_date.) This would be a refinement of a possibly-existing table (and, maybe, drawn from it ...) which lists what departments a particular store has today.
Let us hope that department-numbers do not change, or that your company hasn't acquired other companies with the resulting need to "re-map" these numbers in some way.
Also: frankly, if you have access to a really-good statistics package, such as SAS® or SPSS® ... can "R" do this sort of thing? ... you might find yourself better-off. (And no, I don't mean "Microsoft Excel ...") ;-)
When I have been faced with requirements like these (and I have been, many times ...), a stats-package was indispensable. I found that I had to "massage" the process and the extracted data a number of times, in a system of successive-refinement that gradually lead me to a reporting process that I could trust and therefore defend.

SQL Table handle large amounts of records

I need to make sure that a table of mine can handle in excess of 1,000,000 records.
Can I have some advice on my table code to determine if it can indeed handle this amount of records.
Here is my code:
USE [db_person_cdtest]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [Person](
[PersonID] [numeric](18, 0) IDENTITY(1,1) NOT NULL,
[ID] [varchar](20),
[FirstName] [varchar](50) NOT NULL,
[LastName] [varchar](50) NOT NULL,
[AddressLine1] [varchar](50),
[AddressLine2] [varchar](50),
[AddressLine3] [varchar](50),
[MobilePhone] [varchar](20),
[HomePhone] [varchar](20),
[Description] [varchar](10),
[DateModified] [datetime],
[PersonCategory] [varchar](30) NOT NULL,
[Comment] [varchar](max),
CONSTRAINT [PK_Person] PRIMARY KEY CLUSTERED
(
[PersonID] DESC
)WITH (IGNORE_DUP_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY];
Almost any table structure in almost any database can handle a million records. That is not a large number of records for a modern computer running modern software.
Your structure looks reasonable. One question is whether the fields are always large enough to hold the value in the data. It looks like you are using SQL Server. There is no difference in storage or performance to declaring a varchar(50) versus a varchar(8000). "50" seems on the low side to me.
Another comment is that you have a DateModified column. I would suggest that you also keep a history table of the modifications. It is often important to know what changed, when it changed, and what the values were before the change.
In more advanced systems, you would not be storing a person's address and telephone number in the same table as their unique ids. A person could have more than one address (shipping address, billing address, home address, etc.). A person could have many telephone numbers (landline number, mobile number, work number, work mobile, etc.). And, you have no fields for email address, Facebook id, and so on. Contact information is more complex than a few fields in a table.
Finally, as a matter of habit, I almost always include the following fields at the end of every table:
CreatedBy varchar(255) default system_user,
CreataedAt datetime not null default getdate()
This let's me know who and when a row was created.