I am sending JSON telemetry data from our IoT Hub to Azure Data Lake Gen2 in the form of .parquet files. From the data lake I've then created a view in my Azure Synapse Serverless SQL pool that I can connect to and query data for reports.
CREATE VIEW DeviceTelemetryView
AS SELECT * FROM
OPENROWSET(
BULK 'https://test123.dfs.core.windows.net/devicetelemetry/*/*/*/*/*/',
FORMAT = 'PARQUET'
) AS [result]
This is what my view data looks like:
Most of these reports are based on date time ranges. Therefore I want to be able to write SQL queries that use my date time stamp.
The Current Issue
When I look at the current data type for the dateTimeStamp column, it defaults to varchar(8000) even though I believe my JSON is in the correct datetime format: "2021-11-29T21:45:00". How can I transform this specific field to a datetime field in my view to run queries on it?
When I look at the current data type for the dateTimeStamp column, it defaults to varchar(8000)
I think you would have to look at the data type for that column in the parquet file, it's likely to be a string in your case. Sql interpret as a varchar(8000).
even though I believe my JSON is in the correct datetime format: "2021-11-29T21:45:00".
Even if the timestamp format is correct, I think you'd have to tip system so it knows to cast that string to a datetime
How can I transform this specific field to a datetime field in my view to run queries on it?
I'm not an expert in sql but I think you can convert a string to a timestamp using cast and convert
CREATE VIEW DeviceTelemetryView
AS SELECT corporationid, deviceid, version, Convert(datetime, dateTimestamp, 126) AS NewDateFormat, data FROM
OPENROWSET(
BULK 'https://test123.dfs.core.windows.net/devicetelemetry/*/*/*/*/*/',
FORMAT = 'PARQUET'
) AS [result]
Related
I have dates in the format 01jan2020 (without a space or any separator) and need to convert this to a date type in SQL Server 2016 Management Studio.
The data was loaded from a .CSV file into a table (call it TestData, column is Fill_Date).
To join on a separate table to pull back data for another process, I need the TestData column Fill_Date to be in the correct format (MM-DD-YYYY) for my query to run correctly.
Fill_Date is currently in table TestData as datatype varchar(50).
I want to either see if it is possible to convert it with TestData table or directly insert the result into a 2nd table that is formatted.
Thanks (NEWB)
I ended up solving by converting the data while dropping into a temp table, deleting old value, and then inserting from that table back into the TestData table.
CONVERT(VARCHAR,CONVERT(date,[fill_date]),101) AS fill_date
I'm having old system which is developed in MS Access and I've a requirement to convert it a web application.
In the old application we have a MS Access Database and we need to import it to SQL Server.
Now there are many tables which have datetime columns with datatype as below:
Access: LongText
SQL Server: nvarchar(MAX)
I've successfully imported the MS Access database to SQL Server but now I'm stuck in datetime conversion issue.
There is a table which have around 86000 records and we need to convert it column to datatype datetime.
currently column have nvarchar(MAX) datatype with
"dd-mm-yyyy" and here is the screenshot for the same:
https://www.screencast.com/t/OP9edlsTdbR
I've tried to fix it by doing google but nothing work as expected.
convert your all string values to appropriate style : 105 : dd-mm-yyyy
ALTER TABLE dbo.TableName ALTER COLUMN DateNeed DATETIME NULL[NOT NULL]
GO
UPDATE TableName SET DateNeed = CONVERT(DATETIME, DateNeed , 105)
What is the appropriate Hive data type to store "hh:mm" into a Hive table? I've been using VARCHAR(5) however I've seen that that SMALLINT is used as well, this use case will be for a data warehouse where users will be able to filter data by this field. For example:
SELECT * FROM data WHERE air_time > '10:00' and air_time < '14:00'
For example, in sql server there is a TIME data type that was very convenient.
Any suggestions?
Varchar(5) is the most suitable data type. Looks like you don't need to do arithmetic on this data. Storing it in hh:mm varchar format allows you to do the comparison.
I have a data file I wish to import into Hive which contains timestamps. The timestamps are of the format MM/dd/yyyy HH:mm:ss.
I would like to create a table which contains a timestamp type to hold this value, however I can't figure out how to directly import the data.
My workaround is to import the data into a temporary table with my date as a string, then read the data from this temporary table into my permanent table doing the time format conversion on the fly.
So, my entire two-step load function looks something like this:
create table tempTable(
timeField string
)ROW FORMAT DELIMITED FIELDS TERMINATED BY ",";
--load data local here!
create table finalTable(
timeField timestamp
) stored as RCFILE;
insert into table finalTable select
from_unixtime( unix_timestamp(timeField,'MM/dd/yyyy HH:mm') )
from tempTable;
So finally my question :-)
Is this the 'right' or 'best' way to do it? Am I using an inefficient/stupid workaround?
thanks!
Another work around is change the datetime format of your data file as yyyy/MM/dd HH:mm:ss
Then it will directly cast the data as timestamp datatype to hive table.
Hope this help.
create table temptable(
timefield timestamp
)ROW FORMAT DELIMITED FIELDS TERMINATED BY ",";
This will cast your string 04/05/2014 04:25:55 as timestamp on hive.
I am working on pulling some data in from a Rest-api into my MSSQL database. The issue I am having is the timestamp that I am being given from the api does not appear to be formatted correctly to just insert '2013-09-16T07:00:00+0000'.
example insert:
INSERT INTO [page_fan_adds_unique]([period], [title], [description], [value], [end_time])
VALUES ('day', 'Daily New Likes', 'Daily: The number of new people who have liked your Page (Unique Users)','0', '2013-09-16T07:00:00+0000')
I know changing the format to 2013-09-16T07:00:00+00:00 works but I didn't want to have to manipulate the data before the insert.
You cannot do this without formatting.
From the MSDN about TIMESTAMP:-
Is a data type that exposes automatically generated, unique binary
numbers within a database. timestamp is generally used as a mechanism
for version-stamping table rows. The storage size is 8 bytes. The
timestamp data type is just an incrementing number and does not
preserve a date or a time. To record a date or time, use a datetime
data type.
You may use datetime data instead.
This is what I did. from my PHP code I inserted that date [end_time] = CAST(STUFF('".."',23,3,':00')AS DATETIME2)"
Now when I write my "select" statements I just cast(end_time as datetime) I find that this is a lot easier than trying to cast a string into a datetime in sql.