I would like to store large images in SQL Server and later use them in Reporting Services, Power BI and Analysis Services.
I found some help, but I still don't understand what is the best way to store them and how to convert these images in the correct format.
Should I convert them to Base64? How do I do that?
I found good explanations on Convert Image DataType To String in SQL Server, Storing images in SQL Server?, but none of them worked with me.
So I have on my database, the path of the image, the image it self and the extension as below:
SELECT
NM_DIRETORIO AS NM_PATH ,
NM_FOTO AS NM_PICTURE,
TP_EXTENSAO AS TP_EXTENSION
FROM D_GB_FOTOS
As I saw it on from this video SSRS - Read images from the SQL Server database, he uses varbinary(max) to store images, but I don't how he converted to that. Also, from Chriss Webb's: Storing Large Images In Power BI Datasets, he uses Base64 to show on Power BI.
So my question is, since I'll use large images, how to I convert a simple image (path + picture) to store in my SQL Server database?
Information:
SQL Server 2019 (v15.0.18330.0)
SQL Server Management Objects (SMO) v16.100.37971.0
Microsoft SQL Server Management Studio v18.5
EDIT:
Based on #Peter Schneider answer, I have created a cursor for updating the table, with that value. But I got error on the where clause (e.g: TABLE.ID_COLUMN).
My cursor:
DECLARE #ID_FOTO INT;
DECLARE #CD_ARQUIVO VARCHAR(4000);
DECLARE #CD_ARQUIVO_VARBINARY VARCHAR(4000);
DECLARE #tsql NVARCHAR (4000);
DECLARE CUR CURSOR FOR SELECT ID_FOTO, CD_ARQUIVO, NM_DIRETORIO + '\' + NM_FOTO + TP_EXTENSAO AS CD_ARQUIVO_VARBINARY FROM D_GB_FOTOS WHERE LINORIGEM <> 'CARGA MANUAL'
OPEN CUR
FETCH NEXT FROM CUR INTO #ID_FOTO, #CD_ARQUIVO, #CD_ARQUIVO_VARBINARY
WHILE #CD_ARQUIVO IS NULL BEGIN
SET #tsql = 'UPDATE D_GB_FOTOS' +
'SET CD_ARQUIVO = (SELECT CD_ARQUIVO.* from Openrowset(Bulk' + #CD_ARQUIVO + ', Single_Blob) CD_ARQUIVO)' +
'WHERE ' + #ID_FOTO + ' = D_GB_FOTOS.ID_FOTO;'
PRINT (#tsql)
EXEC (#tsql)
FETCH NEXT FROM cur INTO #ID_FOTO, #CD_ARQUIVO
END
CLOSE cur
DEALLOCATE cur
EDIT 2:
Some adjustments to the query, but there is one final problem where it keeps updating, and doesn't stop with the final ID of the table:
DECLARE #ID_FOTO INT;
DECLARE #CD_ARQUIVO VARCHAR(4000);
DECLARE #CD_ARQUIVO_VARBINARY VARCHAR(4000);
DECLARE #tsql NVARCHAR (4000);
DECLARE #ID_FOTO_MAX INT;
SET #ID_FOTO_MAX = (SELECT MAX(ID_FOTO) AS ID_FOTO FROM D_GB_FOTOS);
DECLARE CUR CURSOR FOR SELECT ID_FOTO, CD_ARQUIVO, (NM_DIRETORIO + '\' + NM_FOTO + TP_EXTENSAO) AS CD_ARQUIVO_VARBINARY FROM D_GB_FOTOS WHERE LINORIGEM <> 'CARGA MANUAL';
OPEN CUR
FETCH NEXT FROM CUR INTO #ID_FOTO, #CD_ARQUIVO, #CD_ARQUIVO_VARBINARY
WHILE (#ID_FOTO <= #ID_FOTO_MAX) BEGIN
SET #tsql = 'UPDATE D_GB_FOTOS ' +
'SET CD_ARQUIVO = (SELECT CD_ARQUIVO from Openrowset(Bulk ''' + #CD_ARQUIVO_VARBINARY + ''', Single_Blob) CD_ARQUIVO)' +
' WHERE D_GB_FOTOS.ID_FOTO = ' + CONVERT(VARCHAR(10),#ID_FOTO) + ';'
PRINT ('ID_FOTO: ' + CONVERT(VARCHAR(10),#ID_FOTO))
PRINT ('ID_FOTO_MAX: ' + CONVERT(VARCHAR(10),#ID_FOTO_MAX))
PRINT ('SELECT STATEMENT: ' + #tsql)
EXEC (#tsql)
FETCH NEXT FROM cur INTO #ID_FOTO, #CD_ARQUIVO, #CD_ARQUIVO_VARBINARY
END
CLOSE cur
DEALLOCATE cur
Example from where I printed the code above:
(1 linha afetada) ID_FOTO: 6529 ID_FOTO_MAX: 6531
(1 linha afetada) ID_FOTO: 6530 ID_FOTO_MAX: 6531
(1 linha afetada) ID_FOTO: 6531 ID_FOTO_MAX: 6531
(1 linha afetada) ID_FOTO: 6531 ID_FOTO_MAX: 6531
(1 linha afetada) ID_FOTO: 6531 ID_FOTO_MAX: 6531
You can use OpenRowSet to read the image from disk and insert it into your table
INSERT INTO YourTableName (ID, VarbinaryMaxColumn) VALUES (1,
(SELECT * FROM OPENROWSET(BULK N'C:\Temp\Testimage.png', SINGLE_BLOB) AS VarbinaryMaxColumn)
)
Related
I have to generate a very long procedure every time for a reporting system, so i created a template for my procedure and replacing the parts are needed to, but i could do it with Concat or +(&)
for example:
set #query = '... and (
--#InnerQueries
)'
set #query = replace(#query,'--#InnerQueries',#otherValues)
vs
set #query += ' and exists (...)'
if(#xxx is not null)
set #query += 'and not exists (...)'
with replace approach it's more readable and maintainable for me, but for sake of optimization, what about Concat and attaching string together?
with replace: there are a lot of searching but less string creation
and with concat: lot's of string creation but no searching
so any idea?
I assume you're talking about using CONCAT or REPLACE to build an SQL then run it. If ultimately you'll process fewer than 100 REPLACEments, I'd go with that approach rather than CONCAT because it's more readable.
If however, you're talking about using concat/replace to create report output data and you will e.g. be carrying out 100 REPLACE operations per row on a million rows, I'd do the CONCAT route
update 2:
there could be something missing here:
if i change first variable :#sourceText_Replace
to a max value of 8000 character, and continue to add to it:
set #sourceText_Replace += '8000 character length'
set #sourceText_Replace +=#sourceText_Replace
set #sourceText_Replace +=#sourceText_Replace
set #sourceText_Replace +=#sourceText_Replace
set #sourceText_Replace +=#sourceText_Replace
set #sourceText_Replace +=#sourceText_Replace
set #sourceText_Replace +=#sourceText_Replace
it works fine, even if go up until: 16384017 character length
so any idea here is as good as mine
orginal answer:
to summarize (and if i didnt make any mistakes):
if you are searching in a long text, dont even think about using replace, it took seconds not milliseconds, but for concat obviously does not make any difference
in the blew code, in first try(small text), i just used variables default values and did not append to them,
but for second try(long Text) , i just append result from previous loop run
for long text, i did not bothered to run the loop more than 20 time, because it took over minutes.
smallText: set #destSmallText_Replace =
longText: set #destSmallText_Replace +=
here is the code for test:
SET NOCOUNT ON
drop table if exists #tempReplace
drop table if exists #tempConcat
create table #tempReplace
(
[txt] nvarchar(max) not null
)
create table #tempConcat
(
[txt] nvarchar(max) not null
)
declare #sourceText_Replace nvarchar(max) = 'small1 text to replace #textToBeReplaced after param text'
declare #text_Replace nvarchar(max) = #sourceText_Replace
declare #textToSearch nvarchar(max) = '#textToBeReplaced'
declare #textToReplace nvarchar(max) = 'textToBeReplaced'
declare #concat_Start nvarchar(max) = 'small1 text to replace'
declare #concat_End nvarchar(max) = 'after param text'
declare #text_Concat nvarchar(max) = #concat_Start
declare #whileCounter int =0
declare #maxCounter int = 5
declare #startTime datetime = getdate();
declare #endTime datetime = getdate();
begin
set #startTime = getDate();
while(#whileCounter <=#maxCounter)
begin
--long text
set #text_Replace += replace(#sourceText_Replace,#textToSearch,#textToReplace + convert(nvarchar(10), #whileCounter)) + #textToSearch
--small text
--set #text_Replace = replace(#sourceText_Replace,#textToSearch,#textToReplace + convert(nvarchar(10), #whileCounter)) + #textToSearch
--print #destSmallText_Replace
insert into #tempReplace values(#text_Replace)
set #whileCounter+=1
end
set #endTime = getDate();
print 'passedTime ' + Convert(nvarchar(20), DATEPART(millisecond, #endTime) - DATEPART(millisecond, #startTime))
end
begin
set #whileCounter = 0;
set #startTime = getDate();
while(#whileCounter <=#maxCounter)
begin
set #text_Concat += concat(#concat_Start,#textToReplace + convert(nvarchar(10), #whileCounter),#concat_End) + #textToSearch
--print #sourceSmallText_Concat
insert into #tempConcat values(#text_Concat)
set #whileCounter+=1
end
set #endTime = getDate();
print 'passedTime ' + Convert(nvarchar(20), DATEPART(millisecond, #endTime) - DATEPART(millisecond, #startTime))
end
I have a stored procedure like the following:
declare #mySymbol as varchar(32)
DECLARE #RefGroupID AS VARCHAR(20)
declare #myJID as varchar(45)
declare Div_csr cursor for
select distinct Symbol2, RefGroupID, jnl_id
from #TDRows td
WHERE TD_CompanyCode_AU = 'A'
open Div_csr
fetch next from Div_csr into #mySymbol, #RefGroupID, #myJID
while ##FETCH_STATUS = 0 begin
SET #InfoMessage = #InfoMessage +CONVERT(varchar(30),getdate(), 120)+ ' ' + #SPname
+ ': RefGroupID:'+ convert(varchar(20),#RefGroupID,101)
+ ' Sym: ' + #mySymbol
+ ' JID: ' + #myJID
+ CHAR(13) + CHAR(10)
fetch next from Div_csr into #mySymbol, #RefGroupID, #myJID
end
close Div_csr
deallocate Div_csr
#mySymbol is the data I want to modified.
For example; #mySymbol is BXP PRB. You can see there is a space between BXP and PRB. I want to this space also show in the comment that was created using the stored procedure.
Right now, the commment's Sym: is parsed by data #mySymbol. But it does not have space now. I do not know why.. .
The comment looks like this:
"DIV2PAY Record receipt from CSH RecordDt=08/04/2017 Intended PayDt= 08/15/2017 Sym=BXPPRB Qty=-100 CashRate=0.328125 Kind= Cash Dividend RowID= 127278 CAEventID= 105226767
".
Could anyone help me with this?
I have written this in SQL:
EXEC sp_add_jobstep
#job_name = N'Every 5 minutes log backup ',
#step_name = N'set T-sql command',
#subsystem = N'TSQL',
#command=N'DECLARE #fileName NVARCHAR(256) BACKUP LOG MM TO DISK =''F:\BackUp\Log\M-''' +dbo.UDF_Gregorian_To_Persian_WithDash(cast(GETDATE() as date))+'''-'''+ REPLACE(cast(isnull(convert(char(5), GETDATE(), 108),'''''') as nvarchar),''':''','''''') + '''.TRN''',
#retry_attempts = 1,
#retry_interval = 1 ;
GO
but SQL says
"Incorrect syntaxt near +" !
Please help me!
You can pass variables into EXEC or literal values but what you're not allowed to use are expressions.
So do your string concatenation as a separate step first:
declare #cmd nvarchar(max)
set #cmd = 'DECLARE #fileName NVARCHAR(256) BACKUP ... ' + dbo.UDF_Gregorian_To_Persian_WithDash(...) + ' ... '
EXEC sp_add_jobstep
#job_name = N'Every 5 minutes log backup ',
#step_name = N'set T-sql command',
#subsystem = N'TSQL',
#command=#cmd,
#retry_attempts = 1,
#retry_interval = 1 ;
A bit of a mess with those apostophes. What you want is:
DISK ='F:\BackUp\Log\M-x-y.TRN'
But by looking at the current string it would rather be:
DISK ='F:\BackUp\Log\M-'x'-'y'.'TRN'
So, I hope this will work better:
#command = N'DECLARE #fileName NVARCHAR(256) BACKUP LOG MM TO DISK = ''F:\BackUp\Log\M-' + cast(dbo.UDF_Gregorian_To_Persian_WithDash(cast(GETDATE() as date)) as nvarchar) + '-' + REPLACE(cast(isnull(convert(char(5), GETDATE(), 108),'''''') as nvarchar),''':''','''''') + '.TRN'''
A tip would be to print this line so you can see exactly how the output will be. In my example above the output will be something like:
..DISK ='F:\BackUp\Log\M-2015-04-27-08:31.TRN'
Edit:
Not sure what UDF_Gregorian_To_Persian_WithDash returns, but unless it is a nvarchar you'll need to CAST() that too (updated my example above with the cast).
I just face a problem while working on a stored procedure,
My situation is as below,
I'm calling a stored procedure inside another stored procedure like for example,
EXEC [SP_ADMIN_INSERT_ITEM_STOCK_DETAILS]
#stk_tran_no = #cash_purchase_no,
#stk_tran_date = GetDate(),
#tran_type = 'Cash Purchase',
#item_code = #item_code,
#quantity = #quantity
Currently in the above code we are passing current date to the parameter #stk_tran_date.
But now I need to pass date to #stk_tran_date by fetching that from some other table like,
select #stk_tran_date = Convert(datetime,cash_purchase_date,103) from Cash_Purchase_14 where cash_purchase_no = 'GOH-9/2014'
If you observe my table name is like Cash_Purchase_14 where 14 is a dynamic value which changes every year, as this is 2014 financial year so it looks like Cash_Purchase_14, next year it will be Cash_Purchase_15.
Because of this i use to write these quires first as string then I'll execute them as shown below,
declare #SQL nvarchar(4000)
set #SQL =N' Declare #cash_purchase_date1 datetime
set #cash_purchase_date1 = (select cash_purchase_date from Cash_Purchase_'+ #Financialyearpart +' where cash_purchase_no = ''' + #cash_purchase_no + ''')
print #cash_purchase_date1'
exec executesql #SQL
But I need the value of the variable #cash_purchase_date1 outside this block like below,
EXEC [SP_ADMIN_INSERT_ITEM_STOCK_DETAILS]
#stk_tran_no = #cash_purchase_no,
#stk_tran_date = #cash_purchase_date1,
#tran_type = 'Cash Purchase',
#item_code = #item_code,
#quantity = #quantity
but it is giving an error like, "declare the variable #cash_purchase_date1"
In Other case i tried like calling the stored procedure in side the string like,
SET #SQL =' Declare #cash_purchase_date1 datetime
set #cash_purchase_date1 = (select cash_purchase_date from Cash_Purchase_'+ #Financialyearpart +' where cash_purchase_no = ' + #qt + #cash_purchase_no + #qt +')
print #cash_purchase_date1
EXEC [SP_ADMIN_INSERT_ITEM_STOCK_DETAILS]
#stk_tran_no = ' + #qt + #cash_purchase_no + #qt +',
#stk_tran_date = #cash_purchase_date1,
#tran_type = ''Cash Purchase'',
#item_code = ' + #qt + #item_code + #qt +',
#quantity = ' + #quantity
exec executesql #SQL
In this scenario the value of #cash_purchase_date1 is not replacing it simply retains the same.
Please help to get the value of the variable outside the block.
Or
How can I append the value 14 at the end of the table name dynamically using a variable.
I Tried like
Declare #cash_purchase_date1 datetime
set #cash_purchase_date1 = cash_purchase_date from Cash_Purchase_+ #Financialyearpart
I think i made the problem bit complicated while explaining. Please help me in solving the issue.
Thanks in advance.
You can return values from dynamic sql by using sp_executesql with an output variable:
declare #SQL nvarchar(4000);
declare #cash_purchase_date datetime;
set #SQL = N'select #cash_purchase_date = cash_purchase_date from Cash_Purchase_' + #Financialyearpart + ' where cash_purchase_no = ''' + #cash_purchase_no + '''';
exec sp_executesql #SQL, N'#cash_purchase_date datetime OUTPUT', #cash_purchase_date = #cash_purchase_date OUTPUT;
I think this will solve your problem.
I am looping through all my databases and aggregating the results into an aggregates database.
In my loop I call
master.dbo.xp_cmdshell osql C:\whatever.SQL
As the loop progresses, the cmdshell takes longer and longer to execute. If I stop the loop and run a single aggregate for one database it executes quickly.
Is there anything I can add to my external SQL script to make it run faster? Maybe something to commit and free the records before the next loop? Or should I add some kind of a pause after every loop?
I want to use an external SQL file because it contains many update statements and it's more manageable for me.
Here's how I loop:
Update dbFoo.dbo.tblBar set Processed = 0
Go
WHILE EXISTS ( SELECT ID FROM dbFoo.dbo.tblBar WHERE Processed = 0)
BEGIN
SELECT #aRow = MIN(tblBar.ID) FROM dbFoo.dbo.tblBar
SELECT #aFoo1 = Foo1 FROM dbFoo.dbo.tblBar WHERE ID = #aRow
SELECT #aFoo2 = Foo2 FROM dbFoo.dbo.tblBar WHERE ID = #aRow
SELECT #aFoo3 = Foo3 FROM dbFoo.dbo.tblWhatever WHERE Foo = #aFoo
EXEC RunPreAgg #Foo1 = #aFoo1, #Foo2 = #aFoo2, #Foo3 = #aFoo3, #RetVal = #aRetVal OUTPUT
SELECT returning = #aRetVal
UPDATE dbFoo.dbo.tblBar SET Processed = 1 WHERE ID = #aRow
END
Then the RunPreAgg stored procedure basically does this:
if db_id('db' + #Foo1 + '_' + #Foo2) is not null
BEGIN
--This bat file creates the SQL File
select #sql = 'master.dbo.xp_cmdshell '''+#path+'wwwRunPreAgg.bat ' + #Foo1 + ' ' + #Foo2 + ' ' + #Foo3 + ''''
exec( #sql )
--execute
select #sql = 'master.dbo.xp_cmdshell ''osql -E -o '+#path+'output\tmp'+#Foo1+'_'+#Foo2+'.txt -i '+#path+'tmp' + #Foo1 + '.SQL'''
exec( #sql )
--This erases the SQL File
select #sql = 'master.dbo.xp_cmdshell '''+#path+'wwwCleanup.bat ' + #Foo1 + ' ' + #Foo2 + ''''
exec( #sql )
Set #retval = 'Done!'
END
ELSE
BEGIN
Set #retval = 'Err: No DataBase'
END
The variable names are changed to protect the innocent. The code works fine, I just need to optimize.
If it is the loops performance that is causing you trouble, you might try reducing the number of selects. Normally I dislike Cursors, but your loop might benefit from one. You can select all the values you need for the loop into memory, then loop through those values without having to run 3 or 4 selects per loop (of course if the performance hit is occurring inside the RunPreAgg SP, then this won't help):
DECLARE cFoos CURSOR FOR
SELECT tblBar.ID, tblBar.Foo1, tblBar.Foo2, tblWhatever.Foo3
FROM dbFoo.dbo.tblBar
INNER JOIN dbFoo.dbo.tblWhatever
ON tblWhatever.Foo = tblBar.Foo
WHERE tblBar.Processed = 0;
OPEN cFoos;
FETCH NEXT FROM cFoos INTO #aRow, #aFoo1, #aFoo2, #aFoo3;
WHILE ##FETCH_STATUS = 0
BEGIN
EXEC RunPreAgg #Foo1 = #aFoo1, #Foo2 = #aFoo2, #Foo3 = #aFoo3, #RetVal = #aRetVal OUTPUT
SELECT returning = #aRetVal
UPDATE dbFoo.dbo.tblBar SET Processed = 1 WHERE ID = #aRow
FETCH NEXT FROM cFoos INTO #aRow, #Foo1, #Foo2, #Foo3;
END
CLOSE cFoos;
DEALLOCATE cFoos;