Azure Stream Analytics - SQL - sql

.json is Input to Azure Stream Analytics. One of the column (body) is in BASE64 format. I can covert this in SQL Server Management Studio with following query.
declare #v varchar(1000) = 'cm9sZToxIHByb2R1Y2VyOjEyIHRpbWVzdGFtcDoxNDY4NjQwMjIyNTcxMDAwIGxhdGxuZ3tsYXRpdHVkZV9lNzo0MTY5ODkzOTQgbG9uZ2l0dWRlX2U3Oi03Mzg5NjYyMTB9IHJhZGl1czoxOTc2NA=='
SELECT
CAST(
CAST('' AS XML).value('xs:base64Binary(sql:column("BASE64_COLUMN"))', 'VARBINARY(MAX)')
AS VARCHAR(MAX)
) AS Result
FROM
(
SELECT #v AS BASE64_COLUMN
) A
But when trying to execute in Stream Analytics Query is warning with Error saying :
Function 'value' is either not supported or not usable in this context. User defined function calls must start with "udf."
Query:
SELECT
CAST(
CAST('' AS XML).value('xs:base64Binary(sql:column("BASE64_COLUMN"))', 'VARBINARY(MAX)')
AS VARCHAR(MAX)
) AS Result
INTO
[IOTPowerBIStreaming]
FROM
(
SELECT body AS BASE64_COLUMN FROM [IOTInput]
) A

Function 'value' is either not supported or not usable in this
context. User defined function calls must start with "udf."
This error message indicates that the ASA runtime mistakenly assumes that you want to execute user defined function in ASA SQL.The syntax is udf.XXX().
ASA SQL still has many different behaviors from normal SQL, actually.The base64 convert method in your question is not supported in ASA SQL.
You could define user defined function is ASA so that you could implement same requirement.For example, please refer to below part from this document.
var Base64={_keyStr:"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=",
encode:function(e){var t="";var n,r,i,s,o,u,a;var f=0;e=Base64._utf8_encode(e);while(f<e.length){n=e.charCodeAt(f++);r=e.charCodeAt(f++);i=e.charCodeAt(f++);s=n>>2;o=(n&3)<<4|r>>4;u=(r&15)<<2|i>>6;a=i&63;if(isNaN(r)){u=a=64}else if(isNaN(i)){a=64}t=t+this._keyStr.charAt(s)+this._keyStr.charAt(o)+this._keyStr.charAt(u)+this._keyStr.charAt(a)}return t},
decode:function(e){var t="";var n,r,i;var s,o,u,a;var f=0;e=e.replace(/++[++^A-Za-z0-9+/=]/g,"");while(f<e.length){s=this._keyStr.indexOf(e.charAt(f++));o=this._keyStr.indexOf(e.charAt(f++));u=this._keyStr.indexOf(e.charAt(f++));a=this._keyStr.indexOf(e.charAt(f++));n=s<<2|o>>4;r=(o&15)<<4|u>>2;i=(u&3)<<6|a;t=t+String.fromCharCode(n);if(u!=64){t=t+String.fromCharCode(r)}if(a!=64){t=t+String.fromCharCode(i)}}t=Base64._utf8_decode(t);return t},
_utf8_encode:function(e){e=e.replace(/\r\n/g,"n");var t="";for(var n=0;n<e.length;n++){var r=e.charCodeAt(n);if(r<128){t+=String.fromCharCode(r)}else if(r>127&&r<2048){t+=String.fromCharCode(r>>6|192);t+=String.fromCharCode(r&63|128)}else{t+=String.fromCharCode(r>>12|224);t+=String.fromCharCode(r>>6&63|128);t+=String.fromCharCode(r&63|128)}}return t},
_utf8_decode:function(e){var t="";var n=0;var r=c1=c2=0;while(n<e.length){r=e.charCodeAt(n);if(r<128){t+=String.fromCharCode(r);n++}else if(r>191&&r<224){c2=e.charCodeAt(n+1);t+=String.fromCharCode((r&31)<<6|c2&63);n+=2}else{c2=e.charCodeAt(n+1);c3=e.charCodeAt(n+2);t+=String.fromCharCode((r&15)<<12|(c2&63)<<6|c3&63);n+=3}}return t}};
var encodedString = Base64.encode(input);
return encodedString;

Related

I want to extract columns from a single message column sql query

I have a logs message that I have to extract columns from with a sql query,
this how the message looks like:
"device=EOHCS-ZA-JIS-FW severity=high from=EOHCloudFAZ(FL1KVM0000005594) trigger=Syslog Critical System Alerts log="logver=54 itime=1528457940 devid=FG1K5D3I13800425 devname=FWJIS01 vd=95_LHC date=2018-06-08 time=13:34:55 logid=0100044546 type=event subtype=system level=information logdesc="Attribute configured" user="JoshuaK" ui="ha_daemon" action=Edit cfgtid=701760128 cfgpath="system.settings" cfgattr="gui-allow-unnamed-policy[disable->enable]" msg="Edit system.settings """
can someone give me an idea
I have a solution for SQL-Server, you could use PATINDEX and extract the log message.
Below is the code to extract from value
declare #input nVarchar(max),#from nVarchar(MAX)
declare #FromStart int,#FromEnd int
set #input='device=EOHCS-ZA-JIS-FW severity=high from=EOHCloudFAZ(FL1KVM0000005594) trigger=Syslog Critical System Alerts log="logver=54 itime=1528457940 devid=FG1K5D3I13800425 devname=FWJIS01 vd=95_LHC date=2018-06-08 time=13:34:55 logid=0100044546 type=event subtype=system level=information logdesc="Attribute configured" user="JoshuaK" ui="ha_daemon" action=Edit cfgtid=701760128 cfgpath="system.settings" cfgattr="gui-allow-unnamed-policy[disable->enable]" msg="Edit system.settings';
SET #FromStart=PATINDEX('%from=%',#input)+5;
SET #FromEnd=PATINDEX('% trigger=%',#input)-#FromStart;
SELECT #from=SUBSTRING(#input,#FromStart,#FromEnd)
SELECT #from
Note : use equivalent of PATINDEX for your corresponding DB server. Also note that this works only if the input string have parameters in a defined order.

Retrieving values from Json in SQL Server

I have a table , which has a column with json Data
[
{"Name":"Territory","Value":"1000000002"},{"Name":"Village","Value":"Bahaibalpur"},
{"Name":"Activity Date","Value":"2016-6-15 5:30:0"},
{"Name":"Start Time","Value":"16:37"},
{"Name":"End Time","Value":"17:38"},
{"Name":"Duration","Value":"1hrs 1 mins"}
]
. I want to get Start time value and end time value inside function.
Try this way
declare #t table(col varchar(1000))
insert into #t(col)
select '[{"Name":"Territory","Value":"1000000002"},{"Name":"Village","Value":"Bahaibalpur"},
{"Name":"Activity Date","Value":"2016-6-15 5:30:0"},
{"Name":"Start Time","Value":"16:37"},
{"Name":"End Time","Value":"17:38"},
{"Name":"Duration","Value":"1hrs 1 mins"}]'
select
substring(col,charindex('Start Time',col)+21,5) as start_time,
substring(col,charindex('end Time',col)+19,5) as end_time
from #t
You don't state which version of SQL you are using, 2016 has built in functionality(https://msdn.microsoft.com/en-us/library/dn921897.aspx)
Otherwise, there is a cool article on parsing Json with TSQL that you can read here.
The other option is to use a CLR function.

Sql Server 2012 How to Get List on Function

Function Details:
alter FUNCTION siralama_puan (#suggestion_id int)
RETURNS int
AS
Begin
Declare #comment_count int,#like_count int,#favorite_count int,#date_point int,#suggestion_point int,#suggestion_date datetime,#fark int
set #comment_count=(select [Suggestion].CommentCount from [Suggestion] where [Suggestion].Id= #suggestion_id)
set #like_count=(select [Suggestion].LikeCount from [Suggestion] where [Suggestion].Id=#suggestion_id)
set #favorite_count=(select [Suggestion].FavoriteCount from [Suggestion] where [Suggestion].Id=#suggestion_id)
set #suggestion_date=(select [Suggestion].Crtm from [Suggestion] where [Suggestion].Id=#suggestion_id)
set #fark =(select DATEDIFF(day,#suggestion_date,GETDATE()))
if #fark<6
set #date_point=30
else if #fark<10 and #fark>=6
set #date_point=20
else
set #date_point=10
set #suggestion_point=(#comment_count*2)+(#like_count)+(#favorite_count*3)+#date_point
RETURN #suggestion_point
End
Calling Function:
select dbo.siralama_puan (122280,1) as puan order by puan desc
but it didn't work.Error:Procedure or function dbo.siralama_puan has too many arguments specified.Multiple arguments not working.
It is not possible to define one parameter of type INT and pass in a list. But there are several approaches:
TYPE
Create your own type with CREATE TYPE read here.... This allows you to pass in a list very similar to a (read only) table
XML
Let your parameter be of type XML and pass in something like
<root>
<prm value="122280"/>
<prm value="1"/>
</root>
It's quite easy to transform this XML within your function to a list with something like
SELECT Prm.value('#value','int') AS PrmValue
FROM #prm.nodes('/root/prm') AS One(Prm)
CSV
A parameter of type VARCHAR(MAX) and a comma separated list like "122280,1". Find a way to split this here: section "Dynamic IN-statement"

Setting a Clob value in a native query

Oracle DB.
Spring JPA using Hibernate.
I am having difficulty inserting a Clob value into a native sql query.
The code calling the query is as follows:
#SuppressWarnings("unchecked")
public List<Object[]> findQueryColumnsByNativeQuery(String queryString, Map<String, Object> namedParameters)
{
List<Object[]> result = null;
final Query query = em.createNativeQuery(queryString);
if (namedParameters != null)
{
Set<String> keys = namedParameters.keySet();
for (String key : keys)
{
final Object value = namedParameters.get(key);
query.setParameter(key, value);
}
}
query.setHint(QueryHints.HINT_READONLY, Boolean.TRUE);
result = query.getResultList();
return result;
}
The query string is of the format
SELECT COUNT ( DISTINCT ( <column> ) ) FROM <Table> c where (exact ( <column> , (:clobValue), null ) = 1 )
where "(exact ( , (:clobValue), null ) = 1 )" is a function and "clobValue" is a Clob.
I can adjust the query to work as follows:
SELECT COUNT ( DISTINCT ( <column> ) ) FROM <Table> c where (exact ( <column> , to_clob((:stringValue)), null ) = 1 )
where "stringValue" is a String but obviously this only works up to the max sql string size (4000) and I need to pass in much more than that.
I have tried to pass the Clob value as a java.sql.Clob using the method
final Clob clobValue = org.hibernate.engine.jdbc.ClobProxy.generateProxy(stringValue);
This results in a java.io.NotSerializableException: org.hibernate.engine.jdbc.ClobProxy
I have tried to Serialize the Clob using
final Clob clob = org.hibernate.engine.jdbc.ClobProxy.generateProxy(stringValue);
final Clob clobValue = SerializableClobProxy.generateProxy(clob);
But this appears to provide the wrong type of argument to the "exact" function resulting in (org.hibernate.engine.jdbc.spi.SqlExceptionHelper:144) - SQL Error: 29900, SQLState: 99999
(org.hibernate.engine.jdbc.spi.SqlExceptionHelper:146) - ORA-29900: operator binding does not exist
ORA-06553: PLS-306: wrong number or types of arguments in call to 'EXACT'
After reading some post about using Clobs with entities I have tried passing in a byte[] but this also provides the wrong argument type (org.hibernate.engine.jdbc.spi.SqlExceptionHelper:144) - SQL Error: 29900, SQLState: 99999
(org.hibernate.engine.jdbc.spi.SqlExceptionHelper:146) - ORA-29900: operator binding does not exist
ORA-06553: PLS-306: wrong number or types of arguments in call to 'EXACT'
I can also just pass in the value as a String as long as it doesn't break the max string value
I have seen a post (Using function in where clause with clob parameter) which seems to suggest that the only way is to use "plain old JDBC". This is not an option.
I am up against a hard deadline so any help is very welcome.
I'm afraid your assumptions about CLOBs in Oracle are wrong. In Oracle CLOB locator is something like a file handle. And such handle can be created by the database only. So you can not simply pass CLOB as bind variable. CLOB must be somehow related to database storage, because this it can occupy up to 176TB and something like that can not be held in Java Heap.
So the usual approach is to call either DB functions empty_clob() or dbms_lob.create_temporary (in some form). Then you get a clob from database even if you think it is "IN" parameter. Then you can write as many data as you want into that locator (handle, CLOB) and then you can use this CLOB as a parameter for a query.
If you do not follow this pattern, your code will not work. It does not matter whether you use JPA, SpringBatch or plan JDBC. This constrain is given by the database.
It seems that it's required to set type of parameter explicitly for Hibernate in such cases. The following code worked for me:
Clob clob = entityManager
.unwrap(Session.class)
.getLobHelper()
.createClob(reader, length);
int inserted = entityManager
.unwrap(org.hibernate.Session.class)
.createSQLQuery("INSERT INTO EXAMPLE ( UUID, TYPE, DATA) VALUES (:uuid, :type, :data)")
.setParameter("uuid", java.util.Uuid.randomUUID(), org.hibernate.type.UUIDBinaryType.INSTANCE)
.setParameter("type", java.util.Uuid.randomUUID(), org.hibernate.type.StringType.INSTANCE)
.setParameter("data", clob, org.hibernate.type.ClobType.INSTANCE)
.executeUpdate();
Similar workaround is available for Blob.
THE ANSWER: Thank you both for your answers. I should have updated this when i solved the issue some time ago. In the end I used JDBC and the problem disappeared in a puff of smoke!

Expression evaluator in sql query SELECT

I have this query
Declare #q varchar(255) = 'A+B'
Select
A, -- 1
B, -- 2
C,
Evaluate(#q) MySum -- 3
From tbl_temp
Assuming the #q is also a column from tbl_temp
Is there any method sql that do something like that?
I have in mind, a function that accepts the expression and also the columns. Just asking if there's a better way than this.
Any help or suggestions will be appreciated.
Thanks
You could create CLR function to evaluate expressions:
[Microsoft.SqlServer.Server.SqlFunction(DataAccess=DataAccessKind.Read)]
public static object Evaluate(SqlString expression, SqlString values)
{
string[] variables = values.Value.Split(',');
Dictionary<string, string> vars = new Dictionary<string, string>();
foreach(string var in variables)
{
string[] varAndValue = var.Split('=');
vars.Add(varAndValue[0], varAndValue[1]);
}
string expr = expression.Value;
foreach(string var in vars.Keys)
{
expr = expr.Replace(var, vars[var]);
}
using (SqlConnection connection = new SqlConnection("context connection=true"))
{
connection.Open();
SqlCommand command = new SqlCommand(string.Format("SELECT {0}", expr), connection);
object result = command.ExecuteScalar();
return result;
}
}
then use it in sql as follow:
declare #t table(A int, B int, Expr varchar(max))
insert into #t(A,B,Expr)
select 5,6, 'A+B' union all
select 1,2, 'A*B'
select dbo.Evaluate(Expr, 'A=' + cast(A as varchar) + ',' + 'B=' + cast(B as varchar))
from #t
Disclaimer: I'm the owner of the project Eval SQL.NET
There is no "eval" method in SQL however this can be easily done using Eval SQL.NET Library.
-- CREATE expression
DECLARE #sqlnet SQLNET = SQLNET::New('A+B')
-- EVALUATE expression from a table
SELECT #sqlnet.SetValue('A', A).SetValue('B', B).Eval()
FROM tbl_temp
Documentation: Dynamically evaluate arithmetic operation and expression in SQL
The basic answer is no. There is no easy way to do this. In general, if you have such expressions, then you would use dynamic SQL to parse them. And, you cannot use dynamic SQL in a function.
This leaves you two basic options. The first is to use your own parsing routine, probably in a CLR (written in an external language). This is a feasible approach if you really have a need for this functionality.
The second is to use one of the stored procedures that you are allowed to use in a function: xp_cmdshell. Note: using this stored procedure is not recommended as a general solution, because of security issues and special permissions. Also, getting the value calculated from the stored procedure requires some cleverness (essentially echo out the results into a pre-defined table variable).