I have a requirement to create some xml structs (to borrow a C-phrase) in sql-server-2005. In order to do this, I change all my values to varchar. The problem arises when I want to make USE of these values, i have to convert them to decimal.
So, my xml code looks like this:
set #result = #result + <VAL>' + coalesce(cast(#val as varchar(20)), '-.11111') + '</VAL>'
this way, if VAL is null, I return a special decimal and I can check for that decimal. The drawback of doing this, is that I can't use coalesce on the other end when I use the value, I have to check if it converted value is equal to 0.
like this:
case when cast(InvestmentReturn.fn_getSTRUCT(...args...).value('results[1]/VAL[1]', 'varchar(40)')as decimal(10,5)) = -.11111
Since performance is unacceptable right now, I thought one way to improve performance might be to use coalesce instead of using a nested case statement and checking the value for equality with my special 'null' equivalent.
Any thoughts?
also, i see that select cast('null' as decimal(10,5)) gives me:
Msg 8114, Level 16, State 5, Line 1
Error converting data type varchar to numeric.
Performance issues can be caused by a number of factors.
The first one is using XML in sql 2005. I don't know the size of the xml data you are using but when I tried this 5 years ago if you crossed a certain size barrier (I think it was 32k, might have been 64k) then processing performance fell off the cliff. 1 extra byte would cause a query to go from 500ms to 60 seconds. We had to abandon letting SQL server deal with XML data itself at that point. It was MUCH faster to do that processing in C#.
The second one is making calls to functions inside a select statement. If that function has to operate on multiple rows, then performance goes down. One example I always use to illustrate this is GETDATE(). If you set a variable to the return of GETDATE() and then use that variable in a select query it will run an order of magnitude faster than calling GETDATE() in the query itself. The little code example you provided could be a killer just because it's calling a function.
This may not be a good answer to your immediate problem, but I really believe you would be much better served yanking any XML processing code out of SQL server and doing it in ANY OTHER language of your choice.
Related
I have a condition in a query:
ABS ( FIB.QUANT1 ) = ( OI.KLINE * :intv1000 ) / 1000.000000
when I run the query with intv1000 = 1000000 - query runs ok.
when I run the query with intv1000 = 1000 I get ORA-01722 (not immediately, after about 5-6 seconds).
Any Idea why ?
QUANT1 - NUMBER(16,2)
KLINE - NUMBER(38)
The condition is self generated from the application, So I can't really change it.
Thank you
The ORA-01722 is an invalid number error.
You would get this error when a non-number -- typically a string -- has to be converted to a number. For example here.
This conversion can occur in many different ways although the three most common are:
to_number()
cast()
implicit conversion
The expression you have highlighted may or may not have anything to do with where the error actually occurs. What it is saying is that:
When :intv1000 = 1000000 then the row(s) with the problem data are filtered out.
When intv1000 = 1000 then the row(s) with the problem data are being processed.
Arrggh. This is very hard to determine. I would suggest starting by looking at the query and finding all explicit conversions to see if they are a problem.
If you find no back data, then you need to resort to looking at all comparisons (including joins), arithmetic expressions, and function calls to find the problem.
In general, I strongly recommend avoiding implicit conversion. Use explicit conversion to avoid such problems! Note: I do make an exception for conversion to strings with string functions and operators. These are usually pretty safe.
Thanks to some wonderful application design, I've come to find myself face-to-face with a real WTF - it seems that the application I support outputs the date and time into two separate columns in one particular table; the date goes into a 'Date' column as the datetime data type, whilst the time goes into a 'Time' column as the money data type in minutes and seconds (so, for example, 10:35:00 would be £10.35).
I need to amalgamate these two columns during a query I'm making to the database so it returns as one complete datetime column but obviously just doing...
...snip...
CAST(au.[Date] as datetime) + CAST(au.[Time] AS datetime) as 'LastUpdateDate'
...snip...
... doesn't work as I hoped (naivély) that it would.
My predecessor encountered this issue and came up with a... "creative" solution to this:
MIN(DATEADD(HOUR,CAST(LEFT(CONVERT(VARCHAR(10),[time],0),CHARINDEX('.',CONVERT(VARCHAR(10),[time],0),0)-1) AS INT),DATEADD(MINUTE,CAST(RIGHT(CONVERT(VARCHAR(10),[time],0),LEN(CONVERT(VARCHAR(10),[time],0)) - CHARINDEX('.',CONVERT(VARCHAR(10),[time],0),0)) AS INT),[date]))) AS CreatedDateTime
Unlike my predecessor, I would rather try to keep this solution as simple as possible. Do you think it would be possible to cast the values in this column to time by:
Casting the money value to string
Replacing the decimal point for a colon
Parsing this as a datetime object (to replace the CAST(au.[Time] as datetime) part of the first code block
Is this feasible? And if not, can anyone assist?
EDIT
Just to be 100% clear, I cannot change the underlying data type for the column as the application relies on the data type being money. This is purely so my sanely-written application that does housekeeping reports can actually read the data in as a complete datetime value.
I'd prefer an arithmetical convertation without any string castings
MIN(
DATEADD(
MINUTE,
FLOOR(au.[Time]) * 60 + (au.[Time]-FLOOR(au.[Time])) * 100,
au.[Date])
) AS CreatedDateTime
You can add a layer of sanity, if changing the column to time outright is not an option:
ALTER TABLE ... ADD SaneDate AS
DATEADD(MINUTE, FLOOR([Time]) * 60 + 100 * ([Time] - FLOOR([Time])), [Date])
One computed column and then you can stick to using that instead of repeating the calculations everywhere. If altering the tables in any way is out of the question, you could at least make a view or table-valued function to capture this logic. (Preferably not a scalar function, although that's more obvious -- those have horrendous performance in queries.)
I tend to prefer DATEADD over string manipulation when possible, simply because the results tend to be more predictable. In this case there's no real issue, since converting money to char(5) is perfectly predictable regardless of language settings, but still.
Just had a look at how to use the REPLACE command and it works as expected:
CAST(au.[Date] as datetime) + CAST(REPLACE(CAST(au.[Time] AS varchar(5)),'.',':') AS datetime) as 'LastUpdateDate'
now outputs 2018-01-10 10:32:00.000 whereas before it was providing some incorrect date and time value.
I suppose you could mathematically convert it as #JeroenMostert has suggested - to be fair I'm not 100% on the performance impact this solution may have against calculating the minutes and converting it that way so I'll give that a try as well just to be sure.
Im writing a stored procedure that will convert float columns to varchar as part of its process when returning the data.
i dont want to convert everything to varchar(max) bec i think its probably more efficient not to. what is the largest size varchar i need to use -
convert(NVARCHAR(????), floatcolumn)
100?
i want to make sure i never get a result that looks like 8397Xe10
Presumably, you are using SQL Server (based on the code in your question). If you don't want exponential notation, then use the str() function (documented here). The length of the string doesn't have a large impact on performance, but you can do something like:
select str(floatcolumn, 100, 8) -- or whatever you think reasonable bounds are for your data
I am having to create a second header line and am using the first record of the Query to do this. I am using a UNION All to create this header record and the second part of the UNION to extract the Data required.
I have one issue on one column.
,'Active Energy kWh'
UNION ALL
,SUM(cast(invc.UNITS as Decimal (15,0)))
Each side are 11 lines before and after the Union and I have tried all sorts of combinations but it always results in an error message.
The above gives me "Error converting data type varchar to numeric."
Any help would be much appreciated.
The error message indicates that one of your values in the INVC table UNITS column is non-numeric. I would hazard a guess that it's either a string (VARCHAR or similar) column or something else - and one of the values has ended up in a state where it cannot be parsed.
Unfortunately there is no way other than checking small ranges of the table to gradually locate the 'bad' row (i.e. Try running the query for a few million rows at a time, then reducing the number until you home in on the bad data). SQL 2014 if you can get a database restored to it has the TRY_CONVERT function which will permit conversions to fail, enabling a more direct check - but you'll need to play with this on another system
(I'm assuming that an upgrade to 2014 for this feature is out of the question - your best bet is likely just looking for the bad row).
The problem is that you are trying to mix header information with data information in a single query.
Obviously, all your header columns will be strings. But not all your data columns will be strings, and SQL Server is unhappy when you mix data types this way.
What you are doing is equivalent to this:
select 'header1' as col1 -- string
union all
select 123.5 -- decimal
The above query produces the following error:
Error converting data type varchar to numeric.
...which makes sense, because you are trying to mix both a string (the header) with a decimal field.
So you have 2 options:
Remove the header columns from your query, and deal with header information outside your query.
Accept the fact that you'll need to convert the data type of every column to a string type. So when you have numeric data, you'll need to cast the column to varchar(n) explicitly.
In your case, it would mean adding the cast like this:
,'Active Energy kWh'
UNION ALL
,CAST(SUM(cast(invc.UNITS as Decimal (15,0))) AS VARCHAR(50)) -- Change 50 to appropriate value for your case
EDIT: Based on comment feedback, changed the cast to varchar to have an explicit length (varchar(n)) to avoid relying on the default length, which may or may not be long enough. OP knows the data, so OP needs to pick the right length.
A cohort of mine is building a somewhat long search query based on various input from the user. We've got NHibernate mapped up using Fluent NHibernate, and aside from some noob mistakes, all seems to be going well.
One issue we can't resolve in the case of this search is that for a particular parameter, NHibernate is creating sql that treats the input as int when we explicitly need it to be a string. We have a string property mapped to an nvarchar(255) column which mostly contains integer numbers, excluding some arbitrary inputs like "temporary" or long numbers like 4444444444444444 which is beyond the int limit.
In the course of testing, I've seen a couple things: 1) If I prepend a 0 to the incoming value, NH generates the sql param as a string, appropriately so; 2) If the value can realistically be converted to an int, the resulting sql treats it as so. In case #2, if I run the generated sql directly through sql server, I get an exception when the query comes across an non-integer value (such as the examples I listed above). For some reason, when I just let NH do it's thing, I'm getting appropriate records back, but it doesn't make sense; I would expect it to fail or at least tell me that something is wrong with some records that can't be evaluated by SqlServer.
The mapping is simple, the data store is simple; I would be ok leaving well enough alone if I at least understood why/how NHibernate is making this work when running the same state manually fails... Any thoughts?
Are you running the exact same code directly into SQL Server?
NHibernate parameterises all of its queries, and will in doing so define what value is passed through to SQL in the parameters. Which is probably what you're asking about, the reason SQL my fail, is that by default it will only know the difference if you input:
select * from table_name
where col_name = 5
in comparison with
select * from table_name
where col_name = '5'
If you do not define it as a string with the 's it will search for an int, and try to convert all the varchar's to ints, which will obviously fail in some cases with strings.