How to SUBSTR a calculated row in SQL Query? - sql

Original Query:
SELECT F4105.COUNCS/10000 FROM F4105
Output:
Numeric Expression
--------------------
111.1643000000000000
111.1633000000000000
111.1633000000000000
101.7654000000000000
101.7654000000000000
112.7258000000000000
I need to remove at least the last 5 zeroes. I tried to do a substring but it didn't work.
Here is the query(s) i tried:
(1)
SELECT SUBSTR((F4105.COUNCS/10000 AS 'co'),length((co)-5) FROM F4105
(2)
SELECT SUBSTR((F4105.COUNCS/10000),length((F4105.COUNCS/10000)-5)) FROM F4105
The 1st query gave me and error:
Token F4105 was not valid. Valid tokens: (.
The 2nd query worked by wrong output.
SUBSTR
00
000000
000000
000000
000000
000000

You are mixing the column alias definition in the expression. So, the correct expression is more like:
SELECT SUBSTR(F4105.COUNCS/10000, length(F4105.COUNCS/10000.0) - 5) as coFROM F4105
I wouldn't recommend doing this, however. You have a numeric expression. Just convert it to a decimal representation that you want, say:
SELECT CAST(F4105.COUNCS/10000.0 as DECIMAL(10, 5))

The syntax for SUBSTR scalar is effectively SUBSTR(expression, start-pos, for-length) IBM i 7.1->Database->Reference->SQL reference->Built-in functions->Scalar functions->SUBSTR
The LENGTH() expression shown used in the OP is specified for the second argument; i.e. the start-pos argument. As a starting position, the result of that string-length minus five calculation is conspicuously incorrect for obtaining the leftmost data; i.e. the starting-position is five bytes less than the length of the string. That would locate, of course, some insignificant zeroes five bytes from the end of the string-representation of the decimal-result of the division.
As effective correction therefore, would be either of • insert the constant integer value of 1 for the start-pos argument [thus making the LENGTH() expression become the third argument] • replace the SUBSTR scalar with the LEFT scalar.Either of those revisions would achieve something that at least resembles what is alluded as the desired output. However without either of the DDL and what should be the explicit output being expressed in the OP, the actual effect of those revised expressions could only be guessed.Anyhow, even with either of those changes, those suggested alternative character-string expressions remain as similarly poor [approaching daft] choice of expressions as the one in the OP, per lack of explicit casting; i.e. the two revised expressions suggested as possibly corrective [yet that remain similarly unlikely to yield desirable results] are:
SUBSTR((F4105.COUNCS/10000), 1,length((F4105.COUNCS/10000)-5))
LEFT((F4105.COUNCS/10000),length((F4105.COUNCS/10000)-5))
Having established data-type\length attributes using explicit casting [i.e. established even without some actual DDL to do so] in a derived-table expression that generates the input values that would produce the output shown in the OP, from a list of literal numeric values, the character-string expression in the following query ensures that only the eleven digits of decimal-precision to the right of the decimal point [i.e. the scale] are maintained; thus visually, the effect is that the trailing five digits are truncated:
with F4105 (COUNCS) as ( values
( dec( 1111643. , 9, 2 ) )
,( dec( 1111633. , 9, 2 ) )
,( dec( 1111633. , 9, 2 ) )
,( dec( 1017654. , 9, 2 ) )
,( dec( 1017654. , 9, 2 ) )
,( dec( 1127258. , 9, 2 ) )
)
SELECT cast( dec( (F4105.COUNCS/10000), 17, 11 ) as varchar(19) )
FROM F4105

Related

SELECT vs UPDATE, Unexpected rounding when using ABS function

Attached is a code sample to run in SQL. This seems like unexpected behavior for SQL Server. What should happen is to remove the negative from the number but when using the same function under the update command it does the absolute value and also rounds the number. Why is this?
DECLARE #TEST TABLE (TEST varchar(2048));
INSERT INTO #TEST VALUES (' -29972.95');
SELECT TEST FROM #TEST;
SELECT ABS(TEST) FROM #TEST;
UPDATE #TEST SET TEST = ABS(TEST);
SELECT TEST FROM #TEST;
Below are the results of that code.
-29972.95
29972.95
29973
This seems more a "feature" of the CONVERT function than anything to do with SELECT or UPDATE (only reason it is different is because the UPDATE implicitly converts the FLOAT(8) returned by ABS(...) back into VARCHAR).
The compute scalar in the update plan contains the expression
[Expr1003] = Scalar Operator(CONVERT_IMPLICIT(varchar(2048),
abs(CONVERT_IMPLICIT(float(53),[TEST],0))
,0) /*<-- style used for convert from float*/
)
Value - Output
0 (default) - A maximum of 6 digits. Use in scientific notation, when appropriate.
1 - Always 8 digits. Always use in scientific notation.
2 - Always 16 digits. Always use in scientific notation.
From MSDN: https://learn.microsoft.com/en-us/sql/t-sql/functions/cast-and-convert-transact-sql?view=sql-server-2017
This can be seen in the example below:
SELECT
[# Digits],
CONVERT(FLOAT(8), CONVERT(VARCHAR(20), N)) AS [FLOAT(VARCHAR(N))],
CONVERT(FLOAT(8), CONVERT(VARCHAR(20), N, 0)) AS [FLOAT(VARCHAR(N, 0))],
CONVERT(FLOAT(8), CONVERT(VARCHAR(20), N, 1)) AS [FLOAT(VARCHAR(N, 1))]
FROM (SELECT '6 digits', ABS('9972.95') UNION ALL SELECT '7 digits', ABS('29972.95')) T ([# Digits], N)
This returns the following results:
# Digits FLOAT(VARCHAR(N)) FLOAT(VARCHAR(N, 0)) FLOAT(VARCHAR(N, 1))
-------- ----------------- -------------------- --------------------
6 digits 9972.95 9972.95 9972.95
7 digits 29973 29973 29972.95
This proves the UPDATE was using CONVERT(VARCHAR, ABS(...)) effectively with the default style of "0". This limited the FLOAT from the ABS to 6 digits. Taking 1 character away so it does not overflow the implicit conversion, you retain the actual values in this scenario.
Taking this back to the OP:
The ABS function in this case is returning a FLOAT(8) in the example.
The UPDATE then caused an implicit conversion that was effectively `CONVERT(VARCHAR(2048), ABS(...), 0), which then overflowed the max digits of the default style.
To get around this behavior (if this is related to a practical issue), you need to specify the style of 1 or 2 (or even 3 to get 17 digits) to avoid this truncation (but be sure to handle the scientific notation used since it is now always returned in this case)
(some preliminary testing deleted for brevity)
It definitely has to do with silent truncating during INSERT/UPDATEs.
If you change the value insertion to this:
INSERT INTO #TEST SELECT ABS(' -29972.95')
You immediately get the same rounding/truncation without doing an UPDATE.
Meanwhile, SELECT ABS(' -29972.95') produces expected results.
Further testing supports the theory of an implicit float conversion, and indicates that the culprit lies with the conversion back to varchar:
DECLARE #Flt float = ' -29972.95'
SELECT #Flt;
SELECT CAST(#Flt AS varchar(2048))
Produces:
-29972.95
-29972
Probably final edit:
I was sniffing up the same tree as Martin. I found this.
Which made me try this:
DECLARE #Flt float = ' -29972.95'
SELECT #Flt;
SELECT CONVERT(varchar(2048),#Flt,128)
Which produced this:
-29972.95
-29972.95
So I'm gonna call this kinda documented since the 128 style is a legacy style that is deprecated and may go away in a future release. But none of the currently documented styles produce the same result. Very interesting.
ABS() is supposed to operate on numeric values and varchar input is converted to float. Most likely explanation for this behavior is that float has highest precedence among all numeric data types such as decimal, int, bit.
Your SELECT statement simply returns the float result. However the UPDATE statement implicitly converts the float back to varchar producing unexpected results:
SELECT
test,
ABS(test) AS test_abs,
CAST(ABS(test) AS VARCHAR(100)) AS test_abs_str
FROM (VALUES
('-29972.95'),
('-29972.94'),
('-29972.9')
) AS test(test)
test | test_abs | test_abs_str
----------|----------|-------------
-29972.95 | 29972.95 | 29973
-29972.94 | 29972.94 | 29972.9
-29972.9 | 29972.9 | 29972.9
I would suggest that you use explicit conversion and exact numeric datatype to avoid this and other potential problems with implicit conversions / floats:
SELECT
test,
ABS(CAST(test AS DECIMAL(18, 2))) AS test_abs,
CAST(ABS(CAST(test AS DECIMAL(18, 2))) AS VARCHAR(100)) AS test_abs_str
FROM (VALUES
('-29972.95'),
('-29972.94'),
('-29972.9')
) AS test(test)
test | test_abs | test_abs_str
----------|----------|-------------
-29972.95 | 29972.95 | 29972.95
-29972.94 | 29972.94 | 29972.94
-29972.9 | 29972.90 | 29972.90
ABS is a mathematical function, that means is designed to work with numeric values, you cannot expect a proper behavior of the function when using other data types like in this case VARCHAR, I suggest first to do the required CAST to a numeric data type before applying the ABS function as follows:
UPDATE #TEST SET TEST = ABS(CAST(TEST AS DECIMAL(18,2)))
After this your query will output
29972.95
This does not solve how it is posible that ABS works fine when selecting and not when updating a value, maybe it is a bug on sqlserver but also it is a really bad practice to avoid casting to proper data types required by functions. Maybe an implicit cast occurs when a SELECT clause is performed but ignored on UPDATE because microsoft is expecting you to do the right thing.

Postresql select the first three numbers including zeros

I did not expect this to be a problem, but I'm struggling to return the first 3 numbers, including the 0's before them. In the below examples, I show a few things I've tried. I want it to return '001'. It either returns '118' or an error. It seems like every solution wants to convert them to a text, which will drop the 0's.
SELECT lpad(00118458582::text, 3, '0')
returns 118
SELECT lpad(00118458582, 3, '0')
ERROR: function lpad(integer, integer, unknown) does not exist
SELECT left(00118458582::text, 3)
returns 118
SELECT left(00118458582, 3)
ERROR: function left(integer, integer) does not exist
SELECT substring(00118458582::text, 1, 3)
returns 118
Can I get any help please? Thanks!
Your problem starts before you try to get the first 3 digits, namely that you're considering 00118458582 to be a valid INTEGER (or whatever numeric type). I mean, it's not invalid, but what happens when you run SELECT 00118458582::INTEGER? You get 118458582. Because leading zeros in those types are senseless. So you'll never have a situation as in your examples (outside of a hardcoded number with leading zeros in your query window) in your tables, because those zeros wouldn't be stored in your number-based data type fields.
So the only way to get that sort of situation is when they're string-based: SELECT '00118458582'::TEXT returns 00118458582. And at that point you can run your preferred function to get the first 3 characters, e.g. SELECT LEFT('00118458582', 3) which returns 001. But if you're planning on casting that to INTEGER or something, forget about leading zeros.
SELECT substring(00118458582::text, 1, 3)
returns 118 because it is a number 118458582 (the leading zeros are automatically dropped), that is converted to text '118458582' and it then takes the first 3 characters.
If you are trying to take the first three digits and then convert to a number you can use try:
select substring('00118458582', 1,3::numeric)
it might actually be:
select substring('00118458582', 1,3)::numeric
I don't have a way to test right now...
lpad() refers to the total length of the returned value. So I think you want:
select lpad(00118458582::text, 12, '0'::text)
If you always want exactly 3 zeros before, then just concatenate them:
select '000' || 00118458582::text

Prevent ORA-01722: invalid number in Oracle

I have this query
SELECT text
FROM book
WHERE lyrics IS NULL
AND MOD(TO_NUMBER(SUBSTR(text,18,16)),5) = 1
sometimes the string is something like this $OK$OK$OK$OK$OK$OK$OK, sometimes something like #P,351811040302663;E,101;D,07112018134733,07012018144712;G,4908611,50930248,207,990;M,79379;S,0;IO,3,0,0
if I would like to know if it is possible to prevent ORA-01722: invalid number, because is some causes the char in that position is not a number.
I run this query inside a procedure a process all the rows in a cursor, if 1 row is not a number I can't process any row
You could use VALIDATE_CONVERSION if it's Oracle 12c Release 2 (12.2),
WITH book(text) AS
(SELECT '#P,351811040302663;E,101;D,07112018134733,07012018144712;G,4908611,50930248,207,990;M,79379;S,0;IO,3,0,0'
FROM DUAL
UNION ALL SELECT '$OK$OK$OK$OK$OK$OK$OK'
FROM DUAL
UNION ALL SELECT '12I45678912B456781234567812345671'
FROM DUAL)
SELECT *
FROM book
WHERE CASE
WHEN VALIDATE_CONVERSION(SUBSTR(text,18,16) AS NUMBER) = 1
THEN MOD(TO_NUMBER(SUBSTR(text,18,16)),5)
ELSE 0
END = 1 ;
Output
TEXT
12I45678912B456781234567812345671
Assuming the condition should be true if and only if the 16-character substring starting at position 18 is made up of 16 digits, and the number is equal to 1 modulo 5, then you could write it like this:
...
where .....
and case when translate(substr(text, 18, 16), 'z0123456789', 'z') is null
and substr(text, 33, 1) in ('1', '6')
then 1 end
= 1
This will check that the substring is made up of all-digits: the translate() function will replace every occurrence of z in the string with itself, and every occurrence of 0, 1, ..., 9 with nothing (it will simply remove them). The odd-looking z is needed due to Oracle's odd implementation of NULL and empty strings (you can use any other character instead of z, but you need some character so no argument to translate() is NULL). Then - the substring is made up of all-digits if and only if the result of this translation is null (an empty string). And you still check to see if the last character is 1 or 6.
Note that I didn't use any regular expressions; this is important if you have a large amount of data, since standard string functions like translate() are much faster than regular expression functions. Also, everything is based on character data type - no math functions like mod(). (Same as in Thorsten's answer, which was only missing the first part of what I suggested here - checking to see that the entire substring is made up of digits.)
SELECT text
FROM book
WHERE lyrics IS NULL
AND case when regexp_like(SUBSTR(text,18,16),'^[^a-zA-Z]*$') then MOD(TO_NUMBER(SUBSTR(text,18,16)),5)
else null
end = 1;

Check if number in varchar2 is greater than n

I've got a Varchar2 field which usually holds two alphabetic characters (such as ZH, SZ, AI,...). Let's call it FOO.
Certain datasets save A or A1 - A9 into the same field. I need to select all rows except exactly those.
I used the function substr to separate the number from the A. So far so good, < or > don't seem to work correctly with the "number-string".
How can I achieve this without converting it to a number? Is there an easier solution?
I haven't found anything on the internet and I reached my limit trying it myself.
This is my WHERE clause so far:
WHERE (substr(FOO, 0, 1) != 'A'
or (substr(FOO, 0, 1) = 'A' AND substr(FOO, 1, 1) > '9'));
It returns all the rows without restrictions.
The only solution I found:
WHERE (FOO NOT IN ('A', 'A1', 'A2', 'A3', 'A4', 'A5', 'A6', 'A7', 'A8', 'A9'));
But this is not optimal if, somewhere in the future, there will be A1 - A50. I would have to add 51 strings to my WHERE clause. And, since the query is in source code, also the code readability would get worse.
The solution should work on ORACLE and SQL Server.
Thanks in advance
(substr(FOO, 0, 1) = (substr(FOO, 1, 1) - Oracle starts with 1 (not 0).
So you should use substr(FOO, 2, 1) to get the second symbol.
However, it won't work in SQL Server which has SUBSTRING (not SUBSTR).
if you're ready to use different approaches in the different DBs you can also try regular expressions:
Oracle
where not regexp_like(foo, '^A[1-9]{1,3}$')
^ begining of the string
$ end of the string
[1-9] any digit from 1 to 9
{1,3} repeat the previous expression 1,2 or 3 times
Examples of FOOs which match / not match '^A[1-9]{1,3}$'
a123 -- may match / may not (depending on NLS settings regarding case sensitivity)
A123 -- match (the first symbol is 'A', the others are 3 digits)
A123b -- doesn't match (the last symbol should be a digit)
A1234 -- doesn't match (there should be 1,2 or 3 digits an the end)
A12 -- match
A1 -- match
SQL Server
REGEXP_LIKE conversion in SQL Server T-SQL
If your requirement is to include all alphabetic values except 'A' alone, consider using a LIKE expression so that it will work with any ANSI-compliant DBMS:
WHERE FOO <> 'A' AND FOO NOT LIKE '%[^A-Z]%'

Determine MAX Decimal Scale Used on a Column

In MS SQL, I need a approach to determine the largest scale being used by the rows for a certain decimal column.
For example Col1 Decimal(19,8) has a scale of 8, but I need to know if all 8 are actually being used, or if only 5, 6, or 7 are being used.
Sample Data:
123.12345000
321.43210000
5255.12340000
5244.12345000
For the data above, I'd need the query to either return 5, or 123.12345000 or 5244.12345000.
I'm not concerned about performance, I'm sure a full table scan will be in order, I just need to run the query once.
Not pretty, but I think it should do the trick:
-- Find the first non-zero character in the reversed string...
-- And then subtract from the scale of the decimal + 1.
SELECT 9 - PATINDEX('%[1-9]%', REVERSE(Col1))
I like #Michael Fredrickson's answer better and am only posting this as an alternative for specific cases where the actual scale is unknown but is certain to be no more than 18:
SELECT LEN(CAST(CAST(REVERSE(Col1) AS float) AS bigint))
Please note that, although there are two explicit CAST calls here, the query actually performs two more implicit conversions:
As the argument of REVERSE, Col1 is converted to a string.
The bigint is cast as a string before being used as the argument of LEN.
SELECT
MAX(CHAR_LENGTH(
SUBSTRING(column_name::text FROM '\.(\d*?)0*$')
)) AS max_scale
FROM table_name;
*? is the non-greedy version of *, so \d*? catches all digits after the decimal point except trailing zeros.
The pattern contains a pair of parentheses, so the portion of the text that matched the first parenthesized subexpression (that is \d*?) is returned.
References:
https://www.postgresql.org/docs/9.6/static/sql-createcast.html
https://www.postgresql.org/docs/9.6/static/functions-matching.html
Note this will scan the entire table:
SELECT TOP 1 [Col1]
FROM [Table]
ORDER BY LEN(PARSENAME(CAST([Col1] AS VARCHAR(40)), 1)) DESC