I'm having problems with dividing two float values that appears to be a simple process.
http://imgur.com/EZz5Q1Z
the floats l and larg have the values of 20 and 10, which should result in a n1 value of 2. That's not the case though.
Thanks in advance.
You put your breakpoint on the line containing the division, which means the debugger is showing the value of the variable before the division takes place. Step to the next instruction to see the correct value.
Related
I'm using google bigquery, and a column has values I want to round. If I do, and the rounded value ends in a zero, the zero is not displayed.
I've tried the function FORMAT, which apparently has some .number function, but I have no idea how to use it. Whenever I include any 2 things separated by a comma inside its brackets, it complains that it only takes 1 value.
You would use FORMAT() with the precision specifier. For four decimal places always -- including zeros:
select format('%.4f', 1.23)
If the BQ documentation does not answer your questions, I find that that the function seems to be inspired by the classic C printf()/sprintf() functions.
Unaware if in BigQuery (haven't used it ever) there is a better way I guess this will fix your problem since I just tried it in their console.
Cast your float to a string and then check if your last digit is a 0. In case it's not add it:
SELECT case when RIGHT(cast(0.9 as string), 1) <> '0' then cast(0.9 as string)+'0' else cast(0.9 as string) end as FormattedNumber
I am trying to run several files of code for an assignment. I am trying to solve an optimization problem using the "quadprog" function from the "optim" package.
quadprog is supposed to solve the optimization problem in a certain format and takes inputs H,f, A,b, Aeq, Beq, lb, ub.
The issue I am having involves my f which is a column vector of constants. To clarify, f looks like c*[1,1,1,1,1,1] where c is a constant. Quadprog seems to run my code just fine for certain values of c, but gives me the error:
error: index (_,49): but object has size 2x2
error: called from
quadprog at line 351 column 32
for other values of c. So, for example, 1/3 works, but 1/2 doesn't. Does anyone have any experience with this?
Sorry for not providing a working example. My code runs over several files and I seem to only be having problems with a specific value set that is very big. Thanks!
You should try the qp native Octave function.
You mention f is: c*[1,1,1,1,1,1] but, if c is a scalar, that is not a column vector. It seems very odd that a scalar value might produce a dimensions error...
I am trying to get the integer part of a number after dividing two variables.
ie, get 3 if the value is 3.75
displaycount and itemcount are both integer variables.
Dim cntr As Integer
cntr = Math.Floor(Math.Abs(itemCount / displaycount))
That code produces a blue squiggly in VS2012 with the comment that "runtime errors may occur when converting Double to Integer" BUT Math.Floor is supposed to take a decimal or double and return an integer.
"Math.Floor is supposed to take a decimal or double and return an integer." No, it isn't. It returns a value of the same type as its argument. See the documentation, e.g. Math.Floor Method (Double).
I would have expected VS to suggest a fix of adding CInt() around the RHS of the assignment; did that not appear for you?
If you need an Integer as result, consider using either the CInt, Int or the Fix functions.
CInt rounds to the nearest integer using the bankers's rounding (n.5 rounds towards the closest even number).
Int removes the fractional parts. Negative numbers are truncated towards smaller numbers
Int(-8.4) = -9.
Fix removes the fractional parts. Negative numbers are truncated towards greater numbers
Fix(-8.4) = -8.
See Conversion.Int Method and Type Conversion Functions (Visual Basic).
Can any one please help me how to get float value as it is from text box
for Ex: I have entered 40.7
rateField=[[rateField text] floatValue];
I am getting rateField value as 40.7000008 but I want 40.7 only.
please help me.
thanks in advance
Thanks Every body,
I tried all the possibilities but I am not able to get what I want. I am not looking to print the value to convert into string.I want to use that value for computation. If i use Number Formatter again when i am converting from number to float it is giving same problem.So i want float value only but it should be whatever i have given in the text box it should not be padded with any values.This is my requirement.Please help me.
thanks®ards Balu
Thanks Every body,
I tried all the possibilities but I am not able to get what I want. I am not looking to print the value to convert into string.I want to use that value for computation. If i use Number Formatter again when i am converting from number to float it is giving same problem.So i want float value only but it should be whatever i have given in the text box it should not be padded with any values.This is my requirement.Please help me.
thanks®ards
Balu
This is ok. There is not guaranteed that you will get 40.7 if you will use even double.
If you want to output 40.7 you can use %.1f or NSNumberFormatter
Try using a double instead. Usually solves that issue. Has to do with the storage precision.
double dbl = [rateField.text doubleValue];
When using floating point numbers, these things can happen because of the way the numbers are stored in binary format in the computers memory.
It's similar to the way 1/3 = 0.33333333333333... in decimal numbers.
The best way to deal with this is to use number formatters in the textbox that displays the value.
You are already resolved float value.
Floating point numbers have limited precision. Although it depends on
the system, float relative error due to rounding will be around 1.1e-8
Non elementary arithmetic operations may give larger errors, and, of
course, error progragation must be considered when several operations
are compounded.
Additionally, rational numbers that are exactly representable as
floating point numbers in base 10, like 0.1 or 0.7, do not have an
exact representation as floating point numbers in base 2, which is
used internally, no matter the size of the mantissa. Hence, they
cannot be converted into their internal binary counterparts without a
small loss of precision. This can lead to confusing results: for
example, floor((0.1+0.7)*10) will usually return 7 instead of the
expected 8, since the internal representation will be something like
7.9999999999999991118....
So if you're using those numbers for output, you should use some rounding mechanism, even for double values.
SQL is detecting that the following string ISNUMERIC:
'07213E71'
I believe this is because the 'E' is being classed as a mathmatical symbol.
However, I need to ensure that only values which are whole integers are returned as True.
How can I do this?
07213E71 is a floating number 7213 with 71 zeros
You can use this ISNUMERIC(myValue + '.0e0') to test for whole integers. Slightly cryptic but works.
Another test is the double negative myValue NOT LIKE '%[^0-9]%' which allows only digits 0 to 9.
ISNUMERIC has other issues in that these all return 1: +, -,
To nitpick: This is a whole integer. It is equivalent to 7213 * 10 ^ 71.
In the documentation it says
ISNUMERIC returns 1 when the input expression evaluates to a valid integer, floating point number, money or decimal type; otherwise it returns 0. A return value of 1 guarantees that expression can be converted to one of these numeric types.
Your number is also float (with exponential notation), therefore the only way to have ISINTEGER is to define it yourself on SQL. Read the following link.
http://classicasp.aspfaq.com/general/what-is-wrong-with-isnumeric.html
Extras:
http://www.sqlteam.com/forums/topic.asp?TOPIC_ID=59049
http://www.tek-tips.com/faqs.cfm?fid=6423
I have encountered the same problem. IsNumeric accepts "$, €, +, -, etc" as valid inputs and Convert function throws errors because of this.
Using "LIKE" SQL statement fixed my problem. I hope it'll help the others
SELECT UnitCode, UnitGUID, Convert(int, UnitCode) AS IntUnitCode
FROM [NG_Data].[NG].[T_GLB_Unit]
WHERE ISNULL(UnitType,'') <>'Department'
AND UnitCode NOT LIKE '%[^0-9]%'
ORDER BY IntUnitCode
PS: don't blame me for using "UnitCode" as nvarchar :) It is an old project :)
You have to ensure it out of the call to the database, whatever the language you work with, and then pass the value to the query. Probably the SQL is understanding that value as a string.