I'm trying a few decimal calculations, in a Rails app, but am getting some weird results.
#pprice = item.price - (( item.price / 100 ) * promo.discount)
#pprice = item.price - promo.discount
Each value (item.price, promo.discount etc) is a decimal. However, when trying to calculate I get mixed results, none of them being the correct total. The results range from nil, 0.0 and -2.0 depending on whether I have .to_f included.
The first calculation is based on a percentage discount and the second on a straight money reduction.
I have performed the calculations in the view (to test) and they display correctly, but when they are moved over to the controller, I get the errors.
Any help is much appreciated.
since u r converting the values to float, ur calculation results arent precise.
see String, decimal, or float datatype for price field?
Related
I am in Google's Data Analytics course and every now and then I get to a spot where I could just press on, but I want to know the answer anyway.
We are learning embedded calculations and a very simply equation was given:
`
SELECT
Date,
Region,
Total_Bags,
Small_Bags,
(Small_Bags / Total_Bags) * 100 AS Small_Bags_Percent
FROM `shaped-fx-370703.avocado_data.avocado_prices`
WHERE
Total_Bags <> 0
`
Resulting column from calculation
I would like to know how to use ROUND so that the result in Small_Bags_Percent only go out to 1 decimel
I understand the syntax of round is ROUND(integer,decimel places) but I don't know how to point the "integer" to the results in Small_Bags_Percent. I've tried THEN and even
ROUND(((Small_Bags / Total_Bags) * 100 AS Small_Bags_Percent),2)
``` (hey, I'm just seeing what sticks lol)
The ROUND function goes round the calculation but not the alias for the calculation e.g.
ROUND((Small_Bags / Total_Bags) * 100,2) AS Small_Bags_Percent
I have a table with three columns: traffic INTEGER, revtraffic INTEGER, ratio REAL.
When I update the ratio column in the following way:
UPDATE table
SET ratio = revtraffic/traffic
It returns 0.0 in all cells. However, if write:
SET ratio = revtraffic*100/traffic
then it displays the right results but obviously, 100x magnitude too much. What is going on?
revtraffic and traffic are integers, so dividing them will be done by integer division, which returns only the "whole" part of the division (i.e, the part to the left of the decimal point). This result is then promoted to a real when you assign it to a real.
You can work around this problem by explicitly casting one of the arguments to a real:
UPDATE table SET ratio = (CAST revtraffic AS REAL)/traffic
So I have two identical values that are results of sum functions with the exact same length (no rounding is being done). (Update, data type is float)
Value_1 = 29.9539194336501
Value_2 = 29.9539194336501
The issue I'm having is when I do an IF statement for Value_1 = Value_2, it comes up as FALSE.
Value_1:
SELECT SUM([INVN_DOL])/SUM([AVG_DLY_SLS_LST_35_DYS]) end as DSO
FROM TABLE A
Value_2:
SELECT SUM ([Total_Inventory_Val]) / SUM ([Daily_Independent_Demand])
FROM TABLE B
Any idea why they may not be exactly equal and what I can do to get a TRUE value since they do match?
Thanks in advance
The issue you are having here is that your are using a calculated value that is held within a float, which will by design be slightly imprecise at higher levels of precision, which is why you are getting your mismatch.
Use data types like decimal with a defined precision and scale to hold your values and calculation results and you should get consistent results.
You can make use ROUND to limit the decimal points
Or
Try the ABS and see if that works out.
I want to store a value that represents a percent in SQL server, what data type should be the prefered one?
You should use decimal(p,s) in 99.9% of cases.
Percent is only a presentation concept: 10% is still 0.1.
Simply choose precision and scale for the highest expected values/desired decimal places when expressed as real numbers. You can have p = s for values < 100% and simply decide based on decimal places.
However, if you do need to store 100% or 1, then you'll need p = s+1.
This then allows up to 9.xxxxxx or 9xx.xxxx%, so I'd add a check constraint to keep it maximum of 1 if this is all I need.
decimal(p, s) and numeric(p, s)
p (precision):
The maximum total number of decimal digits that will be stored (both to the left and to the right of the decimal point)
s (scale):
The number of decimal digits that will be stored to the right of the decimal point (-> s defines the number of decimal places)
0 <= s <= p.
p ... total number of digits
s ... number of digits to the right of the decimal point
p-s ... number of digits to the left of the decimal point
Example:
CREATE TABLE dbo.MyTable
( MyDecimalColumn decimal(5,2)
,MyNumericColumn numeric(10,5)
);
INSERT INTO dbo.MyTable VALUES (123, 12345.12);
SELECT MyDecimalColumn, MyNumericColumn FROM dbo.MyTable;
Result:
MyDecimalColumn: 123.00 (p=5, s=2)
MyNumericColumn: 12345.12000 (p=10, s=5)
link: msdn.microsoft.com
I agree, DECIMAL is where you should store this type of number. But to make the decision easier, store it as a percentage of 1, not as a percentage of 100. That way you can store exactly the number of decimal places you need regardless of the "whole" number. So if you want 6 decimal places, use DECIMAL(9, 8) and for 23.3436435%, you store 0.23346435. Changing it to 23.346435% is a display problem, not a storage problem, and most presentation languages / report writers etc. are capable of changing the display for you.
I think decimal(p, s) should be used while s represents the percentage capability.
the 'p' could of been even 1 since we will never need more than one byte since each digit in left side of the point is one hunderd percent, so the p must be at least s+1, in order you should be able to store up to 1000%.
but SQL doesn't allow the 'p' to be smaller than the s.
Examples:
28.2656579879% should be decimal(13, 12) and should be stored 00.282656579879
128.2656579879% should be decimal(13, 12) and should be stored 01.282656579879
28% should be stored in decimal(3,2) as 0.28
128% should be stored in decimal(3,2) as 1.28
Note: if you know that you're not going to reach the 100% (i.e. your value will always be less than 100% than use decimal(s, s), if it will, use decimal(s+1, s).
And so on
The datatype of the column should be decimal.
I can't get sum() to return decimal and it always returns int64 truncating the decimals. I have Googled for a whole day but still can't find a real work around. I have a DB table called ProductPurchase with
QtyPurchased(int)
and
UnitPurchasePrice(money) columns,
these are mapped to a C# POCO object using NHibernate,
where
QtyPurchase is a int and UnitPurchasePrice is a decimal property.
I have the following HQL query where I want to get the total purchase amount for a given day:
select sum(detail.QtyPurchased * detail.UnitPurchasePrice)
from Domain.Entities.ProductPurchase AS detail
where datediff("day", detail.PurchaseDate, :trading_date) = 0
Now for what ever reason, the query.UniqueResult always returns an Int64 integer, truncating the decimals, whereas the SQL generated obviously returns the correct number complete with decimals. Can someone shed somelight on how to get this to return decimal?
I have noticed that if I use SQL (i.e. CreateSqlQuery), I can get the decimal back. Is this a bug with Nhibernate?
Thanks heaps
Steven Kuo
Inverting the order of the factors did it for me: (price * qty) instead of (qty * price).
I guess it must be checking just the first parameter type, please fill a JIRA issue if this workaround worked for you.