Coalesce returning wrong value after a function call followed by multiplication - sql

I have a report that presents information and I'm getting inconsistent information based on what appears to be some issue with a SQL view or possibly a SQL Function nested within the view. I've tried finding a way to debug the SQL View, however, it looks like SSMS only will debug Stored Procedures, so I'm not really sure how to step through and see what is happening. It really has me stumped and I can't help but wonder if it isn't a rounding issue.
GetItemAverageCost RETURNS DECIMAL(12,2) and the DataType in sitli.QuantityIssuedAtStockUOM is System.Int64 / bigint (sidenote: I'm confused about why LINQPad shows 2 data types for that column. In the tree on the left, after expanding the sitli table and hovering over the QuantityIssuedAtStockUOM the balloon BigInt NOT NULL pops up, but when I Take(100) and hover over the column in the result set it says System.Int64). Anyroad, here is the COALESCE function.
COALESCE((dbo.GetItemAverageCost(ItemModel.IDItemModel)*sitli.QuantityIssuedAtStockUOM) / ISNULL(NULLIF(ItemModel.UOMFactor, 0),1),0) -- 259.73
--ROUND(COALESCE((dbo.GetItemAverageCost(ItemModel.IDItemModel)*sitli.QuantityIssuedAtStockUOM) / ISNULL(NULLIF(ItemModel.UOMFactor, 0),1),0),2) -- 259.73
--COALESCE(ROUND((dbo.GetItemAverageCost(ItemModel.IDItemModel)*sitli.QuantityIssuedAtStockUOM) / ISNULL(NULLIF(ItemModel.UOMFactor,2), 0),1),0) -- 259.70
--COALESCE((ROUND(dbo.GetItemAverageCost(ItemModel.IDItemModel),2)*sitli.QuantityIssuedAtStockUOM) / ISNULL(NULLIF(ItemModel.UOMFactor, 0),1),0) -- 259.73
original / wrong coalesce:
COALESCE(dbo.GetItemAverageCost(ItemModel.IDItemModel)*sitli.QuantityIssuedAtStockUOM,0)
I'm not sure what else to include, but I haven't found many resources online that offer insight into this kind of a situation. Many thanks in advance for your time.
EDIT: GetItemAverageCost:
ALTER FUNCTION GetItemAverageCost
(
#IDItemModel varchar(8000)
)
RETURNS DECIMAL(16,4)
--RETURNS DECIMAL(12,2)
AS
BEGIN
RETURN
(
SELECT
COALESCE(AVG(poli.UnitPrice),0) as AvgCost
-- COALESCE(ROUND(AVG(poli.UnitPrice),0),2) as AvgCost 260.00
FROM ItemModel im
LEFT JOIN VendorItem vi
ON im.IDItemModel = vi.IDItemModel
JOIN POLineItem poli
ON vi.IDVendorItem = poli.IDVendorItem
WHERE
im.IDItemModel = #IDItemModel
GROUP BY
im.IDItemModel,
im.ItemNumber
)
END

To fix; have your function return 16,4 instead of 12,2 and then ROUND two two decimals after multiplying by the quantity.
"When a given report is run, there are no errors thrown. But the calculations are off. For example a part number 12 shows a quantity of 24 were issued at a cost of $259.73. However, each part costs $10.82 so the calculation should be $259.68. I'm not sure where the difference of 5 cents is coming from. The $259.73 is the result of the COALESCE function above. Hopefully this makes sense"
Run the SQL only for part 12 independent of the function and you'll see the average is 10.822083333333333333333333333333 (10.82 5/24ths)
24*unitprice = $259.73
unitprice = 259.73/24
unit price = $10.82 5/24.
You'll see the variance is $.05
10.82 5/24ths. *24 = 259.73
10.82 * 24 = 259.68
That difference of 5cents doesn't go evenly into the remaining 24. thus the rounding error when using your function.
When you think of going to the store and buying something it's always at amounts to the whole penny. When you go to the gas station they charge to the nearest .00001 cents. (or in your case 4 decimals)
The rounding when using fractions of pennies isn't done until multiplied by the quantity or when actual cash needs to change hands. If done too early you get rounding errors you are seeing.
Thus you eliminate over/under charging rounding errors and at most you'll charge a fraction of a penny less or more than you should.

Okay, so many thanks to all who helped along the way. There were a couple of issues preventing me from getting the correct answer. For one thing, I was working with the incorrect expression for much of the time. Secondly, after I figured out which expression to use, it was a matter of placing the ROUND function in the correct place.
So, the expression I should have been using to get my average cost is:
COALESCE(dbo.GetItemAverageCost(Item.IDItemModel) / ISNULL(NULLIF(UOMFactor, 0),1),0)
When I moved this into the WorkOrderItemInstructionPartCosts View, my report was then producing $10.82. Then I added *sitli.QuantityIssuedAtStockUOM to the line and was getting $259.73. Then I applied the ROUND function to the COALESCE function and voila! the correct value ($259.68) is being produced.
The final line looks like this:
ROUND(COALESCE(dbo.GetItemAverageCost(ItemModel.IDItemModel) / ISNULL(NULLIF(UOMFactor, 0),1),0),2)*sitli.QuantityIssuedAtStockUOM
Once again, thank you to all who helped me in the effort to resolve this and sorry for not having accurate information to begin with.
Best,
Jonathan

Related

Inputting float variable adds nines and changes the decimal

I'm creating a basic program to break a user-inputted amount of money down into the smallest number of bills and coins, and I'm running into an issue with float variables. Both when using scanf and when giving the variable an amount while defining it, I've run into an issue where the float is randomly changed. For example, I'll define amount as 200.53, then when I reprint the variable to check, it prints out 200.529999.
I'm very new to C, so there may be a simple solution, but I've tried, rounding, flooring, and other solutions I've found online to no avail. Even after I round amount or floor it using amount = round(amount * 100) / 100, it still prints 200.529999. Thanks!
Here is an image of my code and the output
Thanks!

Function iMA is returning different return value from expected (MQL5)

I'm using MQL5 (my first code).
I want to use a script that uses MA, but first, I wanted to confirm the value to verify I'm doing correctly. Using a very basic code into script:
double x=0;
x = iMA(Symbol(),Period(),100,0,MODE_SMA,PRICE_CLOSE);
Alert("The actual MA from last 100 points of EURUSD actually is: " + x;
The expected value is near the actual price... 1.23456, but this function is returning 10.00000 or 11.0000.
I believe I'm missing something, and https://www.mql5.com/es/docs/indicators/ima helplink is not quite clear enough.
I already saw another similar function: MA[0] which seems to bring the moving average from specific candle, but, I don't know how to manage the Period range (100) or if is related to Close/Open variables on it. I didn't find any specific helplink to review.
Any ideas are very appreciated!!!
x should be int, it is a handler of the MA. So each indicator when created in MT5 receives its handler, and you can use it later to get what you need. If you need several MA's - create several handlers and give each of them different names (x1, x2 or add some sense). Expert advisors in the default build of MT5 are good examples on what to do.
The iMA function Returns the handle of a specified technical indicator, not the "moving average" value.
For example, to get the value of the Moving average you can use this (in MQ4):
EMA34Handler = iMA(NULL,0,34,0,MODE_EMA,PRICE_CLOSE);
EMA34Value = CopyBuffer(EMA34Handler, 0,0);

How to handle precision problems of floating point numbers?

I am using Firebird 3.0.4 (both in Windows and Linux) and I have the following procedure that clearly demonstrates my problem with floating point numbers, and that also demonstrates a possible workaround:
create or alter procedure test_float returns (res double precision,
res1 double precision,
res2 double precision)
as
declare variable z1 double precision;
declare variable z2 double precision;
declare variable z3 double precision;
begin
z1=15;
z2=1.1;
z3=0.49;
res=z1*z2*z3; /* one expects res to be 8.085, but internally, inside the procedure
it is represented as 8.084999999999.
The procedure-internal representation is repaired when then
res is sent to the output of the procedure, but the procedure-internal
representation (which is worng) impacts the further calculations */
res1=round(res, 2);
res2=round(round(res, 8), 2);
suspend;
end
On can see the result of the procedure with:
select proc.res, proc.res1, proc.res2
from test_float proc
The result is
RES RES1 RES2
8,085 8,08 8,09
But one can expect that RES2 should be 8.09.
One can clearly see that the internal representation of the res contains 8.0849999 (e.g. one can assign res to the exception message and then raise this exception), it is repaired during output but it leads to the failed calculations when such variable is used in the further calculations.
RES2 demonstrates the repair: I can always apply ROUND(..., 8) to repair the internal representation. I am ready to go with this solution, but my question is - is it acceptable workaround (when the outer ROUND is with strictly less than 5 decimal places) or is there better workaround.
All my tests pass with this workaround, but the feeling is bad.
Of course, I know the minimum that every programmer should know about floats (there is article about that) and I know that one should not use double for business calculations.
This is an inherent problem with calculating with floating point numbers, and is not specific to Firebird. The problem is that the calculation of 15 * 1.1 * 0.49 using double precision numbers is not exactly 8.085. In fact, if you would do 8.085 - RES, you'd get a value that is (approximately) 1.776356839400251e-015 (although likely your client will just present it as 0.00000000).
You would get similar results in different languages. For example, in Java
DecimalFormat df = new DecimalFormat("#.00");
df.format(15 * 1.1 * 0.49);
will also produce 8.08 for exactly the same reason.
Also, if you would change the order of operations, you would get a different result. For example using 15 * 0.49 * 1.1 would produce 8.085 and round to 8.09, so the actual results would match your expectations.
Given round itself also returns a double precision, this isn't really a good way to handle this in your SQL code, because the rounded value with a higher number of decimals might still yield a value slightly less than what you'd expect because of how floating point numbers work, so the double round may still fail for some numbers even if the presentation in your client 'looks' correct.
If you purely want this for presentation purposes, it might be better to do this in your frontend, but alternatively you could try tricks like adding a small value and casting to decimal, for example something like:
cast(RES + 1e-10 as decimal(18,2))
However this still has rounding issues, because it is impossible to distinguish between values that genuinely are 8.08499999999 (and should be rounded down to 8.08), and values where the result of calculation just happens to be 8.08499999999 in floating point, while it would be 8.085 in exact numerics (and therefor need to be rounded up to 8.09).
In a similar vein, you could try to use double casting to decimal (eg cast(cast(res as decimal(18,3)) as decimal(18,2))), or casting the decimal and then rounding (eg round(cast(res as decimal(18,3)), 2). This would be a bit more consistent than double rounding because the first cast will convert to exact numerics, but again this has similar downside as mentioned above.
Although you don't want to hear this answer, if you want exact numeric semantics, you shouldn't be using floating point types.

Calculating logarithms in vb.net - conflicting results

I am doing some calculations in vb.net, this is my equation:
rms = (20 * (Math.Log(rms / 0.7746))) 'also tried (Math.Log10(rms / 0.7746))
I have tried various different methods of writing this, including separating out the calculations into various steps. However the final result is quite far out.
I have tried declaring my variable 'rms' as a decimal and a double. It does contain decimal places.
In Excel, I have tried the same calculation using this formula:
=20*(LOG(C2/0.7746)) ' where C2 is the RMS value
And the results are consistent with a website I used to check, as well as my pocket calculator.
I have also tried rounding the number to 3 decimal places:
rms = Math.Round(rms, 3)
This too has a minimal effect on the final result.
I can only assume it's the 'operator precedence' in VB but I'm struggling to work this one out.
Any help greatly appreciated as always, thanks.
After a marathon debugging session I found an error in my code.
I have a routine that uses the voltage at a given frequency to "normalise" all the plots I do to 0dBu.
My normalisation routine was broken. Badly.
And finally - to get the correct output from the log maths, I had to change the order in which the calculation was performed.
It was originally rms = (Math.Log10(rms / 0.7746) * 20)
In trying to find the issue, I changed it to rms = (20 * (Math.Log(rms / 0.7746)))
Which yields a different (and incorrect) result.
In any case - it's fixed now.
Thanks to all who responded.

Process to pass from problem to code. How did you learn?

I'm teaching/helping a student to program.
I remember the following process always helped me when I started; It looks pretty intuitive and I wonder if someone else have had a similar approach.
Read the problem and understand it ( of course ) .
Identify possible "functions" and variables.
Write how would I do it step by step ( algorithm )
Translate it into code, if there is something you cannot do, create a function that does it for you and keep moving.
With the time and practice I seem to have forgotten how hard it was to pass from problem description to a coding solution, but, by applying this method I managed to learn how to program.
So for a project description like:
A system has to calculate the price of an Item based on the following rules ( a description of the rules... client, discounts, availability etc.. etc.etc. )
I first step is to understand what the problem is.
Then identify the item, the rules the variables etc.
pseudo code something like:
function getPrice( itemPrice, quantity , clientAge, hourOfDay ) : int
if( hourOfDay > 18 ) then
discount = 5%
if( quantity > 10 ) then
discount = 5%
if( clientAge > 60 or < 18 ) then
discount = 5%
return item_price - discounts...
end
And then pass it to the programming language..
public class Problem1{
public int getPrice( int itemPrice, int quantity,hourOdDay ) {
int discount = 0;
if( hourOfDay > 10 ) {
// uh uh.. U don't know how to calculate percentage...
// create a function and move on.
discount += percentOf( 5, itemPriece );
.
.
.
you get the idea..
}
}
public int percentOf( int percent, int i ) {
// ....
}
}
Did you went on a similar approach?.. Did some one teach you a similar approach or did you discovered your self ( as I did :( )
I go via the test-driven approach.
1. I write down (on paper or plain text editor) a list of tests or specification that would satisfy the needs of the problem.
- simple calculations (no discounts and concessions) with:
- single item
- two items
- maximum number of items that doesn't have a discount
- calculate for discounts based on number of items
- buying 10 items gives you a 5% discount
- buying 15 items gives you a 7% discount
- etc.
- calculate based on hourly rates
- calculate morning rates
- calculate afternoon rates
- calculate evening rates
- calculate midnight rates
- calculate based on buyer's age
- children
- adults
- seniors
- calculate based on combinations
- buying 10 items in the afternoon
2. Look for the items that I think would be the easiest to implement and write a test for it. E.g single items looks easy
The sample using Nunit and C#.
[Test] public void SingleItems()
{
Assert.AreEqual(5, GetPrice(5, 1));
}
Implement that using:
public decimal GetPrice(decimal amount, int quantity)
{
return amount * quantity; // easy!
}
Then move on to the two items.
[Test]
public void TwoItemsItems()
{
Assert.AreEqual(10, GetPrice(5, 2));
}
The implementation still passes the test so move on to the next test.
3. Be always on the lookout for duplication and remove it. You are done when all the tests pass and you can no longer think of any test.
This doesn't guarantee that you will create the most efficient algorithm, but as long as you know what to test for and it all passes, it will guarantee that you are getting the right answers.
the old-school OO way:
write down a description of the problem and its solution
circle the nouns, these are candidate objects
draw boxes around the verbs, these are candidate messages
group the verbs with the nouns that would 'do' the action; list any other nouns that would be required to help
see if you can restate the solution using the form noun.verb(other nouns)
code it
[this method preceeds CRC cards, but its been so long (over 20 years) that I don't remember where i learned it]
when learning programming I don't think TDD is helpful. TDD is good later on when you have some concept of what programming is about, but for starters, having an environment where you write code and see the results in the quickest possible turn around time is the most important thing.
I'd go from problem statement to code instantly. Hack it around. Help the student see different ways of composing software / structuring algorithms. Teach the student to change their minds and rework the code. Try and teach a little bit about code aesthetics.
Once they can hack around code.... then introduce the idea of formal restructuring in terms of refactoring. Then introduce the idea of TDD as a way to make the process a bit more robust. But only once they are feeling comfortable in manipulating code to do what they want. Being able to specify tests is then somewhat easier at that stage. The reason is that TDD is about Design. When learning you don't really care so much about design but about what you can do, what toys do you have to play with, how do they work, how do you combine them together. Once you have a sense of that, then you want to think about design and thats when TDD really kicks in.
From there I'd start introducing micro patterns leading into design patterns
I did something similar.
Figure out the rules/logic.
Figure out the math.
Then try and code it.
After doing that for a couple of months it just gets internalized. You don't realize your doing it until you come up against a complex problem that requires you to break it down.
I start at the top and work my way down. Basically, I'll start by writing a high level procedure, sketch out the details inside of it, and then start filling in the details.
Say I had this problem (yoinked from project euler)
The sum of the squares of the first
ten natural numbers is, 1^2 + 2^2 +
... + 10^2 = 385
The square of the sum of the first ten
natural numbers is, (1 + 2 + ... +
10)^2 = 55^2 = 3025
Hence the difference between the sum
of the squares of the first ten
natural numbers and the square of the
sum is 3025 385 = 2640.
Find the difference between the sum of
the squares of the first one hundred
natural numbers and the square of the
sum.
So I start like this:
(display (- (sum-of-squares (list-to 10))
(square-of-sums (list-to 10))))
Now, in Scheme, there is no sum-of-squares, square-of-sums or list-to functions. So the next step would be to build each of those. In building each of those functions, I may find I need to abstract out more. I try to keep things simple so that each function only really does one thing. When I build some piece of functionality that is testable, I write a unit test for it. When I start noticing a logical grouping for some data, and the functions that act on them, I may push it into an object.
I've enjoyed TDD every since it was introduced to me. Helps me plan out my code, and it just puts me at ease having all my tests return with "success" every time I modify my code, letting me know I'm going home on time today!
Wishful thinking is probably the most important tool to solve complex problems. When in doubt, assume that a function exists to solve your problem (create a stub, at first). You'll come back to it later to expand it.
A good book for beginners looking for a process: Test Driven Development: By Example
My dad had a bunch of flow chart stencils that he used to make me use when he was first teaching me about programming. to this day I draw squares and diamonds to build out a logical process of how to analyze a problem.
I think there are about a dozen different heuristics I know of when it comes to programming and so I tend to go through the list at times with what I'm trying to do. At the start, it is important to know what is the desired end result and then try to work backwards to find it.
I remember an Algorithms class covering some of these ways like:
Reduce it to a known problem or trivial problem
Divide and conquer (MergeSort being a classic example here)
Use Data Structures that have the right functions (HeapSort being an example here)
Recursion (Knowing trivial solutions and being able to reduce to those)
Dynamic programming
Organizing a solution as well as testing it for odd situations, e.g. if someone thinks L should be a number, are what I'd usually use to test out the idea in pseudo code before writing it up.
Design patterns can be a handy set of tools to use for specific cases like where an Adapter is needed or organizing things into a state or strategy solution.
Yes.. well TDD did't existed ( or was not that popular ) when I began. Would be TDD the way to go to pass from problem description to code?... Is not that a little bit advanced? I mean, when a "future" developer hardly understand what a programming language is, wouldn't it be counterproductive?
What about hamcrest the make the transition from algorithm to code.
I think there's a better way to state your problem.
Instead of defining it as 'a system,' define what is expected in terms of user inputs and outputs.
"On a window, a user should select an item from a list, and a box should show him how much it costs."
Then, you can give him some of the factors determining the costs, including sample items and what their costs should end up being.
(this is also very much a TDD-like idea)
Keep in mind, if you get 5% off then another 5% off, you don't get 10% off. Rather, you pay 95% of 95%, which is 90.25%, or 9.75% off. So, you shouldn't add the percentage.