General Algorithm for Primitive Calculation?


RE: I had the idea of rounding twice...
#7
I'm not positive, but the "correct" solution might be your original round4f, with one minor change. Instead of constructing a BigDecimal with just the double value (for arbitrary precision), construct it with the double value and a MathContext.DECIMAL64 as a second argument. Based on the documentation, this should make BigDecimal use IEEE 754 rounding rules, which I would expect to match the C# rounding.

Side note: I'm pretty sure your solution can trigger wrong rounding in other cases. Say, for example, that your calculated value is 0.5036499999999911 instead of 0.5036499999999999. You're going to round up to 0.5037 in that case, while 0.5036 would be the correct result. Also, it seems that you could avoid the whole string thing by rounding the BigDecimal to 8 places, and then rounding that rounded BigDecimal to 4 places.
Reply
« Next Oldest | Next Newest »



Messages In This Thread
RE: I had the idea of rounding twice... - by Travis Cobbs - 2016-08-12, 19:48

Forum Jump:


Users browsing this thread: 2 Guest(s)