General Algorithm for Primitive Calculation?


RE: FYI: "Case study" of some numerical challenges (and the solution [not])
#5
Quote:I got a 0.5036499999999999.

LDPE says it's 0.5036.
PrimGen2 says it's 0.5037.

Arrgh!
I feel your pain... At least I learnt things about rounding Wink
Looks like we were lucky when we cloned the original primgen, once we figured out things, file comparison sturned out OK. Or maybe we didin't stumble on a tricky case? Now I am not sure that a 1/10000 error from time to time would be a real issue...
This last problem is related to this https://en.wikipedia.org/wiki/Rounding#T...7s_dilemma ?
Reply
« Next Oldest | Next Newest »



Messages In This Thread
RE: FYI: "Case study" of some numerical challenges (and the solution [not]) - by Philippe Hurbain - 2016-08-12, 5:46

Forum Jump:


Users browsing this thread: 1 Guest(s)