Hi Peter,
On 07/01/2019 12:24, Peter P. wrote:
Hi,
when I send the numer 0.7 to both inlets of a [* ] object (yes, in correct order) the result is printed as 0.49 If I now subtract [- 0.49] the result is -2.98023e-08 While this is no major concern for me right now, I am still wondering which step in this simple calculation is triggering this. If I remember my CS lecture correctly it is the subtraction, right?
it's all the steps!
0.7 = 7/10 is not exactly representable in binary floating point, so you actually get a nearby fraction with a power of 2 as the denominator: 11744051/16777216 = 0.699999988079071044921875 0.49 = 49/100 is also not exactly representable in binary floating point, so you actually get: 2055209/4194304 = 0.4900000095367431640625 when you multiply two floating point numbers, the result is rounded to fit the available precision (for single precision float, that is 24bits): you get 16441671/33554432 = 0.4899999797344207763671875
subtracting the last two numbers (which is again rounded) reveals the behaviour of floating point that is hidden by Pd only printing 6 digits at most:
0.7*0.7-0.49 computed as float = -1/33554432 = -2.98023223876953125e-8
other environments print more digits, which shows the behaviour sooner:
ghci> (0.7 * 0.7::Float) 0.48999998