On Thu, Dec 8, 2011 at 9:13 AM, Joe Newlin jtnewlin@gmail.com wrote:
Can someone explain what's going on in the attached example? I'm getting 7.1 minus 7 equals .099. I need to split two-digit integers, so this result is giving me problems.
An interesting rounding example. If you resize the numbox you would see it's not 0.099 but 0.0999999. Six significant decimal digits, that is also the maximum precision which a numbox can show.
The fraction of 7.1 is 'not terminating' when converted to binary floating point format. Compare with 1/3 which is not terminating in decimal floating point format. If you type 7.1 in a numbox or message box, 7.0999999 is in fact closer to the 32 bit binary float which represents it. But it would be annoying when you type 7.1 and it changes to 7.0999999 automatically.
I once compiled Pd with the format specifiers set to 8 significant decimal digits, and quickly found why 6 decimal digits is the default instead. Some numbers which I normally use without special consideration were automatically converted to a representation with many more digits. Type 0.02 and you get 0.019999999. It's confusing and it makes object boxes and message boxes larger than you like. On the other hand, I found that 32 bit floats can indeed represent all the integers up till 2^24. So it is a pity we can not see or type these in boxes and save them with patches. Maybe it would be better to represent the numbers 100000 and higher with 8 decimal digits.
See this IEEE 754 Converter if you want to check the rounding of a single precision float:
http://www.h-schmidt.net/FloatApplet/IEEE754.html
Katja