19 May
2021
19 May
'21
1:51 p.m.
On Wed, 2021-05-19 at 13:38 +0200, rolfm@dds.nl wrote:
the value that should be 0.1250 is actually 0.1249999
and 0.75 is actually 0.7499994.
The intricacies of 32bit floating point numbers have been discussed up and down on this list.
tldr; many decimal numbers do not have an exact floating point representation. Incrementing with such numbers increases the error on each iteration
Suggestion: Count with integer numbers and scale afterwards.
Roman