Has anyone had any problems substracting floats? I have the following code: (C166 V4.23 compiler) float dummy, test1, test2; test1 = 1.0e10; test2 = 1.1e10; dummy = test2 - test1; for some reason dummy comes out as 1.000001e9 I've tried declaring them as doubles and get the same result. DS
The data size of your calculator's processor is irrelevant, the original electronic calculators used 4 bit processors. Most calculators use an internal BCD notation (Binary coded Decimal) to avoid rounding errors caused by conversions from decimal to binary. The tradeoff is speed.
"Most calculators use an internal BCD notation (Binary coded Decimal) to avoid rounding errors" There will always be rounding errors whatever representation you use! Some numbers just can't be written down; eg 1/3. On my casio fx-451, if I reciprocate 9.000000001 and then reciprocate the result I get 9.000000002. You can have hours of fun playing with a calculator to find its little errors like this. Or maybe just get out a little more often... ;-)