We are running a survey to help us improve the experience for all of our members. If you see the survey appear, please take the time to tell us about your experience if you can.
I am using the double data type to store a calibration constant. I tried also with the float data type. To form the value, I read from memory decimal digits multiplied by its corresponding decimal:
to form 23.86108 I read from memory 2, 3, 8, 6, 1, 0 and 8
2 x 10 + 3 x 1 + 8 x 0.1 + 6 x 0.01 + 1 x 0.001 + 0 x 0.0001 + 8 x 0.00001
This works fine until I add a decimal digit more. It seems to have no influence.
23.86108 + 5 x 0.000001 = 23.86108 instead of 23.861085
I search on "Options for target" to see if there is a limitation, but does not find any. (" bits to round for float compare" has no effect ). Can anyone help me with this problem?. Perhaps it is something basic, but I can not solve. Thank you.
As indicated you need to look in to how floating point numbers work. It is not like a calculator. If you need something more precise try long integers. that gives you up to 4294967296.