Ranter
Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Comments
-
rithvikp2856yI don't think the math is the problem, I found a similar issue in Python too (something like 50/2 being calculated to 24.9999999999999999). I think it has to do with how floats are stored in memory. Like, 0.1 is not exactly stored as 0.1 and things like that. You can play around with it here: https://h-schmidt.net/FloatConverte...
-
inaba45896yNo. This is an issue with floating point numbers in computers which are difficult since it's all 0s and 1s. See:
https://devrant.com/rants/1392187/...
If you want the most precise calculation then use the base unit. (Or since this is engineering, just use .toFixed() :v) -
byogdc7246yThis is most definitely a float issue. I think your code is treating the float as a double, therefore giving you a double precision result
-
7Raiden8756yYou should give a read to:
What Every Programmer Should Know About Floating Point Arithmetic -
inaba45896y@byogdc This is Javascript though and in Javascript the only kinds of numbers there are are 64 bit IEEE754 floating point numbers
-
You know how in decimal you have infinite periods of numbers for certain numbers? 1/3 becomes 0.333...
The same happens in binary (albeit for different numbers). You don't have infinite bits, so only an approximation gets stored in memory in any language. All that you can control is the level of precision. -
By the way, you get the same effect when dividing by 1000.0 instead of 1000. The '.0' makes it a floating point type.
-
Lisk726yYou actually meant MWh ("Mega") instead of mWh ("Milli"), right? ;)
1000 kWh = 1 MWh, and
1000000 mWh = 1000 Wh = 1 kWh -
@Lisk ia right
I was just about to say it myself.
1Mega -> 10e6 ->1000kilo [M]
1kilo -> 10e3 -> 1000 [k]
1decy -> 10e-1 -> 0.1 [d]
1centy -> 10e-2 -> 0.01 [c]
1milli -> 10e-3 -> 0.001 [m]
So 1kWh -> 1000000mWh -
stacked26716yIn binary you can't represent 0.1 exactly, just like in decimal you can't represent 1/3 or 1/7 exactly.
If you need decimal precision, use decimal arithmetic.
If you need arbitrary precision, use fractions (pairs of numerator, denomination integers).
I have been debugging for like hours trying to figure out the cause an unknown bug spoiling my UI by making my elements overlap.
I'm working on a Unit Converter that takes kWh and then converts to mWh. (Logical Conversion: 1000 kWh = 1 mWh).
Just an easy shit i thought, using Javascript I just passed the dynamically generated kWh value to a function that takes maximum of 6 chars and multiply it by 0.001 to get the required result but this was where my problem started. All values came out as expected until my App hits a particular value (8575) and outputs a very long set of String (8.575000000000001), i couldn't figure the cause of this until i checked my console log and found the culprit value, and then i change the calculation logic from multiply by 0.001 to divide by 1000 and it came out as expected (8.575)
My question is that;
Is my math logically wrong or is this another Javascript Calculation setback?
question
javascript
calculation logic