Improve unit of measurement conversion in the frontend

[TCWORLD]

Proposed change

Note: I'm very much open to discussion as to whether this feature is wanted and also whether this is the right way to go about it. Figured I'd have a go to see what was possible and prompt ideas.

---

Basically, there is currently no complete front-end unit conversion utilities, which means rendering state values is very much fixed to whatever the entity unit was set to. Take for example the statistics graph - that displays statistics data but also the current state value of the entities in question. If however the state values are not all the same unit (particularly kW/W or say *F/*C), while the statistics data provided by the core is converted into a single compatible unit, the state values are not and as such cannot easily be plotted. It would be nice to have a way to convert the values.

In the energy cards there is also a case where state values are converted to form totals. In that case a set of lookup tables were previously added which were specific to that one use case and unit set, and thus not more generally useable.

Furthermore, there are a lot of hard-coded unit strings about in various frontend files, but no nice enumerations or constants for those unit strings. Unlike in core where there is a full set of constants for each of the possible units.

I've thought about some ways this could be rectified:

1. Do the conversion using web api callbacks to the core
   • Pro: No need to implement any conversion in the front end
   • Con: Very slow - don't want to be spamming core with lots of requests when there are frequent state value updates.
2. Have a lookup of conversion factors from core, where needed lookup and cache the value, then used the cached factor for handling state updates.
   • Pro: no need for full unit conversion handling in the front end. Simple query not used frequently.
   • Con: Only works for simple unit conversions. Temperature for example requires (x - 32) / 1.8 to convert *F to *C, so couldn't be returned as a simple scale factor. Would need custom handling.
3. Have lookup of conversion operations from core. Same as (2), but instead of returning a scale factor, return a set of operations that need to be performed to do the conversion.
   • Pro: no need for full unit conversion handling in the front end. Simple query not used frequently.
   • Con: Need to implement essentially a math parser to turn the list of operations into an executable function
     • I did have a go at this method, and got as far as the core side of things, but frontend is trickier with the math parsing.
4. Implement a full unit conversion capability in the front end
   • Pro: no need for extra web requests to core, as all conversion is done in the front end.
   • Pro: nice set of predefined units in the front end, and simple API for converting where needed.
   • Con: requires more code in the front end. Also requires keeping the frontend unit conversion behaviour identical to core.
     • This is what I've proposed in this PR.

---

So what I've proposed here, and for initial review/discussion the three files to focus on are:

1. Add a nice definitive set of unit constants. These can then be used as and when required.

   • This is provided as a set of nicely grouped enums in common/unit-conversion/const.ts
   • The file is basically the same unit enums from const.py in core, converted to typescript.
   • I've intentionally kept the names and groupings as similar as possible to the file in core so that keeping the two in sync if changes are made in the future should be possible
     • UnitOfConcentration is the only exception here, but I'd want to push that change back to the core repo if this goes ahead.

2. Add a set of converter classes to allow converting values between groups of compatible units.

   • This is provided by common/unit-conversion/unit-conversion.ts
   • This file is based heavily on util/unit_conversion.py from core but as typescript.
   • Again I've tried to keep the namings and groupings the same as the core file to help maintenance.

3. A helper function for identifying and instantiating the correct converter class for a given unit and/or unit class

   • This is in common/unit-conversion/unit-conversion.ts.
   • Basically boils down to simply calling getUnitConverter(unit) or getUnitConverter(unit, unitClass).
   • With a converter object, conversion can be attempted by checking isValidUnit() and then calling convert() if valid.

---

For now I've only test updated the energy and statistics-chart routines to use units/converters as those are the main target of this PR.

There are a large number of other place that use hard-coded units which could benefit from at the very least the unit enums proposed here. I updated those to, but for now I've left them on a different branch to reduce the size of this PR.

Screenshots

Nice example of use case - rendering the statistics graph "now" point. This requires unit conversion if it the entities had different units (e.g. kW and W).

Type of change

• Dependency upgrade
• Bugfix (non-breaking change which fixes an issue)
• New feature (thank you!)
• Breaking change (fix/feature causing existing functionality to break)
• Code quality improvements to existing code or addition of tests

Additional information

• This PR fixes or closes issue: fixes #
• This PR is related to issue or discussion: Statistics graph incorrectly has a zero point for each graph's last point #29993
• Link to documentation pull request:
• Link to developer documentation pull request:
• Link to backend pull request: Harmonise unit conversion in core and frontend core#166617

Checklist

• I understand the code I am submitting and can explain how it works.
• The code change is tested and works locally.
• There is no commented out code in this PR.
• I have followed the development checklist
• I have followed the perfect PR recommendations
• Any generated code has been carefully reviewed for correctness and compliance with project standards.

If user exposed functionality or configuration variables are added/changed:

• Documentation added/updated for www.home-assistant.io

To help with the load of incoming pull requests:

• I have reviewed two other open pull requests in this repository.

[MindFreeze]

I am generally in favor of this because it is all eternal constants. The ratios will never change so I don't see a point in calling core every time. We could make this a dynamically loaded module so it doesn't bloat the main bundle.
Will discuss with the frontend team.

[karwosts]

I guess the only point of calling to core is that they are frequently updating with new supported units. If we start to rely on unit conversions we will likely repeatedly find ourselves missing new units as they are added to core only.

Just based on looking at core it seems like there's maybe only two unit conversions currently that use non-linear scaling function:

• Temperature
• Beaufort

[karwosts]

This may also be somewhat useful/related for the requested "UoM Selector"

#30278

[MindFreeze]

We can extract the conversion factors to a shared JSON file. Something like

{
  "temperature": {
    "base": "K",
    "units": {
      "°C": {"scale": 1,   "offset": -273.15},
      "°F": {"scale": 1.8, "offset": -459.67},
      "K":  {"scale": 1,   "offset": 0}
    }
  }
}

[karwosts]

I guess Beaufort uses an exponential function as well, so that may be complex to extract to json, but I'm sure it's something that can be worked around for a small handful of cases.

[TCWORLD]

This is kind of like option 3 but using a shared file rather than API query (which makes more sense). When I was looking at retreiving the operations from the core, there are basically four operations needed:

• Add (+,-)
• Multiply (*,/)
• Power (^)
• Round

All of the unit conversions thus far can be acheived with those four, including Beaufort and the unit inverses (which can be acheived by raising to power of -1).

(For reference: TCWORLD/core@6615c5a)

I suppose the question is whether there is a clean way to import a operations file like that and convert it into executable code. I'm sure it's doable, but I stopped short of that in my API query attempt.

[MindFreeze]

The frontend team consensus is that a shared JSON file with core would be best. Now we have to convince the core team but @frenck seems to be on board.

Given that there are various operations and the order might matter, I'll update my example to

{
  "temperature": {
    "base": "K",
    "units": {
      "°C": [{"scale": 1,   "offset": -273.15}],
      "°F": [{"scale": 1.8, "offset": -459.67}, {"round": 0}]
    }
  }
}

[TCWORLD]

I'm happy to have a go making frontend/core code for parsing unit conversions from JSON if there is a consensus from both sides.

Would we still want the proposed enumerations file, or generate essentially constant arrays of unit strings from the JSON file directly? The former would be easier to strictly type functions using specific sets of units.

---

For the JSON example, I guess it might be easier to parse if we have an array of operations, but keep each operation as a single key-value pair (can also drop redundant ones). That way the parser could be a simple map of operation to function handle, calling each in turn. So:

{
  "temperature": {
    "base": "K",
    "units": {
      "°C": [{"offset": -273.15}],
      "°F": [{"scale": 1.8}, {"offset": -459.67}]
    }
  }
}

Converting from-to different units would be performing the "from" operations in reverse (with offsets negated, and the factors for scale/powers inverted), followed by the "to" operations. From the above, doing *F to *C would be performed as operations:

[{"offset": 459.67},{"scale": 1/1.8},{"offset": -273.15}]

[TCWORLD]

As an initial step, the following is all of the current units of measurement for each unit class, along with the conversion factors to the base unit for each class.

It's not formatted yet in the operations we'd want to perform - literally nothing more than a JSON serialisation of the unit converter array from - but it gives rough idea of the fixed constants for conversion needed for most of the units, along with the special cases that have converter functions.

unit-factors.json

[TCWORLD]

Might as well go the final step in creating the JSON file. This is the full set of unit conversions and operations in the JSON format described above:

unit-factors.json

I decided to leave any conversion which is just a simple scale factor as a bare number. It should be easy for an interpreter to select between simple scale factor (values read as a number), and set of operations (values read as tuple array), and perform the operation accordingly.

I'll look at writing an interpreter in due course. For now I'm going to mark this PR as a draft again.

[MindFreeze]

We probably have to create an Architecture discussion for this in order to get it in core

[TCWORLD]

I've decided to have a go and create the frontend implementation for the JSON approach anyway.

Even if not pushed back to core, it seems to be a much cleaner approach here. One class instead of 20+ and no need for special override functions and inverse checks.

Will probably need some stress testing, and documentation adding, but initial testing seems to work for both temperature and power cases, for both to and from units (F->C and C->F).

[TCWORLD]

I've been looking over the core code to see how possible it is to convert over to use the same JSON file, and its... interesting.

Core seems to rely heavily on their being default instances of unit converters to provide constants in various places, so trying to load that dynamically from JSON would be rather messy. Instead I'm going to try adapting core to use the same operations as here, and then have a script to export those converters to JSON for front-end.

That way if any changes are made to core in the future to add more units, it's a simple case of exporting the JSON from core to update the frontend to match. It might even be possible then to have frontend JSON file be entirely auto-generated by core at run time, but we'll see how that goes.

[TCWORLD]

Sit-rep:

• Core code PR has been created to make the necessary adjustments for conversion using the new operations approach. The Beaufort and unit inverses handling was tweaked slightly in both ends to make sure that the behaviour is identical to the original core code (all the various test vectors there now pass).
• A python script that very simply exports all the core converter objects to a JSON file has been added. This JSON file is basically in the format described above with a slight tweak to how the operations are listed (effectively a tuple rather than dictionary). The file now looks as below. This is actually easier to parse for both on import and export.

  "temperature": {
    "base": "°C",
    "inverse": [],
    "units": {
      "°C": 1.0,
      "°F": [
        ["scale", 1.8],
        ["offset", 32.0]
      ],
      "K": [["offset", 273.15]]
    }
  },
  

• Rudimentary tests continue to work nicely. For example in the graph below one entity is in *C, the other *F. The chart points are converted by core, while the now value is converted by frontend, both produce same results 👍 :