Skip to content

Umlauts/non-ASCII characters corrupted in app log output, independent of container locale/encoding (regression of #990) #2668

Description

@Powerslider26

What happened?

Title: Umlauts/non-ASCII characters corrupted in app log output, independent of container locale/encoding (regression of #990)

Body:

Related to the old, closed issue #990 ("umlauts getting lost when fetching data from Home Assistant"), which never received a documented fix. I'm seeing the same symptom on a current version and was able to narrow it down further.

Environment:
AppDaemon 4.5.13, Home Assistant OS 18.2 (aarch64/green), installed as the official Home Assistant add-on
Container locale confirmed: LANG=C.UTF-8
python3 -c "import sys; print(sys.stdout.encoding)" inside the container returns utf-8

Symptom:
A German string containing "Ö" (e.g. "AT - Österreich"), read from a Home Assistant input_select entity via self.get_state(), is logged correctly as far as its in-memory representation goes, but appears corrupted in the actual log output.

Diagnosis steps taken:

  1. Confirmed via [hex(ord(c)) for c in text] that the Python string itself contains the correct codepoint 0xd6 (U+00D6, "Ö") immediately before logging it.
  2. Confirmed via docker logs | grep ... | xxd that the raw bytes captured on the container's stdout are ef bf bd ef bf bd — i.e. the UTF-8 encoding of two separate U+FFFD replacement characters, exactly matching what you'd get from decoding the correct 2-byte UTF-8 sequence (c3 96) one byte at a time with an ASCII-only decoder using errors='replace'.
  3. Ruled out: container locale (C.UTF-8), sys.stdout.encoding (utf-8), and the data source (HA websocket state; the same string appears correctly in the corresponding HA entity attribute in Developer Tools → States).

Conclusion: The corruption happens strictly inside AppDaemon's own logging pipeline, between the correctly-encoded Python string and the bytes written to stdout — not in locale handling, not in sys.stdout, and not in the source data.

Steps to reproduce:

python
def initialize(self):
self.log(f"Test: {'Österreich'!r}")

Expected: Test: 'Österreich'
Actual (observed via docker logs): Test: 'sterreich' (with the "Ö" replaced by two U+FFFD-derived bytes)

Happy to provide the full apps.yaml / minimal repro app if useful.

Version

0.18.5

Installation type

Home Assistant add-on

Relevant log output

Relevant code in the app or config file that caused the issue

def initialize(self):
self.log(f"Test: {'Österreich'!r}")


### Anything else?

Metadata

Metadata

Assignees

No one assigned

    Labels

    issueSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions