`--json` output contains unescaped characters in fields

[kdeldycke]

Configuration

mas ▁▁▁▁ 7.0.0
slice ▁▁ arm64
slices ▁ arm64
dist ▁▁▁ homebrew/core/mas
origin ▁ https://github.com/mas-cli/mas.git
rev ▁▁▁▁ 7c70ffdfd9f71a654300a78b3b627782e6abe1b4
swift ▁▁ 6.3 (swiftlang-6.3.0.123.5 clang-2100.0.123.102)
driver ▁ 1.148.6
store ▁▁ FR
region ▁ FR
macos ▁▁ 26.4.1 (25E253)
mac ▁▁▁▁ unknown
cpu ▁▁▁▁ Apple M1 Pro
arch ▁▁▁ arm64

Bug description

Every --json-emitting subcommand (search, and by extension list, outdated, lookup/info) can produce JSON in which a string field contains raw, unescaped characters like a literal \n inside description or releaseNotes.

The practical impact is that the output is not safely streamable as newline-delimited JSON (one record per line).

I discovered this while updating my Meta Package Manager project to rely on --json to parse machine readable output: kdeldycke/meta-package-manager@2e12d79

Loosely related to #626 as the latter mentions properly escaping invalid data.

Steps to reproduce

You can reproduce the parse failure with the most natural stream consumer with this one liner:

$  mas search excel --json | python3 -c 'import sys,json; [json.loads(l) for l in sys.stdin if l.strip()]'
...

Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import sys,json; [json.loads(l) for l in sys.stdin if l.strip()]
                      ~~~~~~~~~~^^^
  File "/Users/kde/Library/Application Support/uv/python/cpython-3.14.0+freethreaded-macos-aarch64-none/lib/python3.14t/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ~~~~~~~~~~~~~~~~~~~~~~~^^^
  File "/Users/kde/Library/Application Support/uv/python/cpython-3.14.0+freethreaded-macos-aarch64-none/lib/python3.14t/json/decoder.py", line 345, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kde/Library/Application Support/uv/python/cpython-3.14.0+freethreaded-macos-aarch64-none/lib/python3.14t/json/decoder.py", line 361, in raw_decode
    obj, end = self.scan_once(s, idx)
               ~~~~~~~~~~~~~~^^^^^^^^
json.decoder.JSONDecodeError: Invalid control character at: line 1 column 2516 (char 2515)

[rgoldberg]

I haven't reproduced any bug.

See the attached output.

What is wrong with it? The only escapes I see in it are \ followed by either n or ", both of which are valid & necessary.

If you get different output, please attach it to a reply.

If you're talking about the newlines between the JSON objects, those are intended. It's not a single JSON value, it's a stream of objects separated by newlines.

mas search excel --json.json

[kdeldycke]

Sorry, my reproduction steps weren't specific enough. The bug depends on what the store is returning you, which certainly depends on the locale and geo-fencing of your Apple account I guess.

So here is my output of:

$ mas search excel --json > mas-search-excel.json 

mas-search-excel.json

See the description field of record 1316315027, at line 32, where we have:

• code point U+2028 at column 1514
• code point U+2028 at column 1556
• code point U+0010 at column 2516

[kdeldycke]

The thing is according to https://datatracker.ietf.org/doc/html/rfc8259#section-7 , JSON must escape the control characters from the U+0000-U+001F range.

As for U+2028 / U+2029, it looks like they're indeed legal in JSON. But Python's str.splitlines() treat them as line terminators. And I suspect most other line-based stream consumers does to.

So I propose for mas to escape both classes of characters to keep the output safely streamable as NDJSON.

On my side I implemented a workaround in commit: kdeldycke/meta-package-manager@429a88c

[rgoldberg]

The JSON framework used by mas does not escape some characters that are not valid verbatim in JSON strings to their respective \u<hex> escape sequence.

I've reported a bug at rarestype/swift-json#130.

Blocked by rarestype/swift-json#130

[kdeldycke]

Thanks @rgoldberg for confirming the issue, and also thanks for reporting it upstream! :)