Preserve color codes from os.proc.spawn stdout

[peacefulotter]

Hi,

I would like to get the colors from a subprocess stdout (e.g. git status will show the "modified" lines in green). From what I have seen, there is no way to do that in os-lib, maybe even in Java in general.

I have tried the following:

// ReadBytes and ReadLines have been tested
def processStdout = os.ProcessOutput.ReadBytes((line, len) =>
  val buf = line.slice(0, len).map(_.toChar)
  println(s"'${buf.mkString("Array(", ", ", ")")}'")
)
val proc = os.proc(cmd).spawn(stdout = processStdout) 

In this example, the stdout is printed as an array of Char but it is missing the color info such as �[92m some text here �[0m

Also tried to print each byte using proc.stdout.data.readByte with no luck:

for (i <- 0 until 10000) {
  val s = proc.stdout.data.readByte()
  println((s.toInt, s.toChar))
}

From what I understand, I need to specify to the subprocess that it is a "pseudo terminal" according to this link

Any idea how to do such a thing?

Thanks!

[SethTisue]

Perhaps the subprocess is detecting that it isn't connected to the terminal, and is suppressing the color codes itself?

I don't know what cmd is but if it's e.g. ls you might need to e.g. ls --color=always

demo, not involving a JVM at all:

[peacefulotter]

It is indeed not connected to a terminal, I am writing a terminal emulator using JavaFX. Is there a way I could fake a terminal?
Sure --color=always is available for ls, but how about others like git?

I found this: https://github.com/JetBrains/pty4j which is apparently a way to fake a TTY which seems to be exactly what I need. I will investigate on my side, but if it turns out to work it would be nice to implement this out-of-the-box for os-lib!

[lefou]

If you emulate a terminal, you might want to set the TERM variable to something that's supposed to support colors, like xterm-color. Of course, you need to check which other implication that might have, e.g. what expectations do have commands you run to your emulated terminal then.

[peacefulotter]

Thank you, will try it out tonight

[peacefulotter]

Following up on this, I did the following:

os.proc(cmd :: params).call(
  cwd = wd,
  env = Map("TERM" -> "xterm-color"), // tried also just "xterm"
  stdout = processStdout
)

But I am still not getting the color codes...

And to make matters worse, the pty4j library can't be installed with SBT - at least it threw an error during installation.

I am on Windows 10 by the way, sorry for mentioning this so late.

[peacefulotter]

Any update on this?

[lefou]

Huh, what kind of update? I though, you are the one who wants to emulate a terminal? I guess you need to inspect at least one of the tools you want colored output of, and find out, on what criteria it decides to disable color. After knowning what's needed, we can think about how to provide it via os-lib.

[peacefulotter]

Ok, I think I might have not made myself clear. My "terminal emulator" that I am writing is simply a Scala Swing frame/window.
I use os-lib under the hood to spawn commands and display the output with Swing labels. I would like to color those labels based on the output I get from os.proc.spawn (think of the colors you get with git status for instance). This output is in a byte array format (or string) and does not contain color codes as I would have on my Windows terminal. Throughout this discussion, I think we pinpointed the issue: os.proc.spawn spawns a subprocess and runs the command in there. However, this subprocess is not "tagged" as being a terminal and thus does not provide the color codes. The fix we thought about was to manually set the "TERM" environment variable as "xterm" or "xterm-color" but this changed nothing on my side.
Does that make sense?

[SethTisue]

I don't think this is an issue with os-lib. I believe you would experience the exact same issue using, for example, java.lang.Process, which is the underlying API that os-lib wraps.

I don't fault you for hoping we might have further advice for you, but regardless, I don't think this ticket should remain open, and I'm afraid you may need to seek advice elsewhere, since the os-lib bug tracker is focused on issues with os-lib itself.

[lefou]

I moved this issue into a discussion, so you can still comment and share idea or progress. But as @SethTisue said, it's nothing we can directly "fix" or "improve" in os-lib. I think, tools like ls or git try to be smart to detect if they should output colors or not. It may be, they check the input stream or some env variable or some platform specific way to detect a tty (isatty), I actually don't know. But whatever it is, if you can control it, you can also enable colors. If not, you need to fall back to tool-specific settings. Most tools read env variables (e.g. LESSCOLOR IIRC), so you could provide some "defaults" for known tools.

[peacefulotter]

Thank you all for the advice,

I'll keep trying I guess, and, if you don't mind, post the solution here if I find one.

[s5bug]

The equivalent Maven/Gradle fix for installing pty4j for SBT is as follows:

resolvers +=
  "JetBrains IntelliJ Dependencies" at "https://packages.jetbrains.team/maven/p/ij/intellij-dependencies"

which will let you properly

libraryDependencies +=
  "org.jetbrains.pty4j" % "pty4j" % "0.12.13"

In Ammonite:

@ import coursierapi.MavenRepository
import coursierapi.MavenRepository

@ interp.repositories.update(
>   interp.repositories() ::: List(MavenRepository.of("https://packages.jetbrains.team/maven/p/ij/intellij-dependencies"))
> )

@ import $ivy.`org.jetbrains.pty4j:pty4j:0.12.13`

[lefou]

In Mill, it should like like this:

import coursier.maven.MavenRepository

def repositoriesTask = T.task {
  super.repositoriesTask() ++ 
    MavenRepository("https://packages.jetbrains.team/maven/p/ij/intellij-dependencies")
}

def ivyDeps = T {
  super.ivyDeps() ++ Agg(
    ivy"org.jetbrains.pty4j:pty4j:0.12.13"
  )
}