Odd behavior in Safari with pass-through enabled

[dchandekstark]

This is intended more as a flag than a full bug report, but after trying out iipsrv 1.3 on some pyramid TIFFs displayed in a Mirador viewer, I noticed some strange behvior in Safari in which b/w images would have regions that turned pink-and-green. I have attached a portion on one image, just for reference. The problem was not reproducible on Chrome or Firefox. Disabling pass-through by setting CODEC_PASSTHROUGH=0 appeared to solve the issue.

[ruven]

Thanks for flagging this. Does this happen to all B/W images or just certain images? Can you paste the output of tiffinfo for one of these images (tiffinfo should be included in the libtiff homebrew package if you're using MacOS)?

tiffinfo /path/to/your/image.tif

Alternatively, if you're able to, send me a small example TIFF that has this problem for me to test.

Also, if the image contains an ICC profile, try re-enabling CODEC_PASSTHROUGH=1 and setting MAX_ICC=0 to see if that makes a difference.

[dchandekstark]

It appeared to happen with certain images, but I only checked a few. In some cases, the effect appeared on initial loading; in others, only on zooming.

Attached is the output of tiffinfo -- I removed the XMLPacket fields for brevity. I don't know if the image contains an ICC profile; fwiw, the original from which the PTIF was derived is a JPEG in this case.

tiffinfo.txt

[dchandekstark]

I found an image that has an ICC profile, and your suggestion did not make a difference. I've uploaded the file zipped b/c GitHub would not allow TIFF.

test.tif.zip

Also, the effect happens (in Safari) also with (a not new version of) OpenSeaDragon, fwiw.

[bluebinary]

Hi @ruven and @dchandekstark, I have submitted a fix for this issue in: #282

I originally encountered and this issue last year while testing the new pass-through functionality, and developed a prototype fix at the time, but other commitments delayed finalizing the fix until now.

I have refactored and finalized the fix to take account of the new Compressor::injectMetadata() pathway in the code. This fix has only been implemented for JPEG as I do not believe it affects any other supported file format.

The root of the issue is that some JPEGs are in "Adobe JPEG" format and in those cases they need to have an APP14 marker segment present which notes to the JPEG library responsible for decoding the image how the colourspace information was encoded; when the APP14 marker is absent the colourspace information cannot be correctly interpreted, often leading to image tiles appearing exactly as per the examples above, often with a magenta/pink and/or greenish hue!

The issue generally does not affect edge tiles because they are not an exact match for a tile that can just be extracted as-is from the source image, and thus need to be processed through the regular Compressor::Compress() step such as via JPEGCompressor::Compress() for JPEG format images, as such the edges/bottoms of tiled images usually render just fine, while many of the other tiles will render incorrectly if pass-through mode is enabled.

This fix, which involves creating a jpeg_decompress_struct and a jpeg_create_decompress step to copy the already encoded coefficients from the already JPEG encoded rawtile.data into the output, and then adds in all the necessary and critical metadata. Importantly, it is the jpeg_* functions that are part of libjpeg or libturbojpeg library (whichever is being used) that are responsible for determining what metadata is deemed critical or necessary, so we don't need any special handling to facilitate this.

Furthermore, the decompression step does not actually cause the JPEG data to be decompressed, we simply copy the already compressed coefficients data from the input to the output, which is incredibly fast, and because we are constructing the output JPEG in a standards compliant way all the necessary metadata is being carried over from source image to destination image, including if needed, the APP14 Adobe marker.

One question I do have is that I notice that the call to JPEGCompressor::injectMetadata() short-circuits if no ICC, XMP or EXIF metadata is available, but I believe it is possible for an Adobe flavour JPEG to be affected by the lack of an APP14 marker segment even if no ICC colour Profile is preset, so it may be necessary to relax or remove this short-circuit so that the JPEGCompressor::injectMetadata() can run on all JPEG tiles in pass-through mode, to ensure that if the APP14 marker segment is needed in the output, that it will be added, regardless of whether an ICC colour profile is available or not.

[bluebinary]

Here is a sample tile processed from a Pyramidial TIFF image, that exhibited the missing APP14 marker through the untouched code from the master branch of the latest commit of IIPServer, 921c8fd.

The left image was generated as the output image without any changes, and exhibits the missing APP14 marker:

The right image was generated as the output image with the updated implementation of the JPEGCompressor::injectMetadata() method as provided in this pull request, and I was able to verify with a hex editor that the previously missing Adobe APP14 marker was added, and then visually confirm that the image rendered correctly with the correct colourspace information:

The image below is a capture from a hex editor comparing the tile before the fix with the same tile after the fix, highlighting the absence of the Adobe marker in the initial tile image:

[bluebinary]

Information on the APP14 marker isn't all that easy to come by, but as noted by Phil Harvey, the creator of the incredible ExifTool software, in this ExifTool post, https://exiftool.org/forum/index.php?topic=8695.0, the JPEG metadata reference page on Oracle's site does discuss the APP14 marker:

https://docs.oracle.com/javase/8/docs/api/javax/imageio/metadata/doc-files/jpeg_metadata.html

The relevant excerpt from the above that discusses the APP14 marker is as follows:

When reading, the contents of the stream are interpreted by the usual JPEG
conventions, as follows:

• If a JFIF APP0 marker segment is present, the colorspace is known to be
  either grayscale or YCbCr. If an APP2 marker segment containing an embedded
  ICC profile is also present, then the YCbCr is converted to RGB according to
  the formulas given in the JFIF spec, and the ICC profile is assumed to refer
  to the resulting RGB space.

• If an Adobe APP14 marker segment is present, the colorspace is determined by
  consulting the transform flag. The transform flag takes one of three values:

  2 - The image is encoded as YCCK (implicitly converted from CMYK on encoding).
  1 - The image is encoded as YCbCr (implicitly converted from RGB on encoding).
  0 - Unknown. 3-channel images are assumed to be RGB, 4-channel images are
  assumed to be CMYK.

• If neither marker segment is present, the following procedure is followed:
  Single-channel images are assumed to be grayscale, and 2-channel images are
  assumed to be grayscale with an alpha channel. For 3- and 4-channel images,
  the component ids are consulted. If these values are 1-3 for a 3-channel
  image, then the image is assumed to be YCbCr. Subject to the availability of
  the optional color space support described above, if these values are 1-4
  for a 4-channel image, then the image is assumed to be YCbCrA. If these
  values are > 4, they are checked against the ASCII codes for 'R', 'G', 'B',
  'A', 'C', 'c'. These can encode the following colorspaces:

  RGB
  RGBA
  YCC (as 'Y','C','c'), assumed to be PhotoYCC
  YCCA (as 'Y','C','c','A'), assumed to be PhotoYCCA

  Otherwise, 3-channel subsampled images are assumed to be YCbCr, 3-channel
  non-subsampled images are assumed to be RGB, 4-channel subsampled images are
  assumed to be YCCK, and 4-channel, non-subsampled images are assumed to be
  CMYK.

• All other images are declared uninterpretable and an exception is thrown if
  an attempt is made to read one as a BufferedImage. Such an image may be read
  only as a Raster. If an image is interpretable but there is no Java
  ColorSpace available corresponding to the encoded colorspace (e.g. YCbCr),
  then ImageReader.getRawImageType will return null.

[ruven]

Thank you @bluebinary for this in-depth analysis and code! I'll make some more technical comments directly in the pull request as I seem to be getting some memory-related issues when I test this.

The root of the issue is that some JPEGs are in "Adobe JPEG" format

"Adobe JPEGs" are JPEGs in an RGB color space, rather than the more usual YCbCr color space. Thus, as you say, to be correctly interpreted, the APP14 marker (0xEEFF in your hex dump) needs to be there for these images. Interestingly, libjpeg places this marker right after the very first 0xFFD8 JPEG start of file marker and before the JFIF APP0 marker (0xFFE0). Have you tested this in Safari and other browsers?

One question I do have is that I notice that the call to JPEGCompressor::injectMetadata() short-circuits if no ICC, XMP or EXIF metadata is available, but I believe it is possible for an Adobe flavour JPEG to be affected by the lack of an APP14 marker segment even if no ICC colour Profile is preset

I've tested this out on Firefox and Chrome with RGB (Adobe flavor) JPEGs with no ICC profile and, surprisingly, they render correctly without the APP14 marker. Maybe we need to test this more widely before coming to any absolute conclusions.

By the way, @dchandekstark, although your images are scans of B/W photos, the TIFFs are in fact in color with tiles encoded using this "Adobe JPEG" RGB color space. This is why your images are affected by what is essentially a color interpretation problem. If they had been true 1-band B/W, this discoloration wouldn't occur.