Suggestion: populate INFO name in warning "INFO tag id=146"

[davmlaw]

At the moment, some errors look like:

... Invalid BCF, the INFO tag id=146 is too large at ....

Would it be possible to look up the tagID and replace it with a more human readable string of the name? That would help people debug issues more easily.

An example where a better warning may have helped someone else samtools/bcftools#1532

Thanks!

[pd3]

Yes, that's a sensible request :)
This comes from htslib, transferring the issue..

[pd3]

Any chance you could provide a small test case and some additional detail, where you run into this issue?

[daviesrob]

Unfortunately, it's not possible to print a name when you get this error. HTSlib uses id numbers internally to reference data like INFO and FORMAT names based on where they were found in the header. The message is printed when HTSlib can't find a header record corresponding to the id number, and as there's no header record it doesn't have a name that it can print.

Usually you can get away with this when outputting VCF because the reader adds header records for you when it comes across an INFO name it hasn't seen before. That's when you see the INFO 'whatever' is not defined in the header, assuming Type=String message. It isn't ideal as it doesn't know what type it should use and it's likely the header will already have been written out by the time it does so the problem gets carried over to the next program that tries to read the data. It also doesn't work in some cases, as in the linked one where bcftools norm died when it found one of these records (this is because it uses two copies of the header data, and only one got the new entry).

What we really need to do is improve the not defined in the header message so that it points to the line involved, and also explains how to better fix the problem.

[jkbonfield]

As discussed in our meeting, my gut instinct on this is that the file is in error. INFO tags in the header are not optional and this is a file format error. We shouldn't warn, but bail out. However the flip side to this is we also need a "fix my broken VCF" tool instead, which is an explicit thing to run and not some implicit bodge-it-on-read-and-hope-it-works thing! (Ie what view is doing now).

One possible strategy here would be a longer form of bcftools header which outputs the current header plus a best guess attempt at adding the missing information. Obviously it would need to read the entire file to do this, and it'd need a mechanism of managing that without actually parsing the data via the normal functions or we hit a catch 22 situation (or we just get that code to have a different mode that prints up the missing header line as it adds to the header, as view is already doing bar the printing-bit).

Once we've got something that can print up a new header, with a basic starting point on the missing items, the user can manually edit this text file adding descriptions, setting types or specifying array sizes as appropriate, and then use bcftools reheader to unbreak their input data.

Is it messy? Yes.
Is it a bit laborious and slow? Also yes.

However maybe it'll also encourage people to stop creating broken files and actually care about meta-data and provenance.