Garbage data causes full file to fail decode

[Hydravine]

Me again!

As part of my testing I'm backfilling a lot of old telemetry files with our new extraction tool that uses CCSDSPy. On rare occasions some files completely fail at the split_by_apid step because of (what I believe to be) additional garbage data that made it into the file somehow during data dump from the spacecraft.

I made a quick and dirty alteration in the iter_packet_bytes function of utils.py adding a try/except so that the code could continue if it happens across garbage like this:

line 66 in utils.py

while offset < len(file_bytes):
        try:
            packet_nbytes = get_packet_total_bytes(
                file_bytes[offset : offset + PRIMARY_HEADER_NUM_BYTES].tobytes()
            )
            packet_bytes = file_bytes[offset + delta_idx : offset + packet_nbytes].tobytes()
    
            yield packet_bytes
    
            offset += packet_nbytes
        except:
            print(f"Error in iter_packet_bytes at offset {offset}, Primary header byte sequence of {len(file_bytes[offset : offset + 
                       PRIMARY_HEADER_NUM_BYTES].tobytes())} must be {PRIMARY_HEADER_NUM_BYTES} bytes long")
            offset += len(file_bytes[offset : offset + PRIMARY_HEADER_NUM_BYTES].tobytes())

I tested this version and it works in the sense that the CCSDSPy code can continue and decode the rest of the file instead of failing. I get a single error:

"Error in iter_packet_bytes at offset 35034720, Primary header byte sequence of 2 must be 6 bytes long"

and then the code continues on decoding the rest of the file because it skips over those 2 bytes and continues, vs the current release fails. Would it be possible for you to add something like this to an official release? I think it's very beneficial.

Thanks again for all your help,

Bill

[Hydravine]

When I say the current release fails this is what I mean:

 File "C:\Users\wparsons\.conda\envs\GMOC_SCHM\Lib\site-packages\ccsdspy\decode.py", line 39, in _get_packet_total_bytes
    raise ValueError(
ValueError: Primary header byte sequence of 2 must be 6 bytes long

Note: I added the "of 2" part for my debugging, this is from line 39/40 in decode.py

[fsenay]

I pulled the main branch and confirmed that the latest changes were there. Unfortunately, I still have the same error. Here is the Traceback:

[ddasilva]

You are right! This is a related but different problem. I will write a patch in the next couple days

[ddasilva]

I was able to reproduce the issue and pushed a fix and a new regression test to the main branch. Can you try again?

[fsenay]

I can confirm that it works! Reran a bunch of problematic files with no issues! Thanks for the quick update!

[ddasilva]

Great! A new release has been made