Can not list/retrieve specific request with keyword "timespan"

[pgeier]

What happened?

I am testing encoding output from ERA6 with the additional keyword timespan from metkit feature/mtg2_merge. I have output in a local FDB (/lus/h2resw01/scratch/mapg/5220/mapg-20250721-FORTRAN-20250820-093123-68e8e8e5f5793386b953a291b9cab93d-cmp_dev/fdbs/fdb/etc/fdb/config.yaml on ATOS).

I'm not able to retrieve list/retrieve specific fields if the keyword "timespan" is contained.
Without timespan I can list and retrieve, but the DataRetriever returns me only an empty bytearray() and not the whole grib message.
(I'm interested in the grib message, not the contained values because we do not store any values for testing metadata encoding).

What are the steps to reproduce the bug?

With pyfdb I am listing up all fields with fdb.list(keys=True).
For inspection and comparison I am picking an entry to perform a retrieve:

fdb.retrieve({'param': '162112',
 'levtype': 'ml',
 'date': '20220531',
 'expver': '5220',
 'time': '1800',
 'timespan': 'fs',
 'type': 'fc',
 'class': 'e6',
 'domain': 'g',
 'levelist': '115',
 'year': '2022',
 'step': '2',
 'stream': 'lwda',
 'month': '5',
 'anoffset': '3'})

This gives me an very unexpected error due to an failing ASSERT from fdb (see Relevant LOG Output)

Taking the same request in fdb.list(entry) returns an empty list (although this entry has been returned by fdb before).
Removing timespan from the entry solves the issue and lists the field again:

[{'path': '/lus/h2resw01/scratch/mapg/ERA6-LWDA/mapg-20250721-CPP-20250820-105456-481e7474ac647ce014657e7c08618b22-cmp_dev/fdbs/fdb01/root/e6:5220:lwda:2022:g/5:fc:ml:3:fs.20250820.105851.ad6-198.bullx.13080662202384405.data',
  'offset': 10895664,
  'length': 2964}]

However with the DataRetriever return from d = fdb.retrieve(entry) I'm only getting on empty bytearray(b''). Also d.size() returns 0, although the list showed a length of 2964.

Version

develop

Platform (OS and architecture)

ATOS

Relevant log output

Unexpected error on fdb.retrieve(entry):

Assertion failed: ax (/etc/ecmwf/nfs/dh2_perm_a/mapg/multio-dev/source/fdb5/src/fdb5/types/TypeStep.cc:75 getValues)
backtrace [9] stack has 50 addresses
(/perm/mapg/multio-dev/install/lib/../lib/libeckit.so+eckit::BackTrace::dump[abi:cxx11]())0x1af
(/perm/mapg/multio-dev/install/lib/../lib/libeckit.so+eckit::handle_assert(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, eckit::CodeLocation const&))0x5bd
(/perm/mapg/multio-dev/install/lib/libfdb5.so+fdb5::TypeStep::getValues(metkit::mars::MarsRequest const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string
<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, fdb5::Notifier const&, fdb5::CatalogueReader const*) const)0x17a
(/perm/mapg/multio-dev/install/lib/libfdb5.so+fdb5::MultiRetrieveVisitor::values(metkit::mars::MarsRequest const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, fdb5::TypesRegistry const&, s
td::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&))0x6d
(/perm/mapg/multio-dev/install/lib/libfdb5.so+fdb5::Rule::findMatchingKeys(metkit::mars::MarsRequest const&, fdb5::ReadVisitor&) const)0xa5
(/perm/mapg/multio-dev/install/lib/libfdb5.so+fdb5::RuleDatum::expand(metkit::mars::MarsRequest const&, fdb5::ReadVisitor&, fdb5::Key&) const)0x24
(/perm/mapg/multio-dev/install/lib/libfdb5.so+fdb5::RuleIndex::expand(metkit::mars::MarsRequest const&, fdb5::ReadVisitor&, fdb5::Key&) const)0xf9
(/perm/mapg/multio-dev/install/lib/libfdb5.so+fdb5::RuleDatabase::expand(metkit::mars::MarsRequest const&, fdb5::ReadVisitor&) const)0x81
(/perm/mapg/multio-dev/install/lib/libfdb5.so+fdb5::Schema::expand(metkit::mars::MarsRequest const&, fdb5::ReadVisitor&) const)0x48
(/perm/mapg/multio-dev/install/lib/libfdb5.so+fdb5::Inspector::inspect(metkit::mars::MarsRequest const&, fdb5::Schema const&, fdb5::Notifier const&) const)0xaa
(/perm/mapg/multio-dev/install/lib/libfdb5.so+fdb5::Inspector::inspect(metkit::mars::MarsRequest const&) const)0x39
(/perm/mapg/multio-dev/install/lib/libfdb5.so+fdb5::FDB::retrieve(metkit::mars::MarsRequest const&))0x12e

Accompanying data

/lus/h2resw01/scratch/mapg/5220/mapg-20250721-FORTRAN-20250820-093123-68e8e8e5f5793386b953a291b9cab93d-cmp_dev/fdbs/fdb/etc/fdb/config.yaml

Organisation

ECMWF

[ChrisspyB]

Hi @pgeier , I'll try to reproduce

[ChrisspyB]

Hi @pgeier , I think this might be because metkit is expanding fs to from-start on retrieve, but not on archive (as you pointed out in your other issue). I'd be curious to know if you can retrieve timespan=1h from your FDB.

I have noticed using your config file I can list 1h:

fdb-list param=162112,levtype=ml,date=20220531,expver=5220,time=1800,class=e6,timespan=1h
Listing for request
list,
        class=e6,
        levtype=ml,
        date=20220531,
        time=1800,
        expver=5220,
        param=162112,
        timespan=1h


{class=e6,expver=5220,stream=lwda,year=2022,domain=g}{month=5,type=fc,levtype=ml,anoffset=3,timespan=1h}{date=20220531,time=1800,step=1,levelist=100,param=162112}
{class=e6,expver=5220,stream=lwda,year=2022,domain=g}{month=5,type=fc,levtype=ml,anoffset=3,timespan=1h}{date=20220531,time=1800,step=1,levelist=101,param=162112}
{class=e6,expver=5220,stream=lwda,year=2022,domain=g}{month=5,type=fc,levtype=ml,anoffset=3,timespan=1h}{date=20220531,time=1800,step=1,levelist=102,param=162112}
{class=e6,expver=5220,stream=lwda,year=2022,domain=g}{month=5,type=fc,levtype=ml,anoffset=3,timespan=1h}{date=20220531,time=1800,step=1,levelist=103,param=162112}
...

but not fs

$: /ec/res4/scratch/maby/fdb_github_163/bin/fdb-list param=162112,levtype=ml,date=20220531,expver=5220,time=1800,class=e6,timespan=fs
Listing for request
list,
        class=e6,
        levtype=ml,
        date=20220531,
        time=1800,
        expver=5220,
        param=162112,
        timespan=from-start <---!!!!

[pgeier]

Well, that strongly relates to #161. If expand and validation had happened the error would not be here?

[ChrisspyB]

@pgeier and I discussed on teams.

In summary the issue arises because metkit is assuming from-start to be the canonical form of fs, whereas eccodes seems to use fs. This means the metadata extracted from eckit::message on archive uses timespan=fs. FDB assumes eckit will give it the canonical representation.

Updating the metkit language.yaml fixes the issue:

    before: [from-start, fs,fromstart]
    after:  [fs, from-start, fromstart]

But it is a data governance question whether or not this is the correct.