Fields declaration order really matters when using multi-variables pattern

[devlounge]

It looks to me that the order in which the dataclass fields are declared is super important.

Example directory structure:

environments
  dev
    config
      servers
        server-1
          server.yaml
        server-2
          server.yaml
        server-3
          server.yaml

Datafile definition:

@datafile("./environments/{self.environment}/config/servers/{self.name}/server.yaml")
class Server:
    name: str
    environment: str

When running:

import os

from manager.models import Cluster


if __name__ == '__main__':

    clusters = list(Server.objects.all())

It ends up raising:

FileNotFoundError: [Errno 2] No such file or directory: '/project/configuration/environments/server-1/config/servers/dev/server.yaml'

This seems to come from when the manager.all method does yield self.get(*values) with values ["server-1", "dev"] which ends up building the path posted above as manager.get iterates on the fields in the order they are declared and sets the values accordingly.

If I declare:

@datafile("./environments/{self.environment}/config/servers/{self.name}/server.yaml")
class Server:
    environment: str
    name: str

It works.

[jacebrowning]

Thanks for the clear steps to reproduce!

There may have been a reason it was implemented this way, but if someone wants to try to convert the parse() results into keyword arguments that may make this more robust:

datafiles/datafiles/manager.py

Lines 137 to 150 in 59566a6

	result = parse(pattern, filename) or parse(alt_pattern, filename)
	if result:
	values = list(result.named.values())
	if len(values) > 1 and os.sep in values[-1]:
	parts = values[-1].rsplit(os.sep, 1)
	values[-2] = values[-2] + os.sep + parts[0]
	values[-1] = parts[1]
	if _exclude and values[0].startswith(_exclude):
	log.debug(f"Skipped loading of excluded value: {values[0]}")
	continue
	yield self.get(*values)

[jacebrowning]

Alternatively, datafiles could raise an exception when the field ordering does not match the pattern.

Or we simply call out this gotcha in the documentation.

I'm open to suggestions!