Dont list directorymarkers in iterdir

[ap--]

Should close #226

[danielgafni]

Perhaps we should also check for "ContentType": "application/x-directory; charset=UTF-8", as a precaution? This would prevent the special key from filtering out in case the user has actually set it.

[ap--]

It seems, the way info is implemented in s3fs, ContentType is currently not propagated for these directory markers.

[ap--]

If you could explicitly confirm that the PR as-is fixes your issue I'd be happy to merge ☺️

[danielgafni]

1. I don't have access to AWS S3 right now (I can set it up later), so I am testing it with Cloudflare R2. I also artificially created an empty object at s3://anam-data-dev/rofl/ via s3fs I've also created another empty object at s3://anam-data-dev/rofl/object .
2. I've encountered a different issue now:

/tmp/rofl via 🐍 v3.12.7 (rofl) via   on 🅰 (auto)
› uv pip install git+https://github.com/ap--/universal_pathlib.git@dont-list-directorymarkers-in-iterdir
Resolved 3 packages in 540ms
Uninstalled 1 package in 2ms
░░░░░░░░░░░░░░░░░░░░ [0/1] Installing wheels...                                                                              
Installed 1 package in 6ms
 - universal-pathlib==0.3.8
 + universal-pathlib==0.0.12.post330+gb5d91d25e (from git+https://github.com/ap--/universal_pathlib.git@b5d91d25e2f7acb0e533617a1e38184d375944d9)

/tmp/rofl via 🐍 v3.12.7 (rofl) via   on 🅰 (auto)
› ipython
Python 3.12.7 (main, Oct 16 2024, 04:37:19) [Clang 18.1.8 ]
Type 'copyright', 'credits' or 'license' for more information
IPython 9.9.0 -- An enhanced Interactive Python. Type '?' for help.
Tip: You can find how to type a LaTeX symbol by back-completing it, eg `\θ<tab>` will expand to `\theta`.

In [1]: from upath import UPath

In [2]: path = UPath("s3://anam-data-dev/")

In [3]: list(path.iterdir())
Out[3]:
[S3Path('anam-data-dev/rofl', protocol='s3'),
 S3Path('anam-data-dev/rofl', protocol='s3'),
 S3Path('anam-data-dev/test', protocol='s3'),
 S3Path('anam-data-dev/test.delta', protocol='s3'),
 S3Path('anam-data-dev/video-factory', protocol='s3'),
 S3Path('anam-data-dev/video-generation', protocol='s3')]

In [4]: path = UPath("s3://anam-data-dev/rofl")

In [5]: list(path.iterdir())
---------------------------------------------------------------------------
NotADirectoryError                        Traceback (most recent call last)
Cell In[5], line 1
----> 1 list(path.iterdir())

File /tmp/rofl/.venv/lib/python3.12/site-packages/upath/core.py:1192, in UPath.iterdir(self)
   1190 base_path = base.path
   1191 if not fs.isdir(base_path):
-> 1192     raise NotADirectoryError(str(self))
   1193 name: str
   1194 for info in fs.listdir(base_path):
   1195     # fsspec returns dictionaries

NotADirectoryError: s3://anam-data-dev/rofl

In [6]: path = UPath("s3://anam-data-dev/rofl/")

In [7]: list(path.iterdir())
---------------------------------------------------------------------------
NotADirectoryError                        Traceback (most recent call last)
Cell In[7], line 1
----> 1 list(path.iterdir())

File /tmp/rofl/.venv/lib/python3.12/site-packages/upath/core.py:1192, in UPath.iterdir(self)
   1190 base_path = base.path
   1191 if not fs.isdir(base_path):
-> 1192     raise NotADirectoryError(str(self))
   1193 name: str
   1194 for info in fs.listdir(base_path):
   1195     # fsspec returns dictionaries

NotADirectoryError: s3://anam-data-dev/rofl

So the artificial marker object breaks universal-pathlib right now (for me) when trying to list the marker path. It looks like the requirement here is to have both s3://anam-data-dev/rofl/ and s3://anam-data-dev/rofl/object empty.

[danielgafni]

Also this seems a little weird to me

[ap--]

I don't have access to AWS S3 right now (I can set it up later), so I am testing it with Cloudflare R2. I also artificially created an empty object at s3://anam-data-dev/rofl/ via s3fs I've also created another empty object at s3://anam-data-dev/rofl/object

s3fs doesn't let you do this like the aws console would because it strips the trailing slash

[ap--]

Hmmm, seems like yet another edge case. Okay I'll see if this can be handled in a better way.