Skip to content

BUG: enlarging a DataFrame by adding an extra column with a tz-aware datetime results in object dtype #55423

Description

@MarcoGorelli

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

import datetime as dt

df = pd.DataFrame([{'id': 1},{'id': 2},{'id': 3}])
_time = dt.datetime.utcfromtimestamp(1695887042)
_time = _time.replace(tzinfo=dt.timezone.utc)
df.loc[df.id>=2, 'time'] = _time
print(df)
print(df.dtypes)

Issue Description

the above outputs

   id                       time
0   1                        NaT
1   2  2023-09-28 07:44:02+00:00
2   3  2023-09-28 07:44:02+00:00
id       int64
time    object
dtype: object

Expected Behavior

it would be nice to get to

   id                      time
0   1                       NaT
1   2 2023-09-28 07:44:02+00:00
2   3 2023-09-28 07:44:02+00:00
id                    int64
time    datetime64[ns, UTC]
dtype: object

right away

Installed Versions

Details

[1/1] Generating write_version_file with a custom command

  • /home/marcogorelli/.virtualenvs/pandas/bin/ninja

INSTALLED VERSIONS

commit : 99efe62
python : 3.10.12.final.0
python-bits : 64
OS : Linux
OS-release : 5.10.102.1-microsoft-standard-WSL2
Version : #1 SMP Wed Mar 2 00:30:59 UTC 2022
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_GB.UTF-8
LOCALE : en_GB.UTF-8

pandas : 2.1.0rc0+99.g99efe62afd.dirty
numpy : 1.24.4
pytz : 2023.3.post1
dateutil : 2.8.2
setuptools : 59.6.0
pip : 23.2.1
Cython : 0.29.33
pytest : 7.4.2
hypothesis : 6.86.2
sphinx : 6.2.1
blosc : 1.11.1
feather : None
xlsxwriter : 3.1.4
lxml.etree : 4.9.3
html5lib : 1.1
pymysql : 1.4.6
psycopg2 : 2.9.7
jinja2 : 3.1.2
IPython : 8.15.0
pandas_datareader : None
bs4 : 4.12.2
bottleneck : 1.3.7
dataframe-api-compat: None
fastparquet : 2023.8.0
fsspec : 2023.9.1
gcsfs : 2023.9.1
matplotlib : 3.7.3
numba : 0.57.1
numexpr : 2.8.6
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : 13.0.0
pyreadstat : 1.2.3
python-calamine : None
pyxlsb : 1.0.10
s3fs : 2023.9.1
scipy : 1.11.2
sqlalchemy : 2.0.21
tables : 3.8.0
tabulate : 0.9.0
xarray : 2023.8.0
xlrd : 2.0.1
zstandard : 0.21.0
tzdata : 2023.3
qtpy : None
pyqt5 : None
None

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions