Pandas `.to_pydatetime()` Not Working Inside A Dataframe

I have strings like '03-21-2019' that I want to convert to the native Python datetime object: that is, of the datetime.datetime type. The conversion is easy enough through pandas:

Solution 1:

Depending on what your actual goal is, you've a couple options you didn't mention directly.

1) If you have a static datetime object or a column of (pandas) Timestamps, and you're willing to deal with the Pandas version of a Timedelta (pandas._libs.tslibs.timedeltas.Timedelta), you can do the subtraction directly in pandas:

df = pd.DataFrame(columns=['Date'])
df.loc[0] = [pd.to_datetime('03-21-2019')]
df.loc[:, 'Offset'] = pd.Series([datetime.now()])
df.loc[:, 'Diff1'] = df['Offset'] - df['Date']
df.loc[:, 'Diff2'] = df['Date'] - datetime.now()

2) If you don't care about Dataframes, but are willing to deal with lists / numpy arrays, you can convert the datetimes to python-native datetimes by operating on the series rather than on individual elements. Below, arr is a numpy.ndarray of datetime.datetime objects. You can change it to a regular list of datetime with list(arr):

arr = df['Date'].dt.to_pydatetime()

Solution 2:

Thanks to the Sarah Messer’s answer and this one, I could solve the problem by reassigning the array back to the dataframe and forcing its dtype to object:

arr_date = df['Date'].dt.to_pydatetime()
df['Date']= pd.Series(arr_date, dtype=object)

example:

import pandas as pd
from datetime import datetime

df = pd.DataFrame({"date": [datetime(2021, 8, 28, 4, 10), datetime(2021, 8, 28, 4, 10)]})
df.dtypes

#   date    datetime64[ns]#   dtype: object

arr_date = df["date"].dt.to_pydatetime()
df["date"] = pd.Series(arr_date, dtype="object")
df.dtypes

#   date    object#   dtype: object

df.iloc[0,0]

# datetime.datetime(2021, 8, 28, 4, 10)