BUG: factorize with objects differentiate 0/1 from False, True #62888

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

ser = pd.Series([0, 1, True, False])

>>> ser.factorize()[1]
Index([0, 1], dtype='object')

Issue Description

factorize on object dtype doesn't differentiate 0 from False or 1 from True.

This is half of the issue behind #21108.

Expected Behavior

I'd expect the uniques to include [True, False, 1, 0]

Installed Versions