SettingWithCopyError when scaling subset of columns with StandardScaler - pandas

Question

I have a dataframe df with 100 columns:
index | col1 | col2 | col3 | ...
2021-04-01 | qwe | 1 | 1.1 | ...
2021-04-02 | asd | 2 | 2.2 | ...
2021-04-03 | yxc | 3 | 3.3 | ...
dtypes:
col1: category
col2: int32
col3: float64
I want to scale all columns that are not of type "category" AND return it as a dataframe, not a numpy array.
My code so far:
y_feature = "col2"
y = df[[y_feature]] # Set predictor y
X = df.drop(
[
y_feature,
],
axis=1,
)
days = (
pd.date_range(start=df.index.min(), end=df.index.max())
.to_frame(name="date")
.reset_index()
.drop("index", axis=1)
)
limit_training_days = int(len(days.index) * 0.85)
X_train_limit = days.iloc[limit_training_days, 0]
print(f"Date for training: {X_train_limit}")
X_train, y_train = (
X.query("date <= #X_train_limit").squeeze(),
y.query("date <= #X_train_limit").squeeze(),
)
X_test, y_test = (
X.query("date > #X_train_limit").squeeze(),
y.query("date > #X_train_limit").squeeze(),
)
categorical_feature = X_train.select_dtypes("category").columns.tolist()
num_cols = X.drop(categorical_feature, axis=1).columns.tolist()
X_train[num_cols] = scaler.fit_transform(X_train[num_cols])
X_test_sc[num_cols] = scaler.transform(X_test[num_cols])
After updating my packages it now throws this error for the last 2 lines of code:
SettingWithCopyError: A value is trying to be set on a copy of a
slice from a DataFrame. Try using .loc[row_indexer,col_indexer] =
value instead
See the caveats in the documentation:
https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
How can I scale only non-category columns (keeping category columns untouched) AND return it as a dataframe?

Answer 1

The problem is you trying to change X_train and X_test which are parts of a bigger dataframe. Try:
X_train, X_test = X_train.copy(), X_test.copy()
before scaling.
You can also do:
X_train, y_train = (
X.query("date <= #X_train_limit").squeeze().copy(), # here
y.query("date <= #X_train_limit").squeeze(),
)