{
"cells": [
{
"cell_type": "markdown",
"id": "31b8665b",
"metadata": {},
"source": [
"# Variable selection bias and importance scores\n",
"\n",
"Exhaustive-search trees such as CART favor variables with many distinct values: such a variable offers more cutpoints, so by chance alone one of them is more likely to reduce impurity. GUIDE avoids this bias by choosing the split variable with chi-square tests, whose p-values put variables with different numbers of values on a common scale.\n",
"\n",
"This matters most for interpretation: a tree's split variables, and importance scores built from them, can point to noise. This notebook shows the effect with a null simulation and then compares importance scores on the diabetes data with added noise variables."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "ec0e26ed",
"metadata": {
"execution": {
"iopub.execute_input": "2026-11-02T22:20:59.173554Z",
"iopub.status.busy": "2026-11-02T22:20:59.173316Z",
"iopub.status.idle": "2026-11-02T22:21:00.467782Z",
"shell.execute_reply": "2026-11-02T22:21:00.467363Z"
}
},
"outputs": [],
"source": [
"import json\n",
"\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"import pandas as pd\n",
"from scipy.stats import chisquare\n",
"from sklearn.datasets import load_diabetes\n",
"from sklearn.ensemble import RandomForestRegressor\n",
"from sklearn.inspection import permutation_importance\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.tree import DecisionTreeRegressor\n",
"\n",
"from guide import GuideRegressor, importance"
]
},
{
"cell_type": "markdown",
"id": "ea4efca5",
"metadata": {},
"source": [
"## A null simulation\n",
"\n",
"Five predictors, all independent of the response, differ only in their number of distinct values: 2, 5, 10, 50 and continuous. In each of 500 replications we fit a single split (a stump) with each method and record the variable it splits on. With no bias, each variable should be chosen about 20% of the time."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "73e2bf65",
"metadata": {
"execution": {
"iopub.execute_input": "2026-11-02T22:21:00.469166Z",
"iopub.status.busy": "2026-11-02T22:21:00.469054Z",
"iopub.status.idle": "2026-11-02T22:21:00.814932Z",
"shell.execute_reply": "2026-11-02T22:21:00.814411Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"
"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"GUIDE: chi-square test of uniform selection, p = 0.0908\n",
"CART: chi-square test of uniform selection, p = 6.12e-77\n"
]
}
],
"source": [
"ax = (counts / reps).plot.bar(figsize=(8, 4), rot=0)\n",
"ax.axhline(0.2, color=\"0.3\", ls=\"--\", lw=1)\n",
"ax.set(ylabel=\"proportion of root splits\", title=\"Root split variable under the null\")\n",
"plt.show()\n",
"for method in counts:\n",
" print(f\"{method}: chi-square test of uniform selection, p = {chisquare(counts[method]).pvalue:.3g}\")"
]
},
{
"cell_type": "markdown",
"id": "bf7bcdd4",
"metadata": {},
"source": [
"CART almost always splits on the continuous or 50-value variable and rarely on the binary one. GUIDE's choices are consistent with uniform selection.\n",
"\n",
"## Importance scores with noise variables\n",
"\n",
"Next, the diabetes data plus three noise variables: a continuous one, a binary one, and a 40-level categorical \"clinic\" code. We compare:\n",
"\n",
"- **GUIDE importance** (`guide.importance`, Loh and Zhou 2021): scores from GUIDE's selection statistics, adjusted by their mean under permutations of the response, with a threshold separating important from unimportant variables.\n",
"- **Random forest impurity importance** (`feature_importances_`), which inherits CART's selection bias.\n",
"- **Permutation importance** of the random forest on a held-out 30% of the data, scikit-learn's recommended alternative to impurity importance."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "beee2222",
"metadata": {
"execution": {
"iopub.execute_input": "2026-11-02T22:21:00.909823Z",
"iopub.status.busy": "2026-11-02T22:21:00.909747Z",
"iopub.status.idle": "2026-11-02T22:21:03.545240Z",
"shell.execute_reply": "2026-11-02T22:21:03.544743Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
"
\n",
"
\n",
"
GUIDE adjusted
\n",
"
GUIDE group
\n",
"
RF impurity
\n",
"
RF permutation
\n",
"
GUIDE adjusted rank
\n",
"
RF impurity rank
\n",
"
RF permutation rank
\n",
"
\n",
" \n",
" \n",
"
\n",
"
s5
\n",
"
19.786
\n",
"
highly_important
\n",
"
0.290
\n",
"
0.177
\n",
"
1
\n",
"
2
\n",
"
1
\n",
"
\n",
"
\n",
"
bmi
\n",
"
17.871
\n",
"
highly_important
\n",
"
0.292
\n",
"
0.160
\n",
"
2
\n",
"
1
\n",
"
2
\n",
"
\n",
"
\n",
"
bp
\n",
"
11.916
\n",
"
highly_important
\n",
"
0.087
\n",
"
0.037
\n",
"
3
\n",
"
3
\n",
"
3
\n",
"
\n",
"
\n",
"
s4
\n",
"
9.628
\n",
"
highly_important
\n",
"
0.024
\n",
"
-0.007
\n",
"
4
\n",
"
11
\n",
"
11
\n",
"
\n",
"
\n",
"
s6
\n",
"
9.133
\n",
"
highly_important
\n",
"
0.044
\n",
"
0.004
\n",
"
5
\n",
"
6
\n",
"
6
\n",
"
\n",
"
\n",
"
s3
\n",
"
8.835
\n",
"
highly_important
\n",
"
0.054
\n",
"
-0.015
\n",
"
6
\n",
"
4
\n",
"
13
\n",
"
\n",
"
\n",
"
noise binary
\n",
"
2.948
\n",
"
unimportant
\n",
"
0.008
\n",
"
0.009
\n",
"
7
\n",
"
12
\n",
"
4
\n",
"
\n",
"
\n",
"
s1
\n",
"
2.736
\n",
"
unimportant
\n",
"
0.034
\n",
"
-0.002
\n",
"
8
\n",
"
10
\n",
"
9
\n",
"
\n",
"
\n",
"
s2
\n",
"
2.375
\n",
"
unimportant
\n",
"
0.038
\n",
"
-0.001
\n",
"
9
\n",
"
8
\n",
"
8
\n",
"
\n",
"
\n",
"
age
\n",
"
2.257
\n",
"
unimportant
\n",
"
0.037
\n",
"
0.004
\n",
"
10
\n",
"
9
\n",
"
5
\n",
"
\n",
"
\n",
"
sex
\n",
"
1.876
\n",
"
unimportant
\n",
"
0.007
\n",
"
0.002
\n",
"
11
\n",
"
13
\n",
"
7
\n",
"
\n",
"
\n",
"
noise continuous
\n",
"
1.739
\n",
"
unimportant
\n",
"
0.047
\n",
"
-0.006
\n",
"
12
\n",
"
5
\n",
"
10
\n",
"
\n",
"
\n",
"
clinic
\n",
"
1.265
\n",
"
unimportant
\n",
"
0.038
\n",
"
-0.014
\n",
"
13
\n",
"
7
\n",
"
12
\n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" GUIDE adjusted GUIDE group RF impurity \\\n",
"s5 19.786 highly_important 0.290 \n",
"bmi 17.871 highly_important 0.292 \n",
"bp 11.916 highly_important 0.087 \n",
"s4 9.628 highly_important 0.024 \n",
"s6 9.133 highly_important 0.044 \n",
"s3 8.835 highly_important 0.054 \n",
"noise binary 2.948 unimportant 0.008 \n",
"s1 2.736 unimportant 0.034 \n",
"s2 2.375 unimportant 0.038 \n",
"age 2.257 unimportant 0.037 \n",
"sex 1.876 unimportant 0.007 \n",
"noise continuous 1.739 unimportant 0.047 \n",
"clinic 1.265 unimportant 0.038 \n",
"\n",
" RF permutation GUIDE adjusted rank RF impurity rank \\\n",
"s5 0.177 1 2 \n",
"bmi 0.160 2 1 \n",
"bp 0.037 3 3 \n",
"s4 -0.007 4 11 \n",
"s6 0.004 5 6 \n",
"s3 -0.015 6 4 \n",
"noise binary 0.009 7 12 \n",
"s1 -0.002 8 10 \n",
"s2 -0.001 9 8 \n",
"age 0.004 10 9 \n",
"sex 0.002 11 13 \n",
"noise continuous -0.006 12 5 \n",
"clinic -0.014 13 7 \n",
"\n",
" RF permutation rank \n",
"s5 1 \n",
"bmi 2 \n",
"bp 3 \n",
"s4 11 \n",
"s6 6 \n",
"s3 13 \n",
"noise binary 4 \n",
"s1 9 \n",
"s2 8 \n",
"age 5 \n",
"sex 7 \n",
"noise continuous 10 \n",
"clinic 12 "
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"X, y = load_diabetes(return_X_y=True, as_frame=True)\n",
"rng = np.random.default_rng(1)\n",
"X[\"noise continuous\"] = rng.normal(size=len(X))\n",
"X[\"noise binary\"] = rng.integers(0, 2, len(X)).astype(float)\n",
"clinic = rng.integers(0, 40, len(X))\n",
"X_guide = X.assign(clinic=pd.Categorical(clinic.astype(str))) # GUIDE: categorical\n",
"X_sklearn = X.assign(clinic=clinic.astype(float)) # scikit-learn: integer codes\n",
"\n",
"report = importance(GuideRegressor(), X_guide, y)\n",
"guide_scores = pd.DataFrame(report[\"scores\"]).set_index(\"feature\")\n",
"\n",
"X_train, X_test, y_train, y_test = train_test_split(X_sklearn, y, test_size=0.3, random_state=0)\n",
"forest = RandomForestRegressor(n_estimators=300, random_state=0).fit(X_train, y_train)\n",
"permuted = permutation_importance(forest, X_test, y_test, n_repeats=20, random_state=0)\n",
"\n",
"table = pd.DataFrame({\n",
" \"GUIDE adjusted\": guide_scores[\"adjusted_score\"],\n",
" \"GUIDE group\": guide_scores[\"group\"],\n",
" \"RF impurity\": pd.Series(forest.feature_importances_, X_sklearn.columns),\n",
" \"RF permutation\": pd.Series(permuted.importances_mean, X_sklearn.columns),\n",
"})\n",
"ranks = table[[\"GUIDE adjusted\", \"RF impurity\", \"RF permutation\"]].rank(ascending=False)\n",
"table.join(ranks.add_suffix(\" rank\").astype(int)).sort_values(\"GUIDE adjusted\", ascending=False).round(3)"
]
},
{
"cell_type": "markdown",
"id": "82a1ee14",
"metadata": {},
"source": [
"GUIDE places all three noise variables in its unimportant group. The random forest's impurity importance ranks the continuous noise and the 40-level clinic code above several real variables, while the binary noise variable, with a single possible cutpoint, ranks near the bottom: the same bias as in the simulation.\n",
"\n",
"Held-out permutation importance has no such bias, but beyond the three strongest variables its values are small and noisy; here the binary noise variable ranks fourth by chance."
]
},
{
"cell_type": "markdown",
"id": "e45ef79a",
"metadata": {},
"source": [
"## Boosting\n",
"\n",
"Gradient boosting fits hundreds of trees to the gradients of the loss, and split counts or gain summed over those trees are a common importance measure. `GuideBoostingRegressor` (experimental) chooses each split variable with GUIDE's tests on the signs of the negative gradients, then the split point by the usual second-order gain. The same null simulation, with 50 boosting rounds of depth-3 trees, compares the share of all splits that land on each pure-noise predictor with scikit-learn's `GradientBoostingRegressor`:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "a2a6d6bf",
"metadata": {
"execution": {
"iopub.execute_input": "2026-11-02T22:21:03.546267Z",
"iopub.status.busy": "2026-11-02T22:21:03.546195Z",
"iopub.status.idle": "2026-11-02T22:21:04.101723Z",
"shell.execute_reply": "2026-11-02T22:21:04.101191Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"
\n",
" \n",
"
\n",
"
\n",
"
GUIDE boosting
\n",
"
scikit-learn boosting
\n",
"
\n",
" \n",
" \n",
"
\n",
"
binary
\n",
"
0.140
\n",
"
0.042
\n",
"
\n",
"
\n",
"
5 values
\n",
"
0.198
\n",
"
0.102
\n",
"
\n",
"
\n",
"
10 values
\n",
"
0.200
\n",
"
0.160
\n",
"
\n",
"
\n",
"
50 values
\n",
"
0.248
\n",
"
0.266
\n",
"
\n",
"
\n",
"
continuous
\n",
"
0.214
\n",
"
0.431
\n",
"
\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" GUIDE boosting scikit-learn boosting\n",
"binary 0.140 0.042\n",
"5 values 0.198 0.102\n",
"10 values 0.200 0.160\n",
"50 values 0.248 0.266\n",
"continuous 0.214 0.431"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from sklearn.ensemble import GradientBoostingRegressor\n",
"\n",
"from guide import GuideBoostingRegressor\n",
"\n",
"\n",
"def guide_split_counts(model, p):\n",
" counts = np.zeros(p)\n",
" def walk(node):\n",
" if node.get(\"split\"):\n",
" counts[node[\"split\"][\"feature\"]] += 1\n",
" walk(node[\"left\"])\n",
" walk(node[\"right\"])\n",
" for tree in json.loads(model.to_json())[\"trees\"]:\n",
" walk(tree[\"root\"])\n",
" return counts\n",
"\n",
"\n",
"def sklearn_split_counts(model, p):\n",
" counts = np.zeros(p)\n",
" for tree in model.estimators_.ravel():\n",
" features = tree.tree_.feature\n",
" counts += np.bincount(features[features >= 0], minlength=p)\n",
" return counts\n",
"\n",
"\n",
"names = list(values)\n",
"boost_guide, boost_cart = np.zeros(len(names)), np.zeros(len(names))\n",
"rng = np.random.default_rng(100)\n",
"for _ in range(20):\n",
" X = pd.DataFrame({\n",
" name: rng.normal(size=n) if k is None else rng.integers(0, k, n).astype(float)\n",
" for name, k in values.items()\n",
" })\n",
" y = rng.normal(size=n)\n",
" boost_guide += guide_split_counts(GuideBoostingRegressor(n_estimators=50).fit(X, y), len(names))\n",
" boost_cart += sklearn_split_counts(\n",
" GradientBoostingRegressor(n_estimators=50, max_depth=3, random_state=0).fit(X, y), len(names))\n",
"pd.DataFrame({\"GUIDE boosting\": boost_guide / boost_guide.sum(),\n",
" \"scikit-learn boosting\": boost_cart / boost_cart.sum()}, index=names).round(3)"
]
},
{
"cell_type": "markdown",
"id": "e68daec0",
"metadata": {},
"source": [
"scikit-learn's boosted trees split ten times as often on the continuous noise variable as on the binary one (43% of all splits against 4%), so split-count and gain importance would rank pure noise by its number of distinct values. GUIDE boosting spreads its splits much more evenly. The binary variable's share is somewhat lower because it can split only once along any branch, not because it is selected less often."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.12"
}
},
"nbformat": 4,
"nbformat_minor": 5
}