{ "cells": [ { "cell_type": "markdown", "id": "a4d6ce2f", "metadata": {}, "source": [ "# Linear discriminant trees\n", "\n", "`GuideClassifier` can split on linear combinations of features instead of one feature at a time. With `node_model=\"lda\"` each node selects its own features by one-way ANOVA F tests, splits on its first linear discriminant coordinate, and classifies its rows with linear discriminant analysis (LDA). With `node_model=\"lda_splits\"` the nodes split the same way, but each cut minimizes the Gini impurity of its children and the leaves predict class proportions, as in an ordinary classification tree. Both reproduce GUIDE's options of the same names.\n", "\n", "This notebook compares them with a constant-leaf GUIDE tree, scikit-learn's `DecisionTreeClassifier` (CART) and its `LinearDiscriminantAnalysis` on two scikit-learn datasets and on a synthetic problem whose class boundary is oblique." ] }, { "cell_type": "code", "execution_count": 1, "id": "f0963d66", "metadata": { "execution": { "iopub.execute_input": "2026-11-02T22:20:07.868568Z", "iopub.status.busy": "2026-11-02T22:20:07.868153Z", "iopub.status.idle": "2026-11-02T22:20:09.108476Z", "shell.execute_reply": "2026-11-02T22:20:09.107933Z" } }, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "from sklearn.datasets import load_breast_cancer, load_wine\n", "from sklearn.discriminant_analysis import LinearDiscriminantAnalysis\n", "from sklearn.model_selection import StratifiedKFold, cross_validate\n", "from sklearn.tree import DecisionTreeClassifier\n", "\n", "from guide import GuideClassifier\n", "\n", "rng = np.random.default_rng(0)\n", "n = 600\n", "X_obl = pd.DataFrame(rng.normal(size=(n, 4)), columns=[\"x1\", \"x2\", \"x3\", \"x4\"])\n", "score = X_obl[\"x1\"] + X_obl[\"x2\"] - 0.5 * X_obl[\"x3\"]\n", "y_obl = pd.Series(np.select([score < -0.8, score < 0.8], [\"low\", \"mid\"], \"high\"))\n", "\n", "datasets = {\n", " \"wine\": load_wine(return_X_y=True, as_frame=True),\n", " \"breast cancer\": load_breast_cancer(return_X_y=True, as_frame=True),\n", " \"oblique (synthetic)\": (X_obl, y_obl),\n", "}" ] }, { "cell_type": "markdown", "id": "f52cab45", "metadata": {}, "source": [ "## Cross-validated comparison\n", "\n", "Ten-fold cross-validated accuracy and log loss (lower is better), with the mean number of leaves. The GUIDE trees are pruned by their own internal cross-validation. LDA nodes default to `min_samples_leaf=10`, GUIDE's default for this model." ] }, { "cell_type": "code", "execution_count": 2, "id": "c4e0f7bc", "metadata": { "execution": { "iopub.execute_input": "2026-11-02T22:20:09.109747Z", "iopub.status.busy": "2026-11-02T22:20:09.109630Z", "iopub.status.idle": "2026-11-02T22:20:14.879320Z", "shell.execute_reply": "2026-11-02T22:20:14.878860Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
accuracylog lossleaves
datasetmodel
wineGUIDE constant0.9050.4876.4
GUIDE lda0.9890.0201.0
GUIDE lda_splits0.9500.2323.5
CART0.8543.5807.4
LDA0.9890.019NaN
breast cancerGUIDE constant0.9420.2497.1
GUIDE lda0.9580.1491.7
GUIDE lda_splits0.9670.1582.1
CART0.9331.21114.3
LDA0.9560.130NaN
oblique (synthetic)GUIDE constant0.8400.69333.6
GUIDE lda0.9870.1035.7
GUIDE lda_splits0.9820.1103.0
CART0.8422.76841.3
LDA0.9450.172NaN
\n", "
" ], "text/plain": [ " accuracy log loss leaves\n", "dataset model \n", "wine GUIDE constant 0.905 0.487 6.4\n", " GUIDE lda 0.989 0.020 1.0\n", " GUIDE lda_splits 0.950 0.232 3.5\n", " CART 0.854 3.580 7.4\n", " LDA 0.989 0.019 NaN\n", "breast cancer GUIDE constant 0.942 0.249 7.1\n", " GUIDE lda 0.958 0.149 1.7\n", " GUIDE lda_splits 0.967 0.158 2.1\n", " CART 0.933 1.211 14.3\n", " LDA 0.956 0.130 NaN\n", "oblique (synthetic) GUIDE constant 0.840 0.693 33.6\n", " GUIDE lda 0.987 0.103 5.7\n", " GUIDE lda_splits 0.982 0.110 3.0\n", " CART 0.842 2.768 41.3\n", " LDA 0.945 0.172 NaN" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "models = {\n", " \"GUIDE constant\": GuideClassifier(),\n", " \"GUIDE lda\": GuideClassifier(node_model=\"lda\", min_samples_leaf=10),\n", " \"GUIDE lda_splits\": GuideClassifier(node_model=\"lda_splits\"),\n", " \"CART\": DecisionTreeClassifier(min_samples_leaf=5, random_state=0),\n", " \"LDA\": LinearDiscriminantAnalysis(),\n", "}\n", "\n", "\n", "def leaves(model):\n", " if hasattr(model, \"get_n_leaves\"):\n", " return model.get_n_leaves()\n", " return np.nan\n", "\n", "\n", "rows = []\n", "cv = StratifiedKFold(10, shuffle=True, random_state=0)\n", "for data_name, (X, y) in datasets.items():\n", " for name, model in models.items():\n", " result = cross_validate(model, X, y, cv=cv, scoring=[\"accuracy\", \"neg_log_loss\"],\n", " return_estimator=True)\n", " rows.append({\n", " \"dataset\": data_name,\n", " \"model\": name,\n", " \"accuracy\": result[\"test_accuracy\"].mean(),\n", " \"log loss\": -result[\"test_neg_log_loss\"].mean(),\n", " \"leaves\": np.mean([leaves(m) for m in result[\"estimator\"]]),\n", " })\n", "pd.DataFrame(rows).set_index([\"dataset\", \"model\"]).round(3)" ] }, { "cell_type": "markdown", "id": "6bd14b08", "metadata": {}, "source": [ "On wine the LDA tree never splits: one LDA leaf already separates the three cultivars, so it matches scikit-learn's LDA (0.989 accuracy, log loss about 0.02), well ahead of both single-variable trees.\n", "\n", "On breast cancer the two discriminant trees are the most accurate models (0.958 and 0.967), with one or two leaves on average against 7 for the constant GUIDE tree and 14 for CART. A single LDA still has the lowest log loss.\n", "\n", "On the synthetic problem the three classes are bands along one oblique direction. Trees that split on one feature at a time need more than 30 leaves to approximate the boundary and reach only 0.84 accuracy; `lda_splits` finds it with three leaves and 0.982 accuracy, and the LDA tree does slightly better still. A single LDA (0.945) is limited by its shared covariance and Gaussian class densities, which the bands violate." ] }, { "cell_type": "markdown", "id": "2eb54764", "metadata": {}, "source": [ "## Reading a discriminant tree\n", "\n", "Each split is a linear inequality in the node's selected features; missing values take the node's means." ] }, { "cell_type": "code", "execution_count": 3, "id": "7c44bba1", "metadata": { "execution": { "iopub.execute_input": "2026-11-02T22:20:14.880381Z", "iopub.status.busy": "2026-11-02T22:20:14.880306Z", "iopub.status.idle": "2026-11-02T22:20:14.887399Z", "shell.execute_reply": "2026-11-02T22:20:14.887081Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "guide-rs-0.1-experimental-reference28\n", "1 tree(s), 3 fitted leaves\n", "Tree 1\n", "node 0 (n=600): 2.932656e-2 * x1 + 2.752057e-2 * x2 + -1.291214e-2 * x3 <= -2.422617e-2; missing values take node means\n", " node 1 (n=175): class \"low\"; p=[0.00046866333887964507, 0.9988297282307265, 0.0007016084303937881]\n", " node 2 (n=425): 5.050062e-2 * x1 + 4.730610e-2 * x2 + -2.412141e-2 * x3 <= 4.089382e-2; missing values take node means\n", " node 3 (n=259): class \"mid\"; p=[0.015444015444015444, 0.011583011583011584, 0.972972972972973]\n", " node 4 (n=166): class \"high\"; p=[0.9927906968974066, 0.0004936217415418746, 0.006715681361051527]\n", "\n" ] } ], "source": [ "X, y = datasets[\"oblique (synthetic)\"]\n", "tree = GuideClassifier(node_model=\"lda_splits\", max_depth=2).fit(X, y)\n", "print(tree.to_text())" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.12" } }, "nbformat": 4, "nbformat_minor": 5 }