Machine learning is a way to build software that learns patterns from example data instead of following only hand-written rules. In supervised learning, we give a model input-output pairs (for example, patient features and a diagnosis) so it can predict outputs for new rows. Often the input data is tabular, with rows and columns. The columns can be numerical, categorical, or a mix of both. The outputs can be class labels (classification) or numerical values (regression).
For these kinds of problems, commonly used libraries are scikit-learn and gradient-boosted tree libraries like XGBoost, LightGBM, and CatBoost. These libraries are very good at working with tabular data, but they require training a new model for each dataset. Training takes time, and getting the best performance often requires hyperparameter tuning.
In this blog post, we take a look at a different approach with TabPFN, a foundation model for tabular data from Prior Labs. TabPFN is short for Tabular Prior-Data Fitted Network. It provides classification and regression without training new neural-network weights for every dataset. Instead, you provide the labeled rows as context and ask the model to predict the unlabeled rows.
TabPFN ¶
TabPFN-3 was trained entirely on synthetic tabular tasks. It learned how to solve new classification and regression problems through in-context learning. You find the TabPFN-3 model weights on Hugging Face.
TabPFN-3 supports classification and regression with numerical and categorical columns, and it can handle missing values. It supports up to 160 classes. The model overview documents the following row/feature limits: 1,000,000 rows with 200 features, 100,000 rows with 2,000 features, or 1,000 rows with 20,000 features. The local model is intended for structured tabular data, not unstructured data such as images or raw text.
The TabPFN-3 weights are licensed for non-commercial and non-production use. Commercial or production use requires a separate license from Prior Labs. Check the TabPFN-3 license before using the model or its outputs.
Setup ¶
Prior Labs provides a Python package that wraps the model in a scikit-learn-style API. You can install it with:
uv add tabpfn
Like the classical machine learning libraries, we instantiate a classifier or
regressor, call fit with the training rows, and then call predict or
predict_proba on new rows. The difference is that TabPFN's fit call does not
train new model weights. Depending on the configured
fit_mode,
it prepares and optionally caches the context used for inference.
The first run downloads the model weights and opens a browser where you can sign in to Prior Labs and accept the model license. The authentication token and model weights are cached locally. After that, the model runs locally on your machine.
A GPU is recommended, but the small examples in this post also run on a CPU. The package limits the default TabPFN-3 model to 5,000 samples on a CPU unless you explicitly override this guardrail.
For an offline setup, you can follow the
manual download instructions in the FAQ section
and either set TABPFN_MODEL_CACHE_DIR or pass the downloaded checkpoint with
model_path.
Iris dataset ¶
The first example we take a look at is the classic Iris dataset, and we compare TabPFN with XGBoost. The task is to classify iris flowers into three species based on four numerical features: petal length, petal width, sepal length, and sepal width. The dataset has 150 rows, with 50 samples for each species: setosa, versicolor, and virginica.
The program loads the dataset with scikit-learn:
def load_iris_frame() -> tuple[pd.DataFrame, pd.Series, list[str]]:
iris = load_iris(as_frame=True)
return iris.data, iris.target, list(iris.target_names)
It then reserves 20% of the dataset for validation with scikit-learn's
train_test_split
function:
x, y, target_names = load_iris_frame()
x_train, x_validation, y_train, y_validation = train_test_split(
x,
y,
test_size=args.validation_size,
random_state=args.seed,
stratify=y,
)
With XGBoost, we create a classifier, train it on the training rows with fit,
and use the fitted model to predict the validation rows with predict:
def run_xgboost(
x_train: pd.DataFrame,
x_validation: pd.DataFrame,
y_train: pd.Series,
seed: int,
) -> tuple[object, float]:
classifier = XGBClassifier(
objective="multi:softprob",
num_class=3,
n_estimators=100,
max_depth=3,
learning_rate=0.1,
subsample=1.0,
colsample_bytree=1.0,
eval_metric="mlogloss",
random_state=seed,
n_jobs=1,
)
start = time.perf_counter()
classifier.fit(x_train, y_train)
predictions = classifier.predict(x_validation)
elapsed = time.perf_counter() - start
return predictions, elapsed
In a production environment, training and prediction usually run in two separate programs. We train the XGBoost model, save it to disk, and load it later to predict new rows. For this demo, we keep everything in one program.
The TabPFN code looks similar, but it does not train new model weights. Instead, it uses the training rows as context and predicts the validation rows. The programming model is intentionally similar to scikit-learn, so we can easily switch between the libraries.
def run_tabpfn(
x_train: pd.DataFrame,
x_validation: pd.DataFrame,
y_train: pd.Series,
seed: int,
n_estimators: int,
model_path: Path | None,
device: str,
) -> tuple[object, float]:
from tabpfn import TabPFNClassifier
start = time.perf_counter()
if model_path is None:
print("Loading the default TabPFN-3 classifier checkpoint...", flush=True)
else:
print(f"Loading the TabPFN checkpoint from {model_path}...", flush=True)
classifier = TabPFNClassifier(
model_path=model_path if model_path is not None else "auto",
n_estimators=n_estimators,
device=device,
random_state=seed,
)
with windows_browser_auth_workaround():
classifier.fit(x_train, y_train)
probabilities = classifier.predict_proba(x_validation)
if not np.isfinite(probabilities).all():
raise RuntimeError("TabPFN returned non-finite probabilities.")
predictions = classifier.classes_[np.argmax(probabilities, axis=1)]
elapsed = time.perf_counter() - start
return predictions, elapsed
Unlike XGBoost, TabPFN does not require us to configure a training process. We
can still configure its inference behavior. device selects the inference
device, and
n_estimators
sets the number of ensemble members. Each member is another forward pass with
slightly different input data, not a separately trained model. TabPFN aggregates
the predictions from all members. The package default is eight members, while
this demo requests one for a faster CPU run.
random_state
controls these randomized variants. A fixed integer makes the choices
repeatable in the same environment; None uses fresh randomness. Results can
still differ slightly across hardware.
The demo application uses uv to install dependencies and run the program.
uv sync
uv run python iris_tabpfn_vs_xgboost.py
One run on my laptop with CPU produced:
TabPFN accuracy: 1.0000
TabPFN elapsed: 1.96s
XGBoost accuracy: 0.9333
XGBoost elapsed: 0.07s
Out of the box, TabPFN is more accurate than XGBoost in this example, but it is slower. We can tune XGBoost with a hyperparameter optimization tool such as Optuna, which may improve the result.
Also note that these elapsed times are not a direct performance comparison. The
TabPFN timer includes checkpoint loading, fit, and prediction. The XGBoost
timer includes training and prediction, but not object construction. The times
also depend on the hardware and whether the model is already cached.
The validation split is still important with TabPFN. We need rows that were not provided as context to evaluate how well the model works on unseen data. After model selection and evaluation, we can use all available labeled rows as context when predicting genuinely new rows.
I ran these demos on a CPU. Prior Labs recommends a GPU for larger datasets.
Breast cancer classification ¶
The next demo is another classification problem and a classic introductory machine learning example: scikit-learn's Breast Cancer Wisconsin dataset. It has 569 rows, 30 numerical features, and two classes: malignant and benign.
Like with the Iris dataset, we use scikit-learn to load the data:
def load_breast_cancer_frame() -> tuple[pd.DataFrame, pd.Series, list[str]]:
dataset = load_breast_cancer(as_frame=True)
return dataset.data, dataset.target, list(dataset.target_names)
breast_cancer_tabpfn_vs_xgboost.py
The rest of the program is similar to the Iris demo. We split the dataset into 80% training and 20% validation data, train XGBoost, and use both models to predict the validation rows.
uv run python breast_cancer_tabpfn_vs_xgboost.py
One run on my hardware produced:
TabPFN accuracy: 0.9825
TabPFN elapsed: 5.74s
XGBoost accuracy: 0.9474
XGBoost elapsed: 0.17s
We see the same pattern as with the Iris dataset. Out of the box, TabPFN is more
accurate with this split, but it is slower on my CPU. The XGBoost time includes
training and prediction. The TabPFN time includes loading its checkpoint,
preparing the context with fit, and prediction.
Diabetes regression ¶
The last example we take a look at is a regression problem. The task is to predict a measure of disease progression one year after baseline for diabetes patients. Like with the previous two datasets, we use scikit-learn's diabetes dataset:
def load_diabetes_frame() -> tuple[pd.DataFrame, pd.Series]:
dataset = load_diabetes(as_frame=True)
return dataset.data, dataset.target
The rest of the program is similar to the previous demos. We split the data,
train the XGBoost model with fit, and obtain predictions from both models with
predict.
Run the demo application with:
uv run python diabetes_tabpfn_vs_xgboost.py
For this regression problem, we evaluate the models with
mean absolute error (MAE),
the average absolute difference between predicted and actual values, and
R²,
the coefficient of determination. A lower MAE is better. The best possible R²
is 1.0, and R² can be negative. One run on my CPU produced:
TabPFN MAE: 41.36
TabPFN R^2: 0.4885
TabPFN elapsed: 3.66s
XGBoost MAE: 46.48
XGBoost R^2: 0.3780
XGBoost elapsed: 0.17s
Out of the box, TabPFN has a lower MAE and a higher R² than XGBoost with this split. Hyperparameter optimization may improve the XGBoost result.
TabPFN estimators ¶
The demos in this blog post use one ensemble member for a faster CPU run. In the
TabPFN API, n_estimators is the number of forward passes whose predictions are
aggregated. Each pass receives a slightly different version of the input data.
The package default is eight, and the examples explicitly reduce it to one.
With auto_scale_n_estimators=True, TabPFN automatically increases the number
of members when a table has more features than one member can see. It chooses
the smallest number that lets every feature appear in at least one member. The
three datasets in this blog post are narrow enough that a single member can see
all features. This only guarantees feature coverage; it does not mean that one
member produces the same accuracy as a larger ensemble.
You can increase the number manually. For example, to use five members, run:
uv run python breast_cancer_tabpfn_vs_xgboost.py --tabpfn-estimators 5
Each additional member adds another forward pass and increases inference time and memory use. A larger ensemble may improve average predictive performance, but it does not guarantee a better result on every individual split.
TabPFN caps automatic scaling at 32 members, as documented in the changelog. We can still request a higher number manually.
Wrapping up ¶
We have seen a different approach to machine learning with tabular data. Instead of training a new model for every dataset, we can use a foundation model like TabPFN to predict new rows based on existing labeled rows. TabPFN is a pre-trained model that uses in-context learning to solve new classification and regression problems.
Before using TabPFN in a production system, evaluate it and compare it against tuned models from libraries like XGBoost, LightGBM, or CatBoost. Compare predictive metrics, latency, throughput, and memory use. Note that commercial or production use of TabPFN-3 weights or outputs requires a separate commercial license from Prior Labs.
Another model worth mentioning in this context is Google Research's TabFM. Like TabPFN, it is a foundation model for tabular classification and regression that uses in-context learning.