I wanted to know how often the documented way of converting a tree model to ONNX produces a file that is not the same model. So I built the pairs the way the skl2onnx and onnxmltools documentation shows, with default settings and float32 input, and checked every one with an equivalence checker.
The setup: 7 datasets (iris, wine, breast cancer, digits, diabetes, a synthetic regression, and a synthetic table of integer counts), 6 model types, 3 sizes each (10 trees at depth 3, 50 at depth 6, 100 at depth 8). That is 126 pairs. Each pair was checked twice, once on finite inputs only and once on all inputs including NaN.
| Model type | Finite inputs | Including NaN |
|---|---|---|
| XGBoost via onnxmltools | 21 equivalent | 21 equivalent |
| scikit-learn GradientBoosting | 21 equivalent | 21 equivalent |
| scikit-learn RandomForest | 21 equivalent | 21 not equivalent |
| scikit-learn ExtraTrees | 21 equivalent | 21 not equivalent |
| scikit-learn DecisionTree | 21 equivalent | 21 not equivalent |
| LightGBM via onnxmltools | 6 equivalent, 15 not | 6 equivalent, 15 not |
On real data, none of this shows. I ran every original and every ONNX file on the rows of its own dataset, 19,908 rows in total across the 126 pairs, and not one output differed. A test set check would have passed all 126.
LightGBM: one float32 value goes the wrong way
LightGBM stores split thresholds as float64. The ONNX file stores them as float32. When a threshold is a round decimal, the rounding can move it past the one input value that equals that decimal.
In the breast cancer model with 10 trees, the first split on feature 23 is x <= 868.2000000000002 in LightGBM and x <= 868.2000122070312 in the ONNX file. The input 868.2, as float32, is exactly 868.2000122070312. LightGBM sends it right (it is larger than the threshold), the ONNX file sends it left (it is equal to the threshold).
I took the input the checker reported and ran it through the real LightGBM and the real onnxruntime. LightGBM predicts class 0 (raw score -0.853). The ONNX file predicts class 1 (probability 0.786). The raw scores differ by 2.156, which is what the report said was the largest possible difference. For 9 of the 15 pairs the report says the predicted class can change.
The affected band is tiny: one float32 value per threshold. That is why 19,908 real rows never hit it. It is also why it is a real problem when your features are decimals like prices or measurements, because round decimals are exactly the values people type in.
The 6 LightGBM pairs that were fine are the digits and the integer count datasets, where every threshold is exactly representable.
scikit-learn trees and forests: NaN goes a different way
scikit-learn accepts NaN at prediction time for trees and forests, even if the model never saw a NaN in training. According to its documentation, such samples are sent to the child with the most training samples. The converted ONNX file sends NaN a fixed way instead.
Breast cancer, one decision tree, one NaN in the input: scikit-learn returns probabilities [0.0, 1.0], the ONNX file returns [1.0, 0.0]. The labels are opposite. On the wine random forest, an input with four NaN values gets class 2 from scikit-learn and class 0 from the ONNX file. For 45 of the 63 scikit-learn pairs the report says the predicted class can change.
This only matters if NaN can reach your model, for example from a failed join or a missing sensor value. If you impute upstream, it does not. But then nothing is checking that for you either.
What was fine
All 42 XGBoost and scikit-learn GradientBoosting pairs were equivalent, with and without NaN. I also tried an old XGBoost report (an issue about base_score in onnxruntime) and it no longer reproduces with current versions. Those converters did the right thing in these settings.
Update (2026-10-07)
The scikit-learn NaN difference in the table applies to the released skl2onnx versions: I used 1.16.0 for the audit and rechecked with 1.20.0, the latest on PyPI. The skl2onnx maintainers already fixed it on the main branch (pull request 1254, learned NaN routing), and no PyPI release contains the fix yet. With skl2onnx built from main, I rechecked 11 scikit-learn trees and forests (iris, wine, breast cancer, diabetes) and the checker proves all of them equivalent over every float32 input, with NaN allowed.
The LightGBM finding still reproduces on the current onnxmltools main branch. I filed it here: https://github.com/onnx/onnxmltools/issues/806
One correction to the commands below: audit.py is in the repository, not in the pip package, so clone the repository first (git clone https://github.com/Leafparity/leafparity) and run python leafparity/examples/audit/audit.py --quick.
Limits
These are models I trained on built-in datasets following the documentation. They are not third-party production models, so this says how the documented recipes behave, not how common the problem is in the wild. The checker supports numeric XGBoost, LightGBM and scikit-learn tree models and ONNX TreeEnsemble operators up to opset 3. Categorical splits and HistGradientBoosting are not supported, so they were not part of this.
Run it yourself
The script is in the repository. A quick run takes under a minute, the full run about 12 minutes.
pip install leafparity[all] skl2onnx onnxmltools
python audit.py --quick
Script and README: https://github.com/Leafparity/leafparity/tree/main/examples/audit
I am the author of the checker, Leafparity (open source, Apache 2.0, https://leafparity.com). If you have a model and its ONNX file where a check like this finds something, or finds nothing, I would like to hear about it.
Top comments (0)