Skip to content

Integrations

SLEAP Integration (MAT + PoseKit)

SLEAP inference is executed from the SLEAP conda/mamba environment selected in the UI.

Why this env is separate

Do not rely on the main MAT environment for SLEAP export/runtime dependencies. In practice, the uv-based MAT install path does not guarantee the SLEAP runtime/export modules needed for all SLEAP backends.

Create a dedicated SLEAP environment and select it from:

  • MAT: Analyze Individuals -> Pose Extraction -> SLEAP env
  • PoseKit: Inference -> SLEAP -> Conda environment

To use ONNX/TensorRT SLEAP prediction inside PoseKit, also enable:

  • Inference -> SLEAP -> Allow experimental SLEAP runtimes

Environment Setup

mamba create -n sleap python=3.13 -y
conda activate sleap

Choose one install profile:

# CPU + ONNX CPU export/runtime support
pip install "sleap[nn,nn-export]"

# GPU + ONNX GPU export/runtime support
pip install "sleap[nn,nn-export-gpu]"

# GPU + TensorRT export/runtime support
pip install "sleap[nn,nn-export-gpu,nn-tensorrt]"

Compatibility Matrix

  • macOS (Apple Silicon):
  • Supported: SLEAP native (mps/cpu), ONNX CPU.
  • Not supported: TensorRT.
  • NVIDIA CUDA systems:
  • Supported: SLEAP native CUDA, ONNX GPU, TensorRT (with nn-tensorrt extras and system CUDA/TensorRT compatibility).
  • CPU-only systems:
  • Supported: SLEAP native CPU, ONNX CPU.

Verify SLEAP Integration

conda run -n sleap python -c "import importlib.util as u; print('sleap_nn', bool(u.find_spec('sleap_nn'))); print('onnx', bool(u.find_spec('onnx'))); print('onnxruntime', bool(u.find_spec('onnxruntime')))"
conda run -n sleap python -c "import torch, torchvision; print('torch', torch.__version__); print('torchvision', torchvision.__version__)"
conda run -n sleap sleap-nn export --help

If torchvision import fails with operator torchvision::nms does not exist, reinstall matching torch + torchvision builds from the same channel (CPU or CUDA-specific), then rerun the verification commands above.

Troubleshooting Install/Runtime Issues

If SLEAP preflight fails, use the commands below.

1) operator torchvision::nms does not exist

This usually means torch and torchvision are incompatible builds.

conda run -n sleap python -m pip uninstall -y torch torchvision torchaudio

# Reinstall one matching stack (pick one):
# CPU-only
conda run -n sleap python -m pip install --index-url https://download.pytorch.org/whl/cpu torch torchvision

# CUDA 13.0
conda run -n sleap python -m pip install --index-url https://download.pytorch.org/whl/cu130 torch torchvision

conda run -n sleap python -c "import torch, torchvision; print(torch.__version__, torchvision.__version__)"

2) libtorch_cuda.so: undefined symbol: ncclAlltoAll

This indicates a CUDA/NCCL binary mismatch.

conda run -n sleap python -m pip uninstall -y torch torchvision torchaudio nvidia-nccl-cu12 nvidia-nccl-cu13
conda run -n sleap python -m pip install --index-url https://download.pytorch.org/whl/cu130 torch torchvision
conda run -n sleap python -m pip install --upgrade nvidia-nccl-cu13
conda run -n sleap python -c "import torch, torchvision; print(torch.__version__, torchvision.__version__, torch.cuda.is_available())"

If this succeeds only when unsetting LD_LIBRARY_PATH, your shell is injecting incompatible CUDA/NCCL libraries:

env -u LD_LIBRARY_PATH conda run -n sleap python -c "import torch; print(torch.cuda.is_available())"

In that case, remove conflicting CUDA/NCCL paths from LD_LIBRARY_PATH for SLEAP runs.

For ONNX export smoke test:

conda run -n sleap sleap-nn export "/path/to/sleap_model_dir" --output "/tmp/sleap_export_test_onnx" --format onnx --device cpu --input-height 224 --input-width 224

For ONNX predictor smoke test:

conda run -n sleap python -c "from sleap_nn.export.predictors import load_exported_model as L; p=L('/tmp/sleap_export_test_onnx/model.onnx', runtime='onnx', providers=['CPUExecutionProvider']); print(type(p).__name__)"

See also:

X-AnyLabeling Integration

X-AnyLabeling is useful for adding labels to additional frames and correcting detection misses before rerunning MAT pipelines.

Install (git clone method)

git clone https://github.com/CVHub520/X-AnyLabeling.git
cd X-AnyLabeling
mamba create -n x-anylabeling python=3.11 -y
conda activate x-anylabeling
pip install -r requirements.txt
python app.py

If your local clone uses a different launch command, follow the repository instructions from the current branch README/get-started docs.

  • Export or collect frames where MAT missed animals or produced poor detections.
  • Open those images in X-AnyLabeling.

  • Use one of these tools to add/fix labels:

  • SAM2 for assisted segmentation

  • SAM3 for assisted segmentation
  • Manual segmentation tool (recommended for usability)

  • Manual OBB tool (use when oriented boxes are specifically needed)

  • Prefer segmentation masks over OBB for day-to-day correction speed and annotation quality.

  • Export labels in the format needed for your downstream MAT training or validation workflow.

Notes

  • Segmentation editing is generally more user-friendly than manual OBB editing for correction tasks.
  • OBB remains useful when your model or evaluation requires oriented boxes specifically.
  • Keep class names and label conventions consistent with your MAT dataset configuration.