Model Deployment Quick Start Guide¶

This section outlines the workflow for deploying a model on eIQ Neutron or eIQ Neutron-S based SoC. The process includes model preparation on the host system, such as conversion to TensorFlow Lite (TF Lite), profiling, quantization, and compilation for execution on target platform, followed by deployment and benchmarking.

Overview¶

If the input model’s format is not TF Lite (e.g. ONNX), it must first be converted to TF Lite FP32.
- General model conversion and quantization flows are described in: Conversion & Quantization.
The model must be profiled and quantized using tflite-profiler and tflite-quantizer to produce a quantized TF Lite model:
- A profile file must be generated with the profiling tool tflite-profiler, using the model and a representative dataset.
- The model must then be quantized using profile‑guided quantization via the tflite-quantizer tool.
Running neutron-converter requires, at minimum:
- The target SoC
- The quantized TF Lite input model

Host-Side Tools Setup and Model Preparation¶

1. Install Required Tools¶

Install the ONNX-to-TFLite conversion tool and the eIQ Neutron SDK from the NXP eIQ repository.

pip install eiq-onnx2tflite --index-url https://eiq.nxp.com/repository/
pip install eiq-neutron-sdk --index-url https://eiq.nxp.com/repository/

2. Convert Model to TF Lite FP32¶

If model is already in TF Lite FP32 format, please skip this step.

Convert the ONNX model to TF Lite format using the onnx2tflite tool. The example below generates a FP32 TF Lite model.

./onnx2tflite -i <my_model>.onnx -o <my_model_fp32>.tflite -target fp32

3. Profile the TF Lite Model¶

Profile the FP32 TF Lite model using tflite-profiler and a representative dataset.

./tflite-profiler \
  --input <my_model_fp32>.tflite \
  --dataset <input_data_dir> \
  --output <my_model_fp32_profile>.csv

Ensure that <input_data_dir> contains meaningful input samples in .bin format. The data must match the model’s input dimensions and use float32 precision.

4. Quantize the Model¶

Use tflite-quantizer to generate a quantized TF Lite model based on the profiling results.

./tflite-quantizer \
  --input <my_model_fp32>.tflite \
  --profile <my_model_fp32_profile>.csv \
  --output <my_model_quantized>.tflite

5. Convert the Model for Neutron NPU Execution¶

Compile the quantized TF Lite model for execution on the Neutron NPU using the neutron-converter tool.

./neutron-converter \
  --input <my_model_quantized>.tflite \
  --output <my_model_converted>.tflite \
  --target imx95

eIQ Neutron‑S Target‑Side Deployment and Benchmarking¶

The following example targets i.MX95-EVK board.

6. Deploy the Model¶

Copy the converted TF Lite model to the i.MX95 target board.

7. Benchmark the Model¶

Use the TensorFlow Lite benchmark_model application to evaluate performance. This utility is part of TensorFlow package, part of the NXP Yocto BSP.

/usr/bin/tensorflow-lite-2.19.0/examples/benchmark_model \
  --graph=<my_model_converted>.tflite \
  --external_delegate_path=/usr/lib/libneutron_delegate.so \
  --enable_op_profiling=true \
  --num_threads=5

8. Model Analysis¶

The benchmark_model output reports latency metrics for each graph node. Operations executed via the NeutronDelegate correspond to workloads offloaded to eIQ Neutron-S, while all remaining nodes are executed on the CPU.

Number of nodes executed: 26
============================== Summary by node type ==============================
           [Node type]   [count]   [avg ms]   [avg %]    [cdf %]   [mem KB]  [times called]
       NeutronDelegate         1      2.882   87.919%    87.919%      0.000               1
        Slice (ND, X8)        10      0.175    5.339%    93.258%      0.000              10
         CONCATENATION         2      0.096    2.929%    96.187%      0.000               2
Unary Elementwise (NC)         2      0.078    2.379%    98.566%      0.000               2
         Copy (NC, X8)        11      0.047    1.434%   100.000%      0.000              11

Timings (microseconds): count=270 first=3372 curr=3303 min=3015 max=3537 avg=3290.54 std=76
Memory (bytes): count=0
26 nodes observed