TF Lite Profiler

This tool profiles a FLOAT (FP16/FP32) model by computing the dynamic range [min, max] for each FLOAT tensor and writing the information to a file called profile with the following format:

<tensor_index_1>,<min_value_1>,<max_value_1>,<freq_tensor_1_bin_1>,<freq_tensor_1_bin_2>,..,<freq_tensor_1_bin_n>,
<tensor_index_2>,<min_value_2>,<max_value_2>,<freq_tensor_2_bin_1>,<freq_tensor_2_bin_2>,..,<freq_tensor_2_bin_n>,
...

where:

  • <tensor*_index> is the tensor index used as tensor identifier (the location attribute that you can view in the Netron visualizer).

  • <tensor*_min> is the minimum FLOAT value recorded for this tensor across all dataset samples.

  • <tensor*_max> is the maximum FLOAT value recorded for this tensor across all dataset samples.

  • <tensor*_histogram_bin*> is the frequency count for each histogram bin of the tensor’s value distribution, where the first * represents the tensor index and the second * represents the bin number.

To be noted that the profile file only contains the information related to variable tensors (activations). The constant tensors (weights) are NOT profiled because the data statistics is fixed (independent on dataset) and already part of the model.

Prerequisites

  • Model must be a FLOAT TensorFlow Lite model. To obtain a tflite floating point model there are multiple conversion strategies:

    1. Use TensorFlow’s model conversion API, see example script in scripts/tensorflow_conversion.py. More details here: https://www.tensorflow.org/api_docs/python/tf/lite/TFLiteConverter

    2. Use AI Edge Torch from Pytorch to TFLite conversion: see more details here: https://github.com/google-ai-edge/ai-edge-torch

  • Dataset inputs must be in floating point binary format, having the input range required by the model training (e.g., for image classification there are models trained with input 0..1 or models trained on input normalized to Imagenet which is approximately -2..2).

  • A representative subset of training samples should be used for profiling/calibration purposes, with typically 200-1000 samples being sufficient to capture the dynamic tensor value distributions. In order to validate the profiling/quantization results, it is recommended to use a validation dataset separate from the calibration dataset to measure accuracy.

  • The representative dataset should be provided in a floating-point binary format with input ranges matching the original model’s training configuration. An example script is provided in ./scripts/serialize_image.py.

  • The representative dataset should provide diverse coverage of the model’s expected outputs, with samples distributed across different output categories to ensure comprehensive profiling of activation ranges.

Usage

The tflite-profiler has the following syntax:

tflite-profiler --input <input_model_path> --dataset <input1>,<dataset1>,<input2>,<dataset2>,... [OPTIONS]

Required Parameters

  • --input - The path to the input FLOAT TensorFlowLite model.

  • --dataset - Input dataset description which provides the dataset for each graph input tensor in a comma-separated list with the form: <input1>,<dataset1>,<input2>,<dataset2>,....

    • Inputs input<N> are identified by the tensor name or the tensor index (location) in the model.

    • If input<N> is not specified and only dataset<N> is present, the profiler will automatically assign the tensor name based on the order of the input datasets for models with multiple inputs.

    • Datasets dataset<N> are identified by a directory path containing all the files.

Optional Parameters

  • --output - The path for the output profile file in CSV format. If not provided, the profile file will be written in the same directory as the input model with the suffix _profile added to the filename.

  • --dataset - Input dataset description which provides the dataset for each graph input tensor in a comma-separated list with the form: <input1>,<dataset1>,<input2>,<dataset2>,....

    • Inputs input<N> are identified by the tensor name or the tensor index (location) in the model.

    • If input<N> is not specified and only dataset<N> is present, the profiler will automatically assign the tensor name based on the order of the input datasets for models with multiple inputs.

    • Datasets dataset<N> are identified by a directory path containing all the files.

  • --graph-name - For models with multiple graphs, this specifies the name of the graph to profile. This parameter is ignored for single-graph models.

  • --dump-instrumented-model - Flag to save the instrumented model. Default is false. When true, use --dump-instrumented-model-name to specify the output filename.

  • --dump-instrumented-model-name - Output filename for the instrumented model. Default is the input filename with ** _instrumented** suffix.

  • --histogram-num-bins - Number of bins used to calculate the histogram for each variable tensor. Default is 256.

Example

tflite-profiler --input model.tflite --dataset input_data_dir --output model_profile.csv