# Neutron Runner

The **eIQ Neutron Runner** is a validation and execution tool for TensorFlow Lite models that integrates Neutron's
golden kernels into the TFLite runtime. It enables bit-exact validation of models compiled with the Neutron converter by
executing them using reference implementations of Neutron operators.

## Overview

Neutron Runner operates in two modes:

- **Neutron Mode**: Executes Neutron-converted models using bit-exact golden kernel implementations integrated into the
  TFLite runtime
- **Vanilla Mode**: Executes standard TFLite models using the original TFLite operators

This tool is primarily used for:

- **Bit-exactness validation**: Verify that Neutron-converted models produce identical outputs to the golden reference
- **Functional testing**: Validate model correctness before deployment
- **Debugging**: Compare Neutron operator behavior against reference implementations

### Operator Fallback Behavior

When running in Neutron Mode, if an operator is not supported by the Neutron golden kernels, the tool automatically
falls back to the standard TFLite Vanilla implementation for that operator. This ensures that models containing a mix of
supported and unsupported operators can still be executed.

After execution completes, any unsupported kernels that fell back to TFLite Vanilla will be displayed in the console
output, allowing you to identify which operators were not executed using Neutron golden kernels.

## Prerequisites

### For Neutron Mode

- Model must be converted using **neutron-converter** using `--keep-graphs` option in order to keep all the subgraphs.
- Using above option, the model will contain `NeutronGraph` and `NeutronOperator`.
- Supported targets: `imx95` and `imxrt700`

### For Vanilla Mode

- Any standard TensorFlow Lite model (FP32, FP16, INT8)
- No conversion required

## Usage

The **neutron-runner** has the following syntax:

```bash
neutron-runner --input <model_path> --dataset <dataset_path> [OPTIONS]
```

## Required Parameters

- `--input` - Path to the input TensorFlow Lite model file. Can be either:
    - A Neutron-converted model (for Neutron mode)
    - A standard TFLite model (for Vanilla mode)

- `--dataset` - Path to the input dataset. Supports multiple formats:
    - **Single directory**: All input tensors' data files in one directory
    - **Single file**: One binary file for single-input models
    - **Multiple paths**: Comma-separated directories or files (one per input tensor)

  See [Dataset Format](#dataset-format) section for detailed information.

## Optional Parameters

### Output Configuration

- `--output-results` - Path for the output results file in binary format. If not provided, results will be written to
  the same directory as the input model with the suffix `_results.bin` added to the filename.

### Model Configuration

- `--graph-name` - For models with multiple subgraphs, specifies the name of the graph to execute. This parameter is
  ignored for single-graph models.

- `--graph-index` - For models with multiple subgraphs, specifies the index (0-based) of the graph to execute. Use `-1`
  to auto-select the main graph. Default is `-1`.

### Runtime Configuration

- `--use-neutron-runtime` - Selects the execution mode:
    - `true` (default): Use Neutron Runtime with golden kernel implementations. **Requires a Neutron-converted model.**
      Unsupported operators automatically fall back to TFLite Vanilla.
    - `false`: Use vanilla TFLite runtime with original operators. Works with any standard TFLite model.

- `--target` - Specifies the Neutron hardware target platform. Options: `imx95`, `imxrt700`. Only used when
  `--use-neutron-runtime=true`. Default is `imx95`.

### Debugging Options

- `--verbose` - Enables detailed console output including:
    - Model structure information
    - Dataset mode detection
    - Operator execution details
    - Input/output tensor shapes and data types
    - Execution timing information

  Options: `true`, `false`. Default is `false`.

## Execution Modes

### Neutron Mode (Bit-Exact Validation)

This mode executes Neutron-converted models using golden kernel implementations integrated into the TFLite runtime.

**Use case**: Validate that your Neutron-converted model produces bit-exact results compared to the reference
implementation.

```bash
neutron-runner \
  --input model_neutron.tflite \
  --dataset ./test_inputs \
  --use-neutron-runtime=true \
  --target imx95 \
  --verbose=true
```

**Requirements**:

- Model must be converted with `neutron-converter`
- Model contains `NeutronOperator` custom operators
- Golden kernels are available for the target platform

**Note**: If any operator is not supported by Neutron golden kernels, it will automatically fall back to the TFLite
Vanilla implementation. Unsupported kernels will be listed at the end of execution.

### Vanilla Mode (Standard TFLite Execution)

This mode executes standard TFLite models using the original TFLite operator implementations.

**Use case**: Run inference on standard TFLite models for comparison or baseline validation.

```bash
neutron-runner \
  --input model_standard.tflite \
  --dataset ./test_inputs \
  --use-neutron-runtime=false \
  --verbose=true
```

**Requirements**:

- Any valid TensorFlow Lite model
- No conversion needed

## Dataset Format

The `--dataset` parameter supports multiple input formats to accommodate different model architectures and testing
scenarios.

### Dataset Modes

| Mode                         | Format                                          | Use Case                                        |
|------------------------------|-------------------------------------------------|-------------------------------------------------|
| Single directory             | `--dataset <directory>`                         | Single-input models, multiple datasets          |
| Single file                  | `--dataset <file.bin>`                          | Single-input models, single dataset             |
| Multiple paths (auto-mapped) | `--dataset <path1>,<path2>,...`                 | Multi-input models                              |
| Explicit mapping             | `--dataset <name1>,<path1>,<name2>,<path2>,...` | Multi-input models with explicit tensor mapping |

### Single Directory Input

When `--dataset` points to a directory, each file represents one inference run. **This mode only works for models with a
single input tensor.**

```
test_inputs/
├── input_0.bin    # First inference
├── input_1.bin    # Second inference
└── input_2.bin    # Third inference
```

```bash
neutron-runner --input model.tflite --dataset ./test_inputs
```

### Single File Input

When `--dataset` points to a single file, it should contain raw binary tensor data. **This mode only works for models
with a single input tensor.**

```bash
# For a model with input shape [1, 224, 224, 3] and float32 type
# File size should be: 1 * 224 * 224 * 3 * 4 bytes = 602,112 bytes
neutron-runner --input model.tflite --dataset input.bin
```

### Multiple Paths (Auto-Mapped)

For models with multiple input tensors, you can provide comma-separated paths (directories or files). The paths are
automatically mapped to input tensors in order.

#### Multiple Directories

Each directory corresponds to one input tensor. Files within each directory represent different inference runs:

```
input_tensor_0/
├── sample_0.bin
├── sample_1.bin
└── sample_2.bin

input_tensor_1/
├── sample_0.bin
├── sample_1.bin
└── sample_2.bin
```

```bash
neutron-runner --input model.tflite --dataset ./input_tensor_0,./input_tensor_1
```

#### Multiple Files

For a single inference with multiple input tensors, provide comma-separated file paths:

```bash
neutron-runner --input model.tflite --dataset input0.bin,input1.bin,input2.bin
```

**Note**: The number of paths must match the number of model input tensors.

### Explicit Input Mapping

For precise control over which data maps to which input tensor, use explicit mapping with tensor names:

```bash
neutron-runner --input model.tflite --dataset input_1,./dir1,input_2,./dir2
```

Format: `<input_name1>,<path1>,<input_name2>,<path2>,...`

This is useful when:

- Input tensor order is not obvious
- You want to ensure correct mapping regardless of model input order
- Debugging multi-input models

### Binary Data Format

Binary files should contain raw tensor data in the model's expected input format:

| Data Type | Bytes per Element |
|-----------|-------------------|
| float32   | 4                 |
| float16   | 2                 |
| int8      | 1                 |
| uint8     | 1                 |
| int32     | 4                 |

**File size calculation**: `batch_size × height × width × channels × bytes_per_element`

Example for a `[1, 224, 224, 3]` float32 input:

```
1 × 224 × 224 × 3 × 4 = 602,112 bytes
```

## Output Format

Results are written in binary format to the specified output file (or `<input_model>_results.bin` by default).

### Output Modes

- **Directory input mode**: Results are saved to individual files with pattern `<output_path>_<index>.bin`
- **Single file input mode**: All results are saved to a single file

The output file contains:

- Raw output tensor data in the model's output format
- For multiple outputs, data is concatenated in the order of model output tensors
- For multiple inferences (directory input), results are saved to separate files

## Examples

### Example 1: Validate Neutron-Converted Model

```bash
neutron-runner \
  --input mobilenet_v2_neutron.tflite \
  --dataset ./validation_images \
  --output-results ./results/mobilenet_validation.bin \
  --use-neutron-runtime=true \
  --graph-name "inference_graph" \
  --target imx95 \
  --verbose=true
```

This command:

- Executes a Neutron-converted MobileNetV2 model
- Uses golden Neutron kernels for bit-exact validation
- Processes all images in the validation directory
- Saves results to individual files: `mobilenet_validation_0.bin`, `mobilenet_validation_1.bin`, etc.
- Prints detailed execution information
- Displays any unsupported kernels at the end
- Choose the main graph for execution `inference_graph`.

### Example 2: Multi-Input Model with Multiple Directories

```bash
neutron-runner \
  --input multi_input_model.tflite \
  --dataset ./image_inputs,./metadata_inputs \
  --use-neutron-runtime=true \
  --target imx95
```

This command:

- Executes a model with two input tensors
- Maps `./image_inputs` to the first input tensor
- Maps `./metadata_inputs` to the second input tensor
- Processes all matching files across directories

### Example 3: Explicit Input Tensor Mapping

```bash
neutron-runner \
  --input complex_model.tflite \
  --dataset image_input,./images,mask_input,./masks \
  --use-neutron-runtime=true \
  --target imx95 \
  --verbose=true
```

This command:

- Explicitly maps directories to named input tensors
- `./images` directory → `image_input` tensor
- `./masks` directory → `mask_input` tensor

## Typical Workflow

### Bit-Exactness Validation Workflow

1. **Convert model** using neutron-converter:
   ```bash
   neutron-converter --input model.tflite --output model_neutron.tflite --target imx95 --keep-graphs
   ```

2. **Run with Neutron runtime** (golden kernels):
   ```bash
   neutron-runner --input model_neutron.tflite --dataset inputs/ --use-neutron-runtime=true --target imx95
   ```

3. **Run with vanilla runtime** (reference):
   ```bash
   neutron-runner --input model.tflite --dataset inputs/ --use-neutron-runtime=false
   ```

4. **Compare outputs** to validate bit-exactness:
   ```bash
   diff model_neutron_results_0.bin model_results_0.bin
   ```

5. **Review unsupported kernels** (displayed after Neutron mode execution) to understand which operators fell back to
   TFLite Vanilla.

## Supported Kernels

The following operators have Neutron golden kernel implementations:

| Operator          | Description                      |
|-------------------|----------------------------------|
| Conv2D            | All Conv2D flavours              |
| DepthwiseConv2D   | All DepthwiseConv2D flavours     |
| GlobalAveragePool | Global average pooling operation |
| BatchNorm         | Batch normalization              |
| Add               | Element-wise addition            |

**Note**: Operators not in this list will automatically fall back to TFLite Vanilla implementation when running in
Neutron mode. The list of unsupported kernels encountered during execution will be displayed at the end of the run.

## Supported Targets

| Target     | Description                |
|------------|----------------------------|
| `imx95`    | i.MX 95 platform (default) |
| `imxrt700` | i.MX RT700 platform        |

## Unsupported Kernel Reporting

When running in Neutron mode (`--use-neutron-runtime=true`), the tool will:

1. Execute supported operators using Neutron golden kernels
2. Automatically fall back to TFLite Vanilla for unsupported operators
3. Display a summary of unsupported kernels at the end of execution

Example output:

```
Running Neutron model...
Neutron Runner completed successfully!

Unsupported kernels (executed with TFLite Vanilla):
  - Softmax
  - Reshape
  - Squeeze
```

This information helps you understand which parts of your model are not using Neutron golden kernels for bit-exact
validation.

## Troubleshooting

### Error: "Model does not contain Neutron operators"

**Cause**: Using `--use-neutron-runtime=true` with a non-converted model.

**Solution**: Either convert the model with neutron-converter or use `--use-neutron-runtime=false`.

### Error: "Input data size does not match expected tensor size"

**Cause**: Input file size doesn't match expected tensor dimensions.

**Solution**: Verify input file size matches: `batch_size × height × width × channels × sizeof(datatype)` (applicable
for most vision models)

### Error: "Number of input files does not match number of model inputs"

**Cause**: When using comma-separated paths, the number of paths doesn't match the model's input count.

**Solution**: Ensure you provide exactly one path per input tensor. Use `--verbose=true` to see the number of model
inputs.

### Error: "Single file mode requires a model with exactly 1 input"

**Cause**: Attempting to use a single file dataset with a multi-input model.

**Solution**: Use comma-separated files or directories for multi-input models:

```bash
neutron-runner --input model.tflite --dataset input0.bin,input1.bin
```

### Error: "Graph not found"

**Cause**: Specified `--graph-name` or `--graph-index` doesn't exist in the model.

**Solution**: Use `--verbose=true` to list available graphs, or use `-1` for auto-selection.

### Error: "Dataset path does not exist"

**Cause**: One or more paths in the dataset parameter don't exist.

**Solution**: Verify all paths exist and are accessible. Check for typos in path names.

## Notes

- **Neutron Runner is a validation tool**, not a deployment runtime. It uses golden kernel implementations for bit-exact
  verification.
- For production deployment, use the optimized Neutron hardware runtime.
- Output results are in raw binary format for precise numerical comparison.
- When using directory mode, results are saved to individual files for easier comparison.
- Unsupported operators automatically fall back to TFLite Vanilla, ensuring model execution completes even with partial
  Neutron support.
- Always check the unsupported kernel report to understand which operators were not validated with Neutron golden
  kernels.