Software Stack

Neutron Software Stack has 4 components:

  • Neutron Converter: Is a pre-processing tool responsible for reading a standard TFLite model, determining which of its operators will be computed by Neutron, transforming those operators into Neutron-specific operators appropriate for the specific Neutron target and grouping a subgraph of Neutron-specific operators into a Neutron Graph. For each Neutron Graph, the Neutron Converter generates microcode and possibly some weight data. The output of the Neutron Converter is a TFLite model that contains custom Neutron Graph primitives.

  • Neutron Library: Contains a collection of operator implementations called kernels that are dispatched by the Neutron Firmware at runtime and a set of utilities that are used by the Neutron Converter offline to reorder weights, prepare quantization parameters, compress/encrypt weights, verify kernel support and others.

  • Neutron Driver: Provides the user application with the required abstraction and interface to run an inference for an NN model which was previously pre-processed using the Neutron Converter tool.

  • Neutron Firmware: The software component that directly controls the execution of the Neutron system.

Between these components we have the following dependency graph:

../../../_images/eiqNeutron_sw_stack.png

Figure 8. eIQ Neutron Software components

 

The reason why the Neutron Library contains two sub-libraries is because only that way provides a standalone self-contained functionality and is also testable as a single component.

Since the Neutron Library is integrated internally into the Neutron Converter and Neutron Firmware, from a packaging/release perspective, the final software deliverable will consist in 3 components:

  • Neutron Converter: Will be distributed as a pre-built binary executable (application) for Windows/Linux. Will also be distributed as a pre-built binary library with a public C++/Python API.

  • Neutron Driver: Will be distributed as a pre-built binary (library) plus a public API (header file).

  • Neutron Firmware: Will be distributed as a pre-built binary:

    • For eIQ Neutron the Firmware is distributed as a standalone library that is linked together with the Driver library in the application (since both Firmware and Driver run on the same core).

    • For eIQ Neutron-S the Firmware binary is distributed as a standalone binary that is executed by the RISC-V and which communicates with the Driver binary running on the application CPU. The Firmware image in this case is a bit more special because it also needs to support a paging mechanism required to only bring certain sections of the Firmware into the TCM and not load it entirely which might not be possible due to the TCM size constraints.