IEEE LaTeX template

Download main.tex Copy

% !BIB program = bibtex
\documentclass[conference]{IEEEtran}

\usepackage{cite}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{booktabs}

\begin{document}

\title{Title of the paper}

% One \IEEEauthorblockN and \IEEEauthorblockA per author, separated by \and.
\author{
\IEEEauthorblockN{Your Name}
\IEEEauthorblockA{\textit{Department of Something} \\
\textit{University of Somewhere} \\
Somewhere, Country \\
[email protected]}
\and
\IEEEauthorblockN{Second Author}
\IEEEauthorblockA{\textit{Institute of Something Else} \\
\textit{Other University} \\
Othertown, Country \\
[email protected]}
}

\maketitle

\begin{abstract}
% One paragraph: the problem, what you did, what you found.
Neural networks small enough for a microcontroller are usually run to
completion on every input, which wastes energy on inputs that a much smaller
model would have classified correctly. We evaluate three scheduling policies
that decide, per input, how much of the network to run: a fixed early exit, a
confidence threshold, and a budget aware controller. On a keyword spotting
workload running on a 64~MHz device, the budget aware controller holds accuracy
within 0.4 points of the full network while cutting median energy per inference
by 38 percent. We report the accuracy and energy of each policy, and the
conditions under which the simplest of them is the right choice.
\end{abstract}

\begin{IEEEkeywords}
embedded systems, neural networks, energy efficiency, scheduling
\end{IEEEkeywords}

\section{Introduction}
% What the problem is, why it matters, what this paper adds.
Battery powered devices increasingly run a neural network locally rather than
sending data to a server, and the energy that network costs is now a design
constraint rather than a rounding error. A keyword spotting model on a
microcontroller may run several times a second for years on a single cell, so a
saving of a third in energy per inference is the difference between a product
and a prototype.

Most of that energy is spent on inputs that were never difficult. Silence, a
steady background hum, and a voice speaking words outside the vocabulary are
all resolved by the first few layers, yet the whole network runs anyway. Early
exit architectures address this by attaching classifiers to intermediate layers
and stopping as soon as one of them is confident enough \cite{teerapittayanon2016}.
The saving is real, but the exit rule is usually a constant, which leaves the
device slow when the input distribution shifts and wasteful when it does not.

This paper compares three exit policies on the same network and the same
hardware. Section~\ref{sec:related} reviews the alternatives,
Section~\ref{sec:method} defines the three policies, and
Section~\ref{sec:evaluation} reports accuracy and measured energy.

\section{Related Work}
\label{sec:related}
Early exit networks trade depth for confidence and have been studied mostly on
server hardware, where the metric of interest is latency rather than energy
\cite{teerapittayanon2016}. Quantization and pruning reduce the cost of every
inference instead of the cost of easy ones, and the two approaches compose
\cite{banner2019}. Closest to this work is the dynamic inference controller of
Ahmed and Novak, which selects a network width per input but assumes a
processor with frequency scaling that our target does not have \cite{ahmed2021}.

\section{Method}
\label{sec:method}
The network has four convolutional blocks with a classifier attached after
blocks two, three and four. Write $p_k(x)$ for the class distribution predicted
at exit $k$ and $c_k(x)$ for its confidence, the largest probability in that
distribution. The device stops at the first exit whose confidence clears a
threshold,
\begin{equation}
\label{eq:exit}
k^{*}(x) = \min \{\, k : c_k(x) \geq \tau_k \,\},
\end{equation}
and falls through to the last exit when no threshold is met. The three policies
differ only in how the thresholds $\tau_k$ are chosen.

The fixed policy sets every $\tau_k$ to the same constant, tuned once on a
validation set. The confidence policy sets each $\tau_k$ separately so that the
accuracy at every exit matches the accuracy of the full network on the
validation set. The budget aware controller adjusts the thresholds at run time
to hold a target of inferences per joule, raising them when the energy budget
is underspent and lowering them when it is not.

\section{Evaluation}
\label{sec:evaluation}
All three policies were compiled for the same 64~MHz Cortex class device with
512~kB of flash, and energy was measured with a shunt resistor on the supply
rail over 20{,}000 inferences. Table~\ref{tab:policies} reports the results
against the full network as a baseline.

\begin{table}[t]
\caption{Accuracy and measured energy per inference, keyword spotting workload}
\label{tab:policies}
\centering
\begin{tabular}{lrr}
\toprule
Policy & Accuracy (\%) & Energy (mJ) \\
\midrule
Full network       & 94.6 & 1.81 \\
Fixed exit         & 91.2 & 0.94 \\
Confidence         & 94.0 & 1.32 \\
Budget aware       & 94.2 & 1.12 \\
\bottomrule
\end{tabular}
\end{table}

The fixed policy is the cheapest and the least accurate, and its accuracy is
the one that moves when the recording conditions change: on the noisiest tenth
of the test set it falls to 84.1 percent, while the other two lose less than a
point. The budget aware controller costs 19 percent more energy than the fixed
policy and recovers almost all of the accuracy, because it spends the saved
energy on the inputs that need it.

\section{Conclusion}
Choosing how much of a network to run per input is worth more on a
microcontroller than choosing how to shrink the network, provided the exit rule
can adapt. Where the input distribution is known and stable, a single tuned
threshold captures most of the benefit at none of the complexity.

\bibliographystyle{IEEEtran}
\bibliography{references}

\end{document}

Download references.bib Copy

% One entry per reference. The key is what you pass to \cite.
@inproceedings{teerapittayanon2016,
  author    = {Surat Teerapittayanon and Bradley McDanel and H. T. Kung},
  title     = {Branchy networks: fast inference via early exiting},
  booktitle = {Proceedings of the International Conference on Pattern Recognition},
  pages     = {2464-2469},
  year      = {2016}
}

@article{banner2019,
  author  = {Ron Banner and Yury Nahshan and Daniel Soudry},
  title   = {Post training quantization of convolutional networks},
  journal = {IEEE Transactions on Embedded Computing},
  volume  = {18},
  number  = {4},
  pages   = {112-125},
  year    = {2019}
}

@article{ahmed2021,
  author  = {Sana Ahmed and Petr Novak},
  title   = {A dynamic inference controller for battery powered devices},
  journal = {IEEE Embedded Systems Letters},
  volume  = {13},
  number  = {2},
  pages   = {57-60},
  year    = {2021}
}