\documentstyle[12pt,twoside,psfig,amssymb]{article}
\input{/u/rivest/class/handouts/defs-6045-PRAMs}
\def\implies{\mbox{ implies }}
\renewcommand{\mid}{:}
\def\Ic{\mathify{\{c,I\}}}

\newcommand{\remove}{1}{}
\newcommand{\abs}[1]{\left| #1\right|}
\newcommand{\floor}[1]{\left\lfloor #1 \right\rfloor}
\newcommand{\ceil}[1]{\left\lceil #1 \right\rceil}
\newcommand{\op}{{\bullet}}

% \newtheorem{definition}{Definition}[section]
% \newtheorem{theorem}{Theorem}[section]
% \newtheorem{proposition}[theorem]{Proposition}
% \newtheorem{lemma}[theorem]{Lemma}
% \newtheorem{corollary}[theorem]{Corollary}
% \newtheorem{claim}{Claim}[section]

\begin{document}

\handout{parallel}{Supplementary Notes on Parallel Computation}{May 1, 1992}

\centerline{Nancy Lynch and Ron Rivest}

%[[[Needs doing:
% 1. Check my step-counting hypotheses for my PCT statement, simulating
% PRAMS by TMs.  Any step-counting hypotheses needed for anything in
% section 5?
%
% 2. minor bracket-removal
%
% 3. Anything more on examples for circuits?
% ]]]

%[[[Next time through:
%
% 1. Though it's nonstandard, I'd like to see the circuit uniformity bound
% extended to $D(n)$ rather than $\log(n) space$, 
% and try to get the nicer correspondence all the way up the complexity
% hierarchy. 
%
% 2. More examples of circuits.
%
% 3. More details in the circuit construction.
% 
%]]]
\section{Introduction}

So far, we have studied machine models for different kinds of {\em
sequential} computation.  For example, we studied FA's and PDA's as
models for (structurally) restricted kinds of computation, and we
studied Turing machines, RAM's, and programming language models as
models of general computability.  We also studied complexity-bounded
TM's to begin our study of complexity theory.

This note describes models for {\em parallel} computation, in which
many things can be happening at the same time---that is, ``in
parallel''.  Parallel computers are now commercially available; you
may have heard of such machines from Thinking Machines, CRAY, or
Sequent, for example.  Parallel computation is also an area of active
theoretical research.  In this note we examine two formal models of
parallel computation: PRAMs and Boolean circuits, and introduce the
``complexity theory of parallel computation.''  We develop a theory of
what it means for a parallel computation to be efficient, and develop
some basis for believing that certain problems can {\em not} be solved
much more quickly on a parallel machine than on an ordinary sequential
machine---it seems that some computational problems may be
``inherently sequential.''

The reader should be warned that there are many models of parallel
computation, and that this note focuses on just two of them.
Excellent surveys of this subject, and of related aspects of
theoretical computer science, can be found in the {\em Handbook of
Theoretical Computer Science} (van Leeuwen, ed., Elsevier 1990),
especially the chapter by Karp and Ramachandran.

This note covers the following two models of parallel computation.
\begin{enumerate}
\item {\bf PRAMs (Parallel Random Access Machines).}
These serve as a good programming model for parallel algorithms.  It
is, from a programmer's point of view, an ``ideal'' parallel computer.
Once an algorithm has been designed for a PRAM, it is then often
translated into a more realistic model for actual execution.  The PRAM
model neglects certain costs that are important in practice---most
importantly, ``communication cost''.
\item {\bf Boolean circuits.}
Just as PRAMs are one of the most abstract models of parallel
computation, Boolean circuits are one of the most concrete and least
abstract.  Circuits are much simpler, more basic, and harder to
``program'' than PRAMs, but it is generally easier to prove things
about circuits than it is to prove them about PRAMs.
\end{enumerate}

Both models are used, both in practice and theory.  These models are also
closely related complexity-wise, as we shall see.

\section{The RAM and PRAM models of computation}

A PRAM is a parallel variant of the RAM model of computation.  We begin
with a quick review of the RAM model of computation, and then proceed
to define PRAMs.

\subsection{RAMs (Review)}

%[[[Somebody please put the other kind of formatting in for
%subsections, including the subsection numbers.  Now it is hard to
%distinguish subsection headings from headings of definitions, etc.]]]

A RAM computer consists of a finite-state control and an infinite
sequence of memory registers.  The name RAM refers to the fact that
the registers form a ``random-access memory.'' (Fortunately, the
access is not really ``random''---if you ask for the contents of register
$i$ you get the contents of register $i$ and not that of some random
register.  Perhaps it should have been ``arbitrary-access memory'';
but then AAM is unpronounceable...)

We present a specific version of a RAM model as a basis for our
PRAM extension.  Other versions of the RAM model also exist.

Let the sequence of registers be denoted $R_0, R_1, R_2, \ldots$.
Each register can hold one binary string of arbitrary (finite) length.
The binary string can alternatively be interpreted as a
(two's-complement) integer, so that ``0110'' denotes $6$ and ``110''
denotes $-2$, and $0$ and $\Lambda$ both denote $0$.  By convention,
the input and output of a computation are in register $R_0$.

The following unary operations are provided:
\begin{itemize}
\item $R_i \leftarrow \neg R_j$.  Bit-wise complementation.
\item $R_i \leftarrow -R_j$.  Integer negation.
\item $R_i \leftarrow R_ja$, where $a\in\{0,1\}$.  Concatenation on the
right with symbol $a$.  Similarly, $R_i \leftarrow aR_j$, where
$a\in\{0,1\}$ denotes concatenation on the left with symbol $a$.
\item $R_i \leftarrow \abs{R_j}$.  Compute the length of $R_j$.
\end{itemize}

The model also allows setting a register to a constant ($0$, $1$, or
$\Lambda$) or copying a value from one register into another.  That
is, instructions of the form ``$R_i\leftarrow 0$'' or ``$R_i
\leftarrow R_j$'' are allowed.

The RAM is also able to perform binary operations of the form 
\[
	R_i \gets R_j \circ R_k
\]
where $\circ$ is one of the following standard operators:
\begin{itemize}
\item $\vee$.  Bit-wise OR.  (If one operand is longer than the other, then
the shorter one is extended (by repeating its leftmost bit) until it has
the same length as the longer string.  This convention applies to all the
binary operators.)
\item $\wedge$. Bit-wise AND. 
\item $\oplus$. Bit-wise XOR.
\item $+$. Integer addition.
\item $-$. Integer subtraction.
\item $/$. Integer division; here $R_i \leftarrow \floor{R_j / R_k}$.
\item $\hbox{mod}$.  $R_i$ is the nonnegative remainder when dividing $R_j$ by
$R_k$.
\item $*$.  Truncated integer multiplication.  Multiply $R_j$ by $R_k$, and
return the low-order $\abs{R_j}+1$ bits of the result.  The high-order
bits are dropped. (By first extending $R_j$ on the left with
sufficiently many zeros, you can get the correct full result.)
\item $R_i \leftarrow  R_j >> R_k$.  Drop the $R_k$
least significant (rightmost) bits of $R_j$ (this is equivalent to a
``right shift''); if $R_k\le 0$ do nothing. Similarly, $R_i \leftarrow
R_j << R_k$ means: if $R_k\ge 0$, drop the $R_k$ most significant (leftmost)
bits of $R_j$ and if $R_k
\le 0$, do nothing.
\end{itemize}

Note that an operation never produces a result that is more than one
bit longer than its longest input.  In particular, full integer
multiplication is {\em not} a given primitive operation, nor is
concatenation of two registers.  These operations can double the
length of a register is one step, and increase it by a factor of $2^n$
in $n$ steps.  This capability leads to a number of anomalous
characteristics that we wish to avoid.  Full integer multiplication
and concatenation are still possible in the RAM model, however---they
just must be programmed as a sequence of simpler steps.

The operations above can be extended by the use of ``indirect addressing''.
An input operand or a destination register can be specified as 
$	R_{R_i}
$
to obtain the desired effect.  Indirect addressing is necessary in a 
RAM program to address more than a fixed number of registers, since
with no indirect addressing a RAM program can only give fixed constants
as register indices.

The RAM crashes if an attempt is made to access a nonexistent
register (i.e., 
indirect addressing is used with a negative index), or if
division by zero is attempted.  These are the only ways the RAM can
crash.

The ``finite-state control'' of the RAM is represented as a
{\em RAM program}, with the following kinds of control statements
allowed (in addition to assignment statements involving the operations
described above):

\begin{itemize}
\item {\bf goto} $n$.  Here $n$ is the label of some program statement.
\item {\bf If} $R_i > 0$ {\bf then} {\bf goto} $n$.
(Here we could instead use $\ge$, $\le$, $<$, $=$, or $\ne$ as the
comparison operator. We also allow two registers to be compared (as
integers).)
\item {\bf Halt.}
\end{itemize}
The {\em state} of the finite-state control is the value of the {\em
program counter} that indicates which statement is to be executed
next.  

Each program statement is assumed to execute in constant time,
independent of the size of the operands involved or the size of the
addresses of the registers utilized.

The only thing we did previously in this course with the RAM model was
to show that RAMs are a {\em feasible} model for computation: we
showed that RAMs are polynomially equivalent to Turing machines.

\noindent {\bf Exercise 1:} Write a RAM program that takes an $n$-bit
number $t$ in register $R_0$ as input, and computes as output in $R_0$
the next larger power of $2$: $R_0 = 2^{1+\floor{\log t}}$.  This program
should run in constant time.

\noindent {\bf Exercise 2:} Write a RAM program that takes an $n$-bit
number $t$ in register $R_0$ as input, and computes $\sqrt{t}$ in
register $R_0$ as output.  Your program should run in time 
$O(n)$ (that is, in time $O(\log t)$).  You may assume that $t$ is a 
perfect square.

\subsection{Definition of PRAMs}

A PRAM is a parallel version of the RAM model.  It consists of an
infinite number of finite state controllers (processors) $P_1, P_2,
\ldots$, again represented as RAM programs, and two kinds of
registers.  The ``global'' (shared) registers $R_0, R_1, R_2, \ldots$
are now shared by all the processors, whereas for each $i$, the
``local'' (private) registers $Q_{i, 0}, Q_{i,1}, Q_{i,2}, \ldots$ are
private to processor $i$.  Each processor can access only the global
registers and its own private registers.  All communication between the
processors must take place via the global registers.

The statements are the same as for a RAM, with the exception that a
statement may specify a local register be used instead of a global
register anywhere.  The assignments, test, branches, etc. are
otherwise the same as for a RAM.  In particular, we still have no
binary operations whose output is more than one bit longer than its
longest input.

We assume that all the programs are identical---each PRAM processor is
executing the same program.  Although these programs are started at
the same time, they do not necessarily execute the same sequence of
program statements.  In particular, we assume that the first local
register of processor $i$, $Q_{i, 0}$, is initialized to contain the
``name'' $i$.  In this way the processors can ``know'' which processor
they are, and can specialize their operations based on their index.

We use input and output conventions that are slightly different from
those we used earlier for RAMs.  For clarity of exposition, we will
assume the following input convention, which allows the input to
already be broken up into separate components:
\begin{enumerate}
\item
$R_0$ contains the (binary representation of the) number $n$, which
is defined to be the ``size'' of the input.
\item
$R_1, R_2,\ldots$ contain the components of the input. There can be at
most $n$ components and each component must be a bit-string of
length at most $n$.  (The actual number of components, and the size of
each component, may depend on $n$, but they cannot be larger than
$n$.)
\end{enumerate}
We may, for example, give an $n$-bit binary string by giving one bit
per register in $R_1$,$R_2$,\ldots,$R_n$, left to right.  Or, we could
give the entire string in $R_1$.
Note that $R_0$ must be at least $n$ if one of the input
components has length~$n$.  Note that we define the {\em size} of
the input to be $R_0 = n$, even though the actual
number of data bits provided may be as large as $n^2$.  Given that we
are primarily concerned in this note with polynomial or logarithmic
functions of the input size, a polynomial ``slop'' in the definition
of the input size doesn't really matter.

We assume that the output appears in a similar format: registers
$R_1$, $R_2$, \ldots contain the output components.  Register $R_0$
gives the ``size'' of the output, as defined above.  In the special
case where the machine is deciding membership in a language (computing
a characteristic function), register $R_0$ will contain $1$ on
output, and register $R_1$ contains the output bit ($0$ or $1$).

Now we describe how this machine executes.  The operation of a PRAM is
{\em synchronous}, operating in a series of {\em rounds} in which all
processors take exactly one step (unless they have halted or crashed,
see below).  For a single round, each processor performs the statement
(assignment, {\bf if}, {\bf goto}, or {\bf halt}) indicated
by its program counter.  Note that what we have said so far is not
sufficient to give a unique specification of what happens, because
there is the possibility of interference: different processors can be
reading and/or writing the same global registers, and the order in
which these reads and/or writes happens make a difference.  We have to
be more careful.

Each assignment statement involves performing some number of reads;
because of the fact that there are at most two arguments on the
right-hand side of the assignment, and because of the possibility of
indirect addressing on both the left- and right-hand side, there can
be at most 5 locations read.  Then there is exactly one write, plus a
change to the program counter.  Each {\bf if} statement consists of at
most four reads, plus a (conditional) change to the program counter.
We assume that all the reads, for all the processors, are done first;
it does not matter if several reads occur of the same register,
because they will all get the same value.  Then we assume that all the
writes occur, followed by all the changes to the program counters.

Now, note that there is still a problem with concurrent writes.  If
several processors write the same register at the same round, then we
need a convention to determine what happens; the literature on PRAMs
contains several different conventions for this problem.  Here, we
choose the simple option that concurrent writes to the same register
cause the PRAM to {\em crash}.  (In the literature, our machine would
be called a CREW---``concurrent read, exclusive write'' machine.
Other alternatives discussed in the literature allow concurrent
writes, with a convention that some processor, perhaps an arbitrary
one, or the one with lowest processor index, dominates and stores its
value in the register.  We won't discuss these alternatives any
further.)

The PRAM is defined to halt when all of its processors halt.  It crashes
if any of its processors crash, or if two processors attempt to write
into the same global register on the same round.

\noindent {\bf Exercise 3:} Write a PRAM program that takes an $n$-bit 
number in $R_1$ as input ($R_0=n$, and $R_2,\ldots,R_n$ are empty),
and distributes the individual bits to $R_1$,\ldots,$R_n$.  The
program should use $n$ processors and operate in constant time.

\subsection{Complexity Measures for PRAMs}

In this subsection, we define three complexity measures that will be
useful for PRAMs:  the {\em number of processors}, the {\em time}, and
the {\em global space}.

In principle, a PRAM has an infinite number of processors and an
infinite number of registers.
However, it makes sense, as for Turing machine tape, to count only the
amount of each of these resources that is {\em actually used} during a
computation.
In the case of registers, it is clear what this means.
However, in the case of processors, it is not so clear, because all
the (infinitely many) processors are assumed to begin taking steps 
at round 1.
In order to avoid counting processors that never perform any real
work,
we define a processor $P_i$ to be {\em active} in a computation if it
never attempts (successfully or unsuccessfully) to write any global register.
Otherwise (i.e., if all it does is read global registers,
read and/or write local registers, and perform control statements)
we say that $P_i$ is {\em inactive}.
This definition is the basis of our first parallel complexity measure,
defined below.

We now define the three PRAM complexity measures.  
They all depend on $n$, the size of the input.

\begin{enumerate}
\item Number of Processors

$P(n)$, the {\em number of processors}, defined to be the maximum,
over all inputs of length $n$, of the {\em largest index of a
processor that is active} in the computation on that input.  Note that
we are using the maximum index of any processor that is active, not
just the maximum number of active processors.  (This makes the theory
work out a little more simply.)

\item Time $T(n)$, the {\em time}, is defined to be the maximum, 
over all inputs of length $n$, of the number of rounds that take place
until all processors halt.

\item Space
$S_G(n)$, the {\em global space}, is defined to be the maximum, over all
inputs of length $n$, of the maximum address of a global register that
is ever used by the computation.  When needed, we denote the local space
used as $S_L(n)$---this is defined as the maximum, over all inputs of length
$n$, of the maximum address of any local register that is ever used in the
computation.

\end{enumerate}

Although a PRAM, according to our definition, has an infinite number of
processors, most PRAM programs in the literature are written in terms 
of only a finite number of processors.
We need a convention for writing PRAM programs that use a finite
number of processors.
We assume
that all PRAM processors first examine $R_0$ (the input size $n$) and
then compute (in their local storage) $P(n)$, the maximum index of a
processor that might be needed in this computation.  All processors
whose index (stored in $Q_{i,0}$) is greater than $P(n)$ immediately
halt.  The remaining active processors continue with the parallel
computation.

If a PRAM $M$ has a $P(n)$ processor bound according to our measure, 
for any fixed input size $n$, 
it is simple to design another machine $M'$ (a ``finite-processor
PRAM'') that has exactly $P(n)$ processors 
and that computes the same thing as $M$ on inputs of size $n$.  
Namely, $M'$ is the same as $M$ except that
only processors $1, \ldots, P(n)$ are included in $M'$.
Because the other processors never write anything in shared memory 
anyway, they cannot affect the results of the computation.

The reason we define a PRAM to have infinitely many processors is
so we can talk about the {\em same} PRAM
working on {\em arbitrary} size inputs, just as we do for Turing
machines and the other models studied so far in this course.

\section{Efficient parallel computation}

Now we give some examples of simple, but efficient, parallel
algorithms.  We then give a formal definition of the complexity
class ``$NC$'', which captures the intuitive notion of an ``efficient
parallel computation'', just as $P$ captures the notion of ``efficient
sequential computation''.

\subsection{Matrix Product}

Suppose we want to multiply two $m \times m$ matrices, $A$ and $B$, 
of Booleans to get another $m \times m$ matrix, $C$, so that
\begin{equation}
	C_{i,k} = \bigvee_{1\le j\le m} (A_{ij}\wedge B_{jk})\ ,
		\hbox{~for~} 1\le i,k\le m.
\label{eq:Cik}
\end{equation}
On a RAM this computation can be performed in $O(m^3)$ time.  For the
PRAM, we assume the inputs are represented with one bit per register.
Thus $n = 2m^2$, the $m^2$ bits of $A$ appear initially in
registers $R_1, R_2, \ldots, R_{m^2}$, and the $m^2$ bits of $B$
appear in registers $R_{m^2 + 1}, \ldots, R_{2 m^2}$.  (Each matrix in
row-major order.)  We want to compute the $m^2$ bits of $C$, and put
them into registers $R_1, R_2, \ldots, R_{m^2}$.

\begin{enumerate}
\item Some parallelization ($O(m^2)$ processors)

The number $P(n)$ of active processors will be exactly equal to $m^2$, the
number of result bits.  More specifically, the active processors will
be just $P_1, P_2, \ldots P_{m^2}$.
(The processors can determine if they are
active by comparing twice their index with the input size $n$.)

Processor $P_{(i-1) m + k}$ will be responsible for computing one of
the bits of $C$, namely, bit $C(i,k)$, according to
equation~(\ref{eq:Cik}).  This processor first computes $m$ (as
$\sqrt{n/2}$, using local storage and time $O(\log n)$---see
Exercise~2) and then computes $i$ and $k$.  It then extracts the bits
of the $i$th row of $A$ from locations $R_{(i-1) m + 1}$, $R_{(i-1) m
+ 2}$, \ldots, $R_{i m}$, and the $k$th column of $B$ from locations
$R_{m^2 + k}$, $R_{m^2 + m + k}$, \ldots, $R_{m^2 + (m-1) m + k}$.  It
does the successive Boolean multiplications and accumulates the sum (OR),
all in its local registers.  Finally, it writes the result to register
$R_{(i-l) m + k}$ and halts.  There are no write conflicts, and all
active processors terminate at the same time.

The number of active processors is $m^2$.  The time used is $O(m)$,
and the global space used is $O(m^2)$.  Therefore $P(n)$, $T(n)$ and
$S(n)$ are all $O(n)$.  By using $O(m^2)$ processors we have reduced
the computation time from $O(m^3)$ (in the sequential case) to $O(m)$
(in the parallel case).

\item More parallelization ($O(m^3)$ processors)

The following exercise is important!

\noindent{\bf Exercise 4:}
Write a PRAM program to count the number of $1$'s in a length-$n$
input vector of $0$'s and $1$'s, given in $R_1$,\ldots,$R_n$.  Your
program should use $O(n)$ processors and run in time $O(\log n)$.  Be
sure to handle properly the case that $n$ is not a power of two, and
to keep your processors sufficiently synchronized.  Prove that your
algorithm works.

(Strong hint: on ``step'' $k$, for $0\le k\le \ceil{\log n}$, processor $P_i$
(for all $i>2^k$) does $R_i \leftarrow R_i + R_{i-2^k}$.  The answer appears
in $R_n$; it can be copied into the output register $R_1$.)

The previous matrix multiplication algorithm can be sped up further
using more parallelism and the idea behind the Exercise~4.  (The
exercise involves a summation instead of an OR, but the idea is the
same.)  By having $O(m)$ processors working on each output bit, we can
obtain an algorithm that runs in time $T(m)=O(\log m)$, uses
$P(m)=O(m^3)$ processors and $S(m)=O(m^3)$ global space.  (If we
wanted to write these in terms of the input size $n = 2m^2$ instead of
in terms of $m$, we would write $T(n) = O(\log n)$, and $P(n) = S(n) =
O(n^{3/2})$.)

There are some programming details involved in making sure that each
processor can figure out what it is supposed to do at the right time.
The PRAM program can be written so that these details don't increase
the overall running time by more than a constant factor, therefore the
overall Boolean matrix multiplication takes time $O(\log m)$.  
The details are left as an exercise for the reader.

\noindent{\bf Exercise 5:} Sketch out the details omitted above.

\end{enumerate}

\subsection{Transitive Closure}

The input to the transitive closure problem
is an $m \times m$ Boolean matrix $A$.
The output, $B$, is the transitive closure of $A$, defined as
\[
	B = A^0 \vee A^1 \vee A^2 \vee A^3 \vee \ldots
\]
Here $A^0$ is the $m\times m$ identity matrix $I_m$, $A^k$ is the $k$-fold
product of $A$ with itself (using Boolean matrix multiplication), and
$\vee$ denotes component-wise ORing of matrices.  If we think of $A$
as the adjacency matrix for a directed graph, then $B$ has a $1$ in
position $(i,j)$ if and only if there is a path in the graph from node
$i$ to node $j$.  (If $i=j$, then one such path always exists---it is
the null path having no edges.)

This problem can be solved by repeated squaring, using the
multiplication algorithms described above: just initialize $B$ to $A
\vee I_m$, and then repeat $\ceil{\log m}$ times the assignment
$B\leftarrow B^2$ (Boolean matrix squaring).

For example, using the ``more parallelized'' Boolean matrix
multiplication above the transitive closure algorithm described uses
$O(m^3)$ processors, $O(\log^2 m)$ time, and $O(m^3)$ global space.
(Or, in terms of $n$, this is $P(n) = S(n) = O(n \sqrt{n})$, and $T(n)
= O(\log^2 n)$.)

The time bound can be reduced to $O(\log m)$ with a more integrated
implementation.  One processor is allocated to each triple $(i,j,k)$
of indices.  It is responsible for anding together locations $B(i,j)$
and $B(j,k)$, and if the result is $1$, getting it written into
location $B(i,k)$.  Because we have an ``exclusive write'' PRAM,
however, the process of writing the $1$ into $B(i,k)$ has to be
accomplished by a fan-in tree using the OR operation.  But instead of
doing the fan-in completely after each examination of the two
locations, the fan-in can be ``pipelined''.  Thus, the processors are
continually examining their two locations, and continually writing any
$1$ that is found into their location in the fan-in tree.  The fan-in
tree is operated in parallel with the matrix examination steps.

Thus, we can get $P(n) = O(m^3) = O(n \sqrt{n})$, 
$T(n) = O(\log m) = O(\log n)$ and $S(n) = O(m^3) = O(n \sqrt{n})$.

%[[[--- Lecture 2 might start here ---]]]

\subsection{Parallel Prefix}

The parallel prefix algorithm is a very useful primitive for many
PRAM algorithms.

The nature of the problem is similar to that of Exercise~4, except
that we are not only interested in the overall result (combining all
$n$ inputs), but also the partial result corresponding to each prefix
of the input array.

Here is an example.  Suppose we are given $n$ $0$-$1$-valued inputs $x_1$, $x_2$, \ldots, $x_n$ in
$R_1, \ldots, R_n$, and we want to achieve
\[
	R_i = \sum_{1 \leq j \leq i}  x_j, \hbox{~for~} 1 \leq i \leq n\ .
\]
That is, each $R_i$ contains the sum of the first $i$ of the $x_j$'s.

To obtain the general parallel prefix problem, we only need to let the
inputs $x_j$ be arbitrary, and to replace the operator $+$ by an
arbitrary associative operation, which we denote $\op$.  The operator
$\op$ could be an arbitrary associative operation, such as $\max$,
$+$, $\vee$, $\wedge$, addition modulo $5$, etc. We assume only that
each elementary $\op$ operation can be performed in constant time; otherwise
our time analyses would have to be appropriately modified.

A priori, it might seem that solving the parallel prefix example above
would require time $O(n)$, since calculating any result for $i$ seems
to require knowing the result for $i-1$.  However, there is an $O(\log
n)$ PRAM algorithm for this problem.

We claim that the solution given in the hint to Exercise~4 solves the
parallel prefix problem (for the operator $+$)!  Modifying the solution
to use $\op$ instead of $+$ solves the general parallel prefix problem.
For the proof, note that after the $k$th step, $R_i$ contains the value
\[
	R_{i-2^{k+1}+1} \op R_{i-2^{k+1}+2} \op \cdots \op R_i\ ,
\]
where any registers specified with indices less than $1$ are omitted
from this expression.  This can be proved by induction on $k$.

The parallel prefix algorithm works with $P(n) = S(n) = O(n)$ and
$T(n) = O(\log n)$.  The interesting thing to notice is the
logarithmic bound on the time; this indicates that the problem is
efficiently parallelizable.  The obvious sequential algorithm is
exponentially worse---parallelism gives us an exponential speed-up.

\subsection{Pointer Jumping}

One last technique for PRAM algorithm design is called ``pointer
jumping''.  Here we consider data structures where one register
``points to'' (contains the index of) another register.  In this
problem, we consider a simple linked list, represented as follows.
Let $R_1$,\ldots,$R_n$ denote the first $n$ PRAM global registers, and
let $S_1$,\ldots,$S_n$ denote another set of $n$ PRAM registers (say
$R_{n+1}$,\ldots,$R_{2n}$).  The pair $(R_i,S_i)$ may constitute one
element of the linked list: $R_i$ denotes the value stored at that
element, and $S_i$ points to the next (successor) element, or else
$S_i=0$ if there is no next element.  (So $0\le S_i \le n$.)  The pair
$(R_1,S_1)$ is the head of list.  The list contains at most $n$
elements, but may contain as few as one element.  The elements of the
list are not stored in any particular order.

Suppose now that we have an associative operation $\op$, and
we want $R_1$ to end up containing the result 
$R_1 \op R_{S_1} \op R_{S_{S_1}} \op R_{S_{S_{S_1}}} \ldots$, where
the values in this expression denote the {\em original} values in all
the registers.
(During the computation, it is OK to modify the
$R$ and $S$ registers.)

Of course, we could do a sequential search and computation down the
list (with one processor), which would take time $O(n)$.  
However, this problem can also be parallelized down to time $O (\log
n)$, as follows.  The process is very similar to the parallel prefix
algorithm, except that our sequence of values is stored in a list rather
than in a linear array.

We use $n$ processors $P_1$,\ldots,$P_n$.  Processor $P_i$ simply
repeats the following until $S_i=0$: If $S_i>0$, then set $R_i := R_i
\op R_{S_i}$, and set $S_i := S_{S_i}$.  

It is simple to prove that the above algorithm terminates with the
correct answer in $\ceil{\log n}$ steps.  After the $k$th step
($k=0,1,\ldots$) $R_i$ contains the $\op$ of the $2^{k+1}$ values in the
list beginning with $R_i$ (or to the end of the list, if there are
fewer than $2^{k+1}$ elements from $R_i$ on), and $S_i$ points to the
element that was $2^{k+1}$ steps after $R_i$ in the original list (or is
$0$, in the same exceptional case).

Note that this does more than just produce the correct answer in the
first cell---for each cell, it produces the answer for the suffix of
the list from that point on.

Thus, this takes time $O(\log n)$, with $O(n)$ processors and memory.

\subsection{The complexity class $NC$}

The examples given above illustrate a general theme: 
problems that have
efficient parallel solutions often have solutions that use poly($n$)
processors, poly($n$) memory, and poly($\log(n)$) time.  We formalize
this in the complexity class ``$NC$''.  
The class $NC$ plays the intuitive
role for parallel computation that $P$ plays for sequential computation:
a problem that is in $NC$ is deemed to be ``efficiently solvable'' (on a 
parallel machine).

(In case you are wondering,
the name $NC$ stands for ``Nick's Class'',
in honor of Nick Pippenger, the first person to define and study 
the properties of this class.  
Side note:  Nick studied computer science at MIT.)

We will start by defining $NC$ as a class of languages.
In order to do this, we must fix the input conventions for PRAMs that
decide membership in languages.
Earlier, the input conventions were left somewhat flexible, allowing,
for example, for an input string to either be split up, one symbol in
each register $R_1, R_2, \ldots$, or for it to be placed in its
entirety in register $R_1$.
Now we will be more specific:  for the rest of these notes, we will
assume that the input
string is split up, one bit in each register $R_1, R_2, \ldots$.
%[[[I was somehow thinking that the inputs and outputs were over
%arbitrary alphabets.  Looking back now, I see that the earlier defs
%just say ``bit strings''.  I am hoping that I have not used a more
%general alphabet anywhere else.]]]
The output, a single bit, will be placed in $R_1$.
As usual, the length $n$ of the input is placed in $R_0$.

%[[[Ron:  I'm still having trouble with this choice.
%It seems that too much is being allowed for unit cost.
%For example, we have a basic operation of integer addition, in unit
%cost.
%If the numbers can be of length $n$, there is a nontrivial algorithm
%needed to implement this addition in polylog parallel time.
%We never present such an algorithm to the students, so they may
%wonder about the reasonableness of the unit-cost assumption.
%Maybe we should give some kind of exercise about this?]]]
%[[[Ron comment:  Just admit ``slop'' of log n factor.]]]

\noindent {\bf Definition:}
Let $NC$ be the class of languages that are decidable by PRAMs  
for which there exists a polynomial $p$ such that
\[
	T(n) \leq p(\log n)\ ,
\]
\[
	P(n) \leq p(n)\ ,
\]
\[
	S_G(n) \leq p(n)\ , \hbox{~and}
\]
\[
	S_L(n) \leq p(n)\ .
\]

\begin{theorem}
Every problem in $NC$ is also in $P$. That is, $NC \subseteq P$.
\end{theorem}

\begin{proof}
A polynomial-time Turing Machine can easily simulate the operation of
the PRAM that solves the problem in NC.  It maintains a representation
of the complete state of the PRAM---the global registers, the local
registers and the processor states---on its tape, and simulates the
operation of the PRAM one step at a time, say by simulating each of
the processors in turn.  It is easy to check that the entire
simulation takes polynomial time.
\end{proof}

It is not known whether or not $NC = P$,
that is, whether every problem in $P$
can be efficiently parallelized.  
We will return to this question later.

The definition of $NC$ is usually extended to include function
computation problems, where the output 
string is produced, one bit in each
of the first few registers, $R_1, \ldots$.
With these conventions, we may say, for example, that Boolean matrix
multiplication and transitive closure are problems in $NC$.
The parallel prefix and pointer-jumping problems are also in $NC$, but
in order to show this, some recoding of our algorithms is necessary 
(to make them fit our fixed input and output
conventions, with one symbol per register).
Courses on parallel algorithms such as 6.848/18.435 and 6.849/18.436
would describe many other such examples.
The Karp-Ramachandran survey paper is another good source.

\section{PRAM Time is Bounded by Turing Machine Space}

Now that we have developed some techniques for programming PRAMs, we
can give a general and important simulation result.  
This result says that 
a language that can be recognized with a certain amount of space on
a Turing machine
can also be recognized in a similar amount of time on a PRAM.
In particular,
{\em a language that can be recognized with very small space on a Turing
machine can be recognized very quickly on a PRAM}.
More precisely, any Turing machine that that recognizes some language
and that works in space $S(n)\geq\log n$ can be simulated by a PRAM
that runs in time polynomial in $S(n)$ and uses a number of processors
that is at most $2^{c S(n)}$ for some constant $c>1$.  The simulation
preserves the language that is decided.

\begin{theorem}
\label{PCT-TMs-to-PRAMs}
{\bf [Efficient PRAM simulation of small-space TM]}
Let $S$ be a step-counting function\footnote{
     For the purpose of this section of the course, we redefine a {\em
     step-counting function} $S$ so that its input $n$ and output
     $S(n)$ are both represented in binary.  The time to compute
     $S(n)$ is still required to be bounded by $S(n)$, as before.}, 
where $S(n) \geq\log(n+2)$.
Let $M$ be a deterministic 
Turing machine that works in space $S(n)$.
Then there is a PRAM $M'$ that decides the same language as $M$ 
for which the number of processors $P(n)$, the global space $S_{G}(n)$ and the local space $S_L(n)$ are all
at most $2^{c S(n)}$ for some constant $c>1$
and for which the time $T(n)$ is $O(S(n))$.
\end{theorem}

\begin{proof}
The construction is based on pointer-jumping.  We assume in this proof
that the input, output, and work tape alphabets of $M$ are the binary
alphabet $\{0,1\}$.  The modifications required to handle other
alphabets are minor and are omitted here.

Consider the computation of the Turing machine $M$ on some particular
(binary) input $x_1 x_2 \ldots x_n$ of length $n$.  The computation
proceeds to a decision in space $S(n)$, writes a single bit on the
output tape, and halts.  Recall that when discussing space complexity,
we always use a Turing machine with separate read-only input, write-only
output, and read/write work tapes, and that the space bound only
applies to the work tape.

We define a {\em configuration} of the Turing machine 
to consist of the state of
the finite control, the contents of the work tape, the work tape head
position, the output tape contents (at most a single bit) and the
input head position (an integer between $0$ and $n+1$).  Each such
configuration can be represented in a bit-string of length 
\[
	\ceil{\log(Q)}+S(n)+\ceil{\log(S(n))}+2+\log(n+2) \le c S(n)
\]
for some $c>1$, since $S(n)\ge\log (n+2)$.  (Here $Q$ denotes the set
of states of the TM.)  There are thus at most $2^{c S(n)}$
configurations.  Also, the computation time cannot be larger than the
number of configurations.

In the PRAM simulation, each configuration will be represented
explicitly as a bit string in one of the PRAM global registers.  The
various components (state, work tape contents, head positions, etc.)
can be represented in fixed positions within the string representing
the configuration.  The shift and mask (AND, OR, etc) instructions can
be used to access any desired portion of the configuration in constant
time.

\noindent {\bf Exercise 6:} Illustrate how a TM configuration can be
stored within a register.  Show that the {\em successor} function can
be computed in {\em constant} time by a single-processor
PRAM program.  (The successor of
a TM configuration $C$ is the next configuration the TM would go to.  If the
configuration $C$ is a halting configuration, or if the TM crashes from $C$
when trying to make the transition to the next configuration, then the
successor of $C$ is defined to be $C$.)  You may assume that various 
required constants (field widths, etc.), have been precomputed.
Assume also that the TM input $x_1$, $x_2$, \ldots, $x_n$ is stored
in registers $R_1$, $R_2$, \ldots, of the PRAM.  (Note that the
the input tape contents is {\em not} part of a configuration.)

In the simulation of the TM by the PRAM, we assume 
as described above that the number $n$
is given in $R_0$, and that
registers $R_1, \ldots, R_n$ contain the successive bits $x_1, \ldots,
x_n$ of the input $x$.

The first step in the simulation by the PRAM is to compute $n$ and
$S(n)$.  Since $S$ is a step-counting function, it can be evaluated in
time $O(S(n))$ on a TM, and can thus be evaluated in time $O(S(n))$ by
any processor of the PRAM.
(Each processor can use the input registers as if they were tape
squares!)

The second step in the simulation is for the PRAM to compute $c S(n)$
and any necessary precomputed constants for the TM simulation
described above.  Any processor with index greater than $2^{c S(n)}$
halts; the remaining processors stay active. (Note that $2^{c S(n)}$
can be computed in time $O(S(n))$. 
The input values stored in $R_1$, $R_2$, \ldots, $R_n$ are
then copied to registers 
$R_{2^{c S(n)+1}}$,
$R_{2^{c S(n) + 1} +1}$, 
\ldots, $R_{2^{c S(n) + 1} + n - 1}$ for ``safekeeping'' and later access.

The third step is for the PRAM to compute in $R_0,\ldots,R_{2^{c S(n)}-1}$
each of the possible TM configurations.  This simply means that
processor $P_i$ stores $i$ in register $R_i$, for $0\le i\le 2^{c S(n)}$.
(It doesn't matter if some of these are ``illegal'' configurations somehow.)

Let $S_0, S_1, \ldots, S_{2^{c S(n)}-1}$ denote the $2^{c S(n)}$
global registers
beginning with $R_{2^{c S(n)}}$.  These will be the ``link'' or
``successor'' pointers.  Our problem will be modeled as a linked-list
problem, similar to the one we studied with the pointer-jumping
technique.  Think of the successive configurations in this computation
as a linear list of strings.  For each configuration except the final
(halting) configuration, there is a unique successor configuration.
It's unique because the computation is deterministic.

The fourth step is to compute all of the successor links: processor $P_i$
computes the successor to the configuration $R_i$ and stores it in
$S_i$.  This can be done in constant time, by Exercise~6.
(Note that the processors may be simultaneously accessing bits in
the copy of the input saved in $R_{2^{c S(n) +1}}$, etc.  The is OK, since
our PRAM is a ``concurrent read'' PRAM.)

What we have now is not exactly a singly-linked list.  Each
configuration has exactly one successor.  There can be more than one
configuration with the same successor, however.  (There could even be
cycles----but these correspond to unreachable configurations since the
TM is known to halt for all inputs.)

The fifth step is to do a pointer-jumping-style algorithm.  The
operation $\op$ we use will be just the projection on the second
argument, so $c_1 \op c_2 = c_2$ for any configurations $c_1$ and
$c_2$.  We do the same algorithm discussed above, even though we don't
exactly have singly-linked lists.  However, the same invariants as
before hold, when we interpret the ``linked list'' mentioned in those
invariants as the list beginning at the designated register, following
the unique outgoing pointers.  Note that the length of the linked list
that starts from the initial configuration is at most $2^{c S(n)}$,
because of the time bound for the Turing machine.  So after $c S(n)$
phases, the cell that starts with the initial configuration will
contain the final configuration for input $x$.  It takes time
$O(S(n))$ to do this pointer-jumping.

It remains for the PRAM only to look in the cell that started with the
initial configuration to determine whether the final answer
is accept or reject. (Recall that the contents of the output tape are
part of the configuration.)
(Note that the index of the cell with the initial configuration is
a $(c S(n))$-bit quantity, and so can be constructed in time at most
$c S(n)$.)

Therefore the entire TM computation can be simulated by the PRAM in time
$O(S(n))$, as was to be shown.
The processor and space bounds are straightforward.
\end{proof}

Note that this simulation can also be extended to simulate function
computation rather than language recognition,
by treating a function computation as a collection of language
recognition problems, one per bit of the output.  
By simply allowing for enough processors to do all these language
problems in parallel, we could produce all the output bits in parallel.
This would say that function computation by a Turing machine in space 
$S(n)$ is simulable by a PRAM in time $O(S(n))$.
The number of
processors used is $2^{c S(n)}$ times the number of bits in the function
output.  The number of bits in the output is at most $2^{c S(n)}$, so the
total number of processors is still at most $2^{c S(n)}$ (for a constant
$c$ that may be slightly larger).

A corresponding result for nondeterministic TM's follows from
Savitch's theorem and the above result for deterministic TM's.

\begin{corollary}
Let $M$ be a nondeterministic 
Turing machine that works in space $S(n) \geq\log (n+2)$.
Then there is a PRAM $M'$ that decides the same language as $M$ 
for which the number of processors $P(n)$ and the global space 
$S_{G}(n)$ are both at most $2^{S^2(n)}$,
and for which the time $T(n)$ is polynomial in $O(S(n))$.
\end{corollary}

\begin{proof}
By Theorem \ref{PCT-TMs-to-PRAMs} and Savitch's theorem.

Alternatively, a direct proof for nondeterministic Turing machines
can be given, which is similar to that for 
Theorem~\ref{PCT-TMs-to-PRAMs}, but uses
a transitive closure algorithm instead of pointer jumping on the
connectivity graph for Turing machine configurations. (Details omitted.)
\end{proof}

%[[[--- Lecture 3 might start here. ---]]]

We shall see that, surprisingly, the converse to
Theorem~\ref{PCT-TMs-to-PRAMs} is also true: computations that can be
performed efficiently in parallel can also be performed in small
space.
More precisely, the following result holds:

\begin{theorem}
\label{PCT-PRAMs-to-TMs}
Let $T$, $P$,
$S_G$, and $S_L$
be step-counting functions, where
$T(n) \geq \log n$, $P(n)$, $S_G(n)$, $S_L(n) \geq n$, 
and $P(n)$, $S_G(n)$, $S_L(n) \leq 2^{p(T(n))}$, where $p$ is a polynomial.
%[[[Is this exactly what's needed?  I'm not exactly sure about the
%(binary) step-counting definition being exactly what we want here.]]]
Let $M$ be a PRAM that works in time $T(n)$, with number of processors
at most $P(n)$, global space at most $S_G(n)$ and 
local space at most $S_L(n)$.
Then there is a Turing machine $M'$ that decides the same language as
$M$
for which the space $S(n)$ is polynomial in $T(n)$.
\end{theorem}

Together, Theorems \ref{PCT-PRAMs-to-TMs} and \ref{PCT-TMs-to-PRAMs}
imply a close correspondence between the sequential space complexity
measure and the parallel time measure.
This equivalence is known as the {\em Parallel Computation Thesis}.

A direct proof can be given for Theorem \ref{PCT-PRAMs-to-TMs},
consisting of a rather complicated simulation of a PRAM by a Turing
machine.
The simulation is complicated because the Turing machine has to do two
difficult things at the same time:  
(1) keep track of the complicated PRAM memory management scheme, including
its indirect addressing, and
(2) simulate a PRAM computation involving PRAM configurations of size
that is {\em exponential} in $T(n)$ (because they are
polynomial in $P(n)$, $S_G(n)$ and $S_L(n)$), 
but only using space that is {\em polynomial} in $T(n)$.  This
involves keeping only a small portion of the computation in the Turing
machine memory at once.

The bookkeeping involved in doing both of these tasks at the same time
is so complicated that we will take some pains to separate them.
This separation involves defining a second model for parallel
computation, namely, Boolean circuits.
Then a simulation of PRAMs by Turing machines can be broken up into a
simulation of PRAMs by Boolean circuits (the proof of which involves
task (1) above), and
a simulation of Boolean circuits by Turing machines
(the proof of which involves task (2)).

\section{Circuits}

The other common model for parallel computation is the {\em Boolean
circuit}.  We shall see that circuits and PRAMs are capable of
simulating each other efficiently, so that the complexity classes
(such as $NC$) that we can define using PRAMs can also be defined using
circuits.   For some purposes circuits are easier to work with, however.
In particular, we shall use the circuit model to prove the remaining
half of the Parallel Computation Thesis, i.e., to prove that 
sequential space is bounded by parallel time.

\subsection{Definition of Circuits}


One major difference between circuits and PRAMs has to do with what is
called {\em uniformity}.  A PRAM program can accept inputs of any
size, while a Boolean circuit only accepts inputs of some fixed size
$n$.  After defining Boolean circuits, we make the notion uniform
with a definition of a {\em Boolean circuit family}.

{\noindent {\bf Definition:}} A {\em Boolean circuit $B$ with $n$ inputs
and $m$ outputs} is a finite sequence of at least 
$\max (m,n)$ {\em elements},
$e_1$, $e_2$, \ldots, $e_c$ 
(here $c$ is the {\em complexity} (or {\em size}) of the circuit) such that
\begin{itemize}
\item 
The first $n$ elements, $e_1, e_2, \ldots, e_n$, are the $n$ inputs.
\item
The last $m$ elements , $e_{c - m + 1}, \ldots, e_{c-1}, e_{c}$, 
are the $m$ outputs.
\item 
For $i>n$, element $e_i$ consists of:
	\begin{enumerate}
	\item a left input, which is some element $e_{j_i}$ with $j_i<i$,
	\item a right input, which is some element $e_{k_i}$ with $k_i<i$,
	\item a binary Boolean operator $\circ_i$, which is one of the
		sixteen possible Boolean operators with two Boolean inputs.
		(This can be defined with a four-entry truth table.)
	\end{enumerate}
The non-input elements are also called {\em gates}.
\end{itemize}

There are standard conventions for drawing Boolean circuits as directed
acyclic graphs, with the gates represented with icons
denoting the operators involved.  

Example:
%99999999999999999999999999999999999999
The circuit shown in figure~\ref{fig:circuit} computes 
$((e_1 \cdot e_2) + e_3) \oplus  \neg e_4$.

\begin{figure}[ht]
\centerline{\mbox{\psfig{figure=cuircuit.ps,height=1.5in}}}
\caption{Example of a circuit}
\label{fig:circuit}
\end{figure}



There are many variant definitions of a Boolean circuit in the
literature.  For example, some definitions allow only the binary
operators AND and OR and the unary operator NOT. 
Other definitions allow associative
operators (AND, OR, XOR) to have an arbitrary number of inputs.  Our
definition is nonetheless commonly used, and is, we find, the easiest
to work with.

Suppose we are given a Boolean circuit $B$ with $n$ inputs and $m$
outputs.
Then given a sequence of $n$ Boolean input values,
$x_1, x_2 \ldots, x_n$, 
one can evaluate circuit $B$ by evaluating its elements in order.
First, the elements $e_1$, \ldots, $e_n$ are assigned the values of the
corresponding inputs $x_1, \ldots, x_n$.  Then each element $e_i$,
for $i>n$, is evaluated in turn according to the formula:
\begin{equation}
	e_i = e_{j_i} \circ_i e_{k_i}\ .
\label{eq:gate}
\end{equation}
Here the names of the elements are used also to stand for their
values, a convention we shall use throughout. 
The output values computed by $B$ are the values of its
last $m$ elements.

By regarding the sequence of inputs $x_1, \ldots, x_n$ as a length $n$
binary string $x = x_1 \ldots x_n$ and similarly for the sequence of
outputs, 
a Boolean circuit $B$ with $n$ inputs and $m$ outputs 
can be regarded as computing a Boolean function
function $B: \{0,1\}^n \rightarrow \{0,1\}^m$.
In the special case where $m = 1$,
the circuit can be regarded as recognizing a language.

The above procedure for evaluating a circuit is described in a sequential
manner.  A circuit can also be evaluated in a more parallel manner, in which
an element is evaluated as soon as its inputs are ready.  
With this idea,
we can define the {\em evaluation time} $t(i)$ of an element $e_i$
recursively as follows: 
\begin{itemize}
\item $t(i) = 1$ for each input element, $1\le i\le n$,
\item $t(i) = 1+\max(t(j_i),t(k_i))$ for $i>n$.
\end{itemize}
In calling this the ``evaluation time'',
we are assuming that each Boolean operator can be evaluated in
unit time, once its inputs are ready.

The {\em depth} $d$ of a Boolean circuit is maximum evaluation time of
any of its elements:
\[
	d = \max \{ t(i) : 1\le i\le n\}\ .
\]

It is not difficult to see that any Boolean circuit of complexity $c$ and
depth $d$ can be simulated (evaluated) by a PRAM program with $c$
processors in time $d$, once an appropriate pre-computation has been
performed that makes the values $(j_i,k_i,\circ_i)$ available to
processor $i$.  We can assume that global register $R_i$ is used to
hold the value of element $e_i$, and that the values of the inputs
$x_1$,\ldots,$x_n$ are initially stored in $R_1$,\ldots,$R_n$.  On
each time step, each processor $P_i$, for $i>n$ reads the values of
its input elements $e_{j_i}$ and $e_{k_i}$ from $R_{j_i}$ and
$R_{k_i}$, computes the value of $e_i$ according to
equation~(\ref{eq:gate}) and writes this value into global memory
register $R_i$.  
After $d$ such parallel steps, the values ``settle
down'', and $R_{c-m+1}$,\ldots,$R_c$ then contain the desired $m$
Boolean output values.

The Boolean circuit model differs from the other models we
have considered in this course in that a Boolean circuit is 
designed to work only for inputs of a particular size $n$, rather than
for inputs of arbitrary size.
In order to relate this model to others (esp., Turing machines and PRAMs),
we need to define an augmented model that can handle inputs of
arbitrary sizes.
We do this by defining the notion of a {\em Boolean circuit family}.
A {\em Boolean circuit family} will be defined to be a family of Boolean
circuits $\{B_n\}$, where $B_n$ has $n$ inputs.  
We need to say a little more, however.
In order to permit other models to efficiently simulate Boolean
circuit families
(something that is done, for example, in Theorems \ref{circuits to
PRAMs} and \ref{PCT-Circuits-to-TMs}), 
we need to require that the individual members of
the family can be efficiently generated.

\noindent {\bf Definition:} A {\em Boolean circuit family} ${\cal B}$
is a set $\{ B_n : n \ge 0 \}$ of Boolean circuits, where $B_n$ has
$n$ inputs, and such that there exists a Turing Machine $M$ that can
produce a description of $B_n$ from $n$ in space $O(\log n)$.  

(Note: the description of $B_n$ consists of $C(n)$ (the
circuit size), $D(n)$ (its depth), $m(n)$ (the number of output bits), and
a triple $(j_i,k_i,\circ_i)$ for each $i$ such that $n<i\le c(n)$.)

Logspace computability of the circuit descriptions is a slightly
awkward condition, 
quite different in flavor from anything we have encountered
in any of the other computing models in this course.
It is, however, a standard restriction in the research literature on
Boolean circuits.
Unfortunately, the condition is not entirely innocuous, because it
implies a limitation on the sizes of the circuits in the family:
the circuits $B_n$ in a Boolean circuit family must have
sizes that are bounded by some polynomial in $n$, since they can be
generated by a machine that operates in logarithmic space.  

This limitation on circuit size means that the results in this section
will only apply to ``small complexity'' circuits.
In particular, we will not actually prove the result we stated in
Theorem \ref{PCT-PRAMs-to-TMs}, but only a ``small complexity''
special case.
While it is possible to prove the stronger results using circuits, 
this would require generalizing the standard uniformity notion to allow
larger space bounds (specifically, space $O(D(n))$)
for generating the circuits.
We leave this generalization to the reader (or to a later pass through
this course!).
Note that the small complexity case is the interesting one, because
we are mainly interested in {\em efficient} parallel computation.

We say that a Boolean circuit family ${\cal B}=\{B_n\}$ has complexity
(or size) $C(n)$ and depth $D(n)$ if $C_n$ has complexity $c = C(n)$
and depth $d=D(n)$ for all $n>0$.
A Boolean circuit family can be regarded as computing a function from
$\{0,1\}^*$ to $\{0,1\}^*$, by ``pasting together'' the functions
computed by the individual circuits in the family.
In the special case where $m(n) = 1$ for all $n$, the circuit family
recognizes a language.

\subsection{Equivalence Between Circuits and PRAMs}

We now show that Boolean circuit families and efficient PRAMs are
equivalent, by showing each can simulate the other.

\begin{theorem}
\label{circuits to PRAMs}
{\bf [Simulation of Circuit Families by PRAMs]} Let ${\cal B}=\{B_n\}$
be a Boolean circuit family with size $C(n)$ and depth $D(n)$, where
$D(n) \ge \log n$.  
Then there is a PRAM $M$ that decides the same language as ${\cal B}$,
for which the number of processors $P(n)$, the global space $S_G(n)$
and the local space $S_L(n)$ are all polynomial in $n$,
and for which the time $T(n)$ is $O(D(n))$.
\end{theorem}

%[[[Now that we have changed the output convention back again, I think
%this simulation now works for function computation as well as
%language recognition.  But probably we should leave it this way, 
%for simplicity.]]]

%[[[The more general development seems to have a uniformity assumption
%of space $D(n)$ computability of the circuits.
%We must assume that $C(n) \leq 2^{cD(n)}$.
%The conclusions then seem to be that the number of processors, etc.,
%are $O(2^{cD(n)}$, and the time is still $O(D(n))$.  Correct?]]]

\begin{proof}
The PRAM simulator is given $n$ in $R_0$ and the 
$n$ Boolean inputs $x_1, \ldots, x_n$ in registers $R_1, \ldots, R_n$.
The operation of the PRAM has two phases:
\begin{enumerate}
\item Given $n$, generate a description of the circuit $B_n$.
\item Simulate $B_n$ on the given inputs.
\end{enumerate}

Since the description of the circuit $B_n$ can be generated in space
$O(\log n)$ on a TM, it can be generated in time $O(\log(n))$ on the
PRAM, according to Theorem~\ref{PCT-TMs-to-PRAMs}. 
(Here we use the
extended version of Theorem~\ref{PCT-TMs-to-PRAMs}
applied to functions.
%[[[
Before this computation is begun, the string representing $n$, which
starts out in $R_0$, must be split up into separate registers, 
one per bit, since that is what the PRAM producing the circuit expects
for its input.)
%]]]
By assumption,
$D(n) \ge \log(n)$, so the time for phase 1 is $O(D(n))$.
The number of processors and the global 
%[[[
and local
%]]]
memory used in this phase are $2^{c \log(n)}$ for some
constant $c>1$; the number of processors 
%[[[
and the amount of global and local memory are
%]]] 
thus polynomial in $n$.

When phase 1 is completed, we can assume that $(j_i, k_i, \circ_i)$ is
available to processor $P_i$, for $n<i\le C(n)$.  
(Although the output of phase 1 has the circuit description written out
in our standard format of one bit per register, we can ensure that this
format is easily decodable so that $P_i$ can quickly figure out
$j_i$, $k_i$, and $\circ_i$.  For example, the circuit description could
use fixed-width fields for each such quantity.  Since $j_i$ and $k_i$ are
$O(\log n)$ in length, they can easily be built up in $O(\log n)$ time by
processor $P_i$.)
In phase 2 the PRAM
simulates the circuit $B_n$ on the given inputs in time $D(n)$ using
$C(n)$ processors and $O(C(n))$ space (which is polynomial in $n$),
using the procedure described above.
%[[[
At the end, the output bit is moved to $R_1$.
%]]]
\end{proof}

%[[[I got confused here for a while.  I thought we might need $C$ to
%have some kind of step-counting restriction, since I thought the
%processes would need to compute $C(n)$ in order to see if they should
%participate in the computation.  I ended up getting out of the mess
%by changing the definition of ``active process'' to allow inactive
%processes to read any register.  Here, that would allow them to read
%$R_1$, which contains the circuit description, which should let them
%figure out if they're needed.
%]]]

We now turn to the description of the converse result: efficient PRAM
programs can be simulated efficiently by a Boolean circuit family.

\begin{theorem}
\label{PRAMs to circuits}
{\bf [Simulation of efficient PRAMs by Boolean circuits]} 
%[[[Restated for consistency with other theorem statements:
Let $T$, $P$, $S_G$ and $S_L$ be functions,
where $T(n)$ is bounded by a polynomial in $\log(n)$ 
and $P(n)$, $S_G(n)$, and $S_L(n)$ are bounded by a polynomial in $n$.
Let $M$ be a PRAM that works in time $T(n)$, with number of processors
at most $P(n)$, global space at most $S_G(n)$ and local space at most
$S_L(n)$.
Then there is a Boolean circuit family ${\cal B}=\{B_n\}$ that decides
the same language as $M$, 
for which the size $C(n)$ is polynomial in $n$ 
and the depth $D(n)$ is polynomial in $\log(n)$.
%]]]
\end{theorem}
%[[[No step-counting assumptions here?  Maybe not, because of the
%small complexity bounds.  But that's pretty special to these
%particular bounds.
%What is the statement of the extension of this result to higher
%complexity bounds (for later passes through the course).]]]

\begin{proof} (Sketch!)
This proof basically describes how to build a powerful parallel
computer out of elementary logic gates!  As such, there are numerous
details to be described and analyzed.  We only give the main ideas
here; the details are left for the highly motivated reader to fill in.
%[[[
(Or you could try to build one over the summer...)
%]]]

The overall approach is straightforward.  The circuit $B_n$ contains
$T(n)$ levels.  The $i$th level contains the circuitry needed to transform
a description of the PRAM state at time $i$ to a description of the PRAM
state at time $i+1$.  The PRAM state is described by elements specifying
the state of global and local memory and the state of each processor;
this requires only a polynomial number of elements.  

The heart of the proof lies in demonstrating that a level of $B_n$ can
be implemented with Boolean circuitry that has size polynomial in $n$
and depth polynomial in $\log(n)$.

Most of the circuitry is devoted to implementing the memory read and write
operations.  The following exercise is representative of what is
needed to implement a parallel ``read'' operation.

{\bf Exercise 7:}
Design a Boolean circuit that takes as input $n$ Boolean inputs
$x_1$, $x_2$, \ldots, $x_n$ and $n$ ``address'' inputs $y_1$, $y_2$,
\ldots, $y_n$, where $1\le y_i \le n$ and each $y_i$ is represented with
$\ceil{\log(n)}$ input elements.  The circuit should produce as
output the $n$ Boolean outputs:
\[
	x_{y_1}, x_{y_2}, \ldots, x_{y_n}\ .
\]
Furthermore, the circuit should have size that is polynomial in $n$, and
depth that is polynomial in $\log(n)$.

(Hint: Create $n$ pairs $(i,x_i)$ and $n$ triples $(y_i,i,{\bf ?})$.
Sort these $2n$ object in order by their first components, placing
pairs ahead of triples for ties.  Replace the {\bf ?} in the triples
by the closest preceding $x_i$ value in a pair (using a parallel
prefix circuit!).  Delete all the pairs.  Sort the triples into order by their
second component.  Replace each triple by its third component.  For
the sorting circuits, use Batcher's network (see Knuth, volume III),
which has polynomial size and depth polynomial in $\log(n)$.)

%[[[Ugh!  This will look opaque to the students.  (It's not so clear
%to me...)  The reason is that
%this is the first time they have seen any circuit algorithms at all.
%What seems needed is some earlier examples, after the definition of
%circuits, of some particular circuit designs (to give them some
%experience with circuit programming.  For instance, we should do some
%really simple example, then describe a trivial sorting algorithm,
%then show show how parallel prefix can be done by a circuit.
%Are Sipser's notes a good source for examples here?  Do you have a
%copy?  Or we can do some inventing...
%For the sorting example, not just Booleans, but multi-bit numbers
%would be nice.  I'm not sure the Batcher network in Knuth is adequate
%for this case.
%]]]

{\bf Exercise 8:}
%[[[
Modify the problem in
%]]]
Exercise 7 to model ``write'' operations, where each of the
$n$ processors can specify zero or one of the $x$ values to update, and 
the circuit should return the new array of $x$ values.  
Solve.

A further extension of these ideas to ternary values handles the fact
that the PRAM registers can have variable lengths---we can think of
each PRAM register as having a fixed length, where some of the
(leftmost) positions are ``undefined'' rather than $0$ or $1$.  Note
that we have a polynomial upper bound on the length of each PRAM
register contents.
%[[[This ``polynomial'' is a vast overestimate, no?  Isn't it polylog,
%given our new conventions?
%The students may be confused if we just say ``polynomial'' here.]]]
More precisely, the longest any PRAM register value can be at the
end of the computation is at most $T(n)$ bits longer than the length
of the longest register contents at the beginning of the computation.

%[[[What happened to indirect addressing?]]]

The finite-state control of each PRAM processor is relatively straightforward
to implement.  We note that all of the primitive RAM operations can be
implemented with a number of gates that is polynomial in $n$ and depth
that is logarithmic in $n$, when operating on inputs of size $n$.  

{\bf Exercise 9:}
Show that the RAM shift operator $R_j >> R_k$ can be 
implemented with a number of gates that is polynomial in $n$ and depth
that is logarithmic in $n$, when $R_j$ has $n$ bits (assume that
$n$ is a power of 2, $n=2^t$) and $R_k$ has exactly $t=\log(n)$ bits.

This completes our brief sketch of this construction.  
%[[[Ugh!  That sure is brief.  Let's discuss, so at least I understand
%how it all goes.  Can we give the students a reference for this
%construction?]]]
(We said it was
only going to be sketch!)  To complete the proof of the theorem, it
also needs to be shown that a description of the circuit can be
constructed by some Turing Machine $M$ that use logarithmic space.  This
is not hard, just tedious.
\end{proof}

Thus we see that Boolean circuit families and PRAMs are very closely
related.  In particular, Boolean circuit depth corresponds to PRAM
time, and the number of PRAM processors corresponds to Boolean circuit
size.  

It follows from Theorems \ref{circuits to PRAMs} and \ref{PRAMs to
circuits} 
that $NC$ has an alternative definition in terms of circuits instead
of PRAMs:

\begin{theorem}
A language $L$ is in $NC$ if and only if it can be recognized by a
Boolean circuit family of size polynomial in $n$ and depth polynomial
in $\log n$.
\end{theorem}

\section{The Parallel Computation Thesis}

Here is the intuitive statement of the Parallel Computation Thesis:

\centerline{\framebox{Parallel time = sequential space}}

We showed earlier that any computation that can be done in sequential
space $S(n)$ can also be done in parallel time $S(n)$, as long as
$S(n) \ge \log(n+2)$.  Now we show the opposite direction, that
anything that can be done in a certain amount of parallel time can
also be done in approximately the same amount of sequential space.

%[[[
Since Boolean circuit families and PRAMs are so closely related,
we can use either of these models as convenient to obtain basic
results about parallel computation.
Thus, we complete our proof of the Parallel Computation Thesis using
circuits rather than PRAMs.
Specifically, we show how Boolean circuit families of depth $D(n)$ 
can be simulated by a TM working in space $O(D(n)\log n)$.
%]]]

%[[[A problem here:  this only applies to small complexity computations.]]]

\begin{theorem}
\label{PCT-Circuits-to-TMs}
{\bf [Evaluation of shallow circuits with small-space TM]}
Let ${\cal B}=\{B_n\}$ be a Boolean circuit family with size $C(n)$
and depth $D(n)$, where $D(n)\ge \log(n)$. 
%[[[
Then there is a Turing Machine $M$ that computes the same function as
${\cal b}$, for which the space $S(n)$ is $O(D(n) \log n)$.
%]]]
\end{theorem}
%[[[A switch here to functions instead of languages.
%Maybe restrict to languages for consistency?]]]

\begin{proof}
Suppose $B_n$ has $m$ output bits.  The basic idea is to have $M$
perform $m$ depth-first searches of the circuit $B_n$; each
depth-first search computes one of the output bits.  Because the circuit
has depth $D(n)$, each depth-first search can be implemented in 
$O(D(n) \log n)$ space.

We note that because a description of $B_n$ is computable in
logarithmic space, any particular component of this description can be
computed in logarithmic space.  In particular, $m=m(n)$ and $c=C(n)$ 
can be computed in logarithmic space.  Furthermore, given $n$ and $i$,
where $n<i\le c(n)$, the values $(j_i,k_i,\circ_i)$ can be computed in
logarithmic space.

The Turing machine $M$ works as follows on input $x_1x_2\ldots x_n$.
\begin{enumerate}
\item Compute $m=m(n)$ and $c=C(n)$ in logarithmic space.
\item For $i=c-m+1, c-m+2, \ldots, c$, output the value $E(i)$.
\end{enumerate}

Here $E$ is an auxiliary recursive procedure that computes, on input
$i$, the value of the element $e_i$ of the circuit $B_n$ on input
$x$.  Here $x$ is a fixed input of length $n$
always available on $M$'s read-only input tape, so $x$
and $n$ do not need to be passed as arguments to $E$.  The work tape is
used as a stack to aid in the recursive computation $E$.
The evaluation of $E(i)$ proceeds as follows:
\begin{enumerate}
\item If $i<n$, return the value $x_i$.
\item (Otherwise.) Compute the values $(j_i,k_i,\circ_i)$ in logarithmic space.
\item Recursively evaluate $e_{j_i} = E(j_i)$.
\item Recursively evaluate $e_{k_i} = E(k_i)$.
\item Return the value $e_i = e_{j_i} \circ_i e_{k_i}$.
\end{enumerate}

%[[[I like this high-level recursive description, even though we don't
%have a general way of implementing recursion on TMs.  If we want to
%be really careful about the analysis, this might need a lot more
%implementation detail, but for this level of argument, it seems
%fine.]]

Because the circuit $B_n$ has depth $D(n)$, the recursive procedure
$E$ never recurses more than $D(n)$ levels before ``bottoming out'' at
the input variables.  The storage required is thus the storage
required at one level, plus the logarithmic storage required to
perform step~2 above when necessary.  At one level, storage is
required to store $j_i$, $k_i$, $\circ_i$, $e_{j_i}$, and $e_{k_i}$.
The last three values only require a constant amount of storage.  The
values $j_i$ and $k_i$ require at most $\ceil{\log(C(n))}$ bits each
to represent.  Therefore the storage required at one level is
$O(\log(n) + \log(C(n))) = O(\log C(n)) = O(\log n)$, and the overall storage
required is $O(D(n) \log n)$.
\end{proof}

This completes our ``proof'' of the Parallel Computation Thesis.  We
have shown that computations that require small sequential space can
be evaluated efficiently in parallel (Theorem~\ref{PCT-TMs-to-PRAMs})
and that computations that can be require a small amount of parallel
time can be evaluated with a small amount of space
(Theorem~\ref{PCT-Circuits-to-TMs}).  

The Parallel Computation Thesis is a rather surprising result!  

%[[[--- Lecture 4 might start here. ---]]]

\section{$P$-Completeness, Revisited}

We now revisit the $P$-complete problems, arguing that if any
$P$-complete problem can be parallelized, then all problems in $P$
can be efficiently parallelized.

\begin{theorem}
If $A \le_L B$ and $B \in NC$ then $A \in NC$.
\end{theorem}

\begin{proof}
%[[[
Let $f$ be the assumed logspace reduction from $A$ to $B$.
By the Parallel Computation Thesis
(specifically, Theorem \ref{PCT-TMs-to-PRAMs}, extended to functions
as described),
%]]]
%[[[We should pull that statement out as a theorem also, since we use
%it so often.  Then consider all the other places where results could
%be stated for languages and/or functions.  Needs a consistent treatment.]]]
$f$ is computable by a PRAM in time polylog in $n$, 
using a number of processors that is polynomial in $n$,
on any input $x$ of length $n$.
Moreover, the length of the output $f(x)$ is bounded by a polynomial in the
length $n$ of the input $x$.

So let a PRAM for $A$, on input $x$ of length $n$,
first compute the reduction $f(x)$, then simulate the
PRAM for $B$ on input $f(x)$.
The time is the sum of the times for the two subroutines:
polylog in $n$ for the reduction, plus polylog in the length of $f(x)$
for the second phase.
But since the length of $f(x)$ is bounded by a polynomial in $n$,
the time for the second phase
is polylog(poly(n)), which is polylog(n).
Thus, the total time is polylog in $n$.

The number of processors for the first phase is polynomial in $n$, by the
PCT.  For the second phase, the input length is $|f(x)|$, which is
polynomial in $n$.  Since the PRAM for $B$ uses a number of processors
that is polynomial in the length of its input, the total number of
processors is polynomial in $n$.

This is as needed for the definition of $NC$, so $A\in NC$.
\end{proof}

This has the following consequence.

\begin{theorem}
Suppose the problem $A$ is logspace-complete for $P$.
If $A\in NC$, then $P = NC$.
\end{theorem}

At the moment, we know that $NC \subseteq P$, but we don't know
whether or not $NC = P$.  It would be rather surprising if it turned
out that $NC = P$; this would mean that every problem that is
efficiently solvable on a sequential computer (i.e., in $P$), would also
be efficiently solvable on a parallel computer (i.e. in $NC$).  Many
theoretical computer scientists would support the conjecture that $NC
\ne P$, on much the same basis as they would support the conjecture
that $P \ne NP$ --- no one has been able to prove otherwise by finding
efficient algorithms!  This may seem like ``weak'' evidence, but it
is the best we have to go on for now.  As time goes one, the strength
of this sort of evidence improves.  Eventually, we may develop
techniques for actually proving negative statements of the form $NC
\ne P$.  At the moment, we are sadly lacking in techniques capable of
proving such a result.

Problems that are logspace complete for $P$ are thus good candidates
for problems that can be solved efficiently on a sequential computer,
but for which parallelism provides no dramatic speed-up.  (By applying
the preceding theorem: if the problem were in $NC$, then $NC = P$,
which we think is unlikely, so we conclude that it is unlikely that
the problem is in $NC$.)

There are some ``natural'' problems which are logspace complete for $P$.
It thus seems unlikely that parallel computers will provide dramatic
speedups in their solution.  Here are some such P-complete problems.
\begin{itemize}
	\item Unification.
	\item Linear Programming.
	\item The circuit value problem.
\end{itemize}
There are other problems, such as depth-first search, for which we know of
no good parallel algorithms, but which are also not known to be logspace
complete for $P$.

\end{document}








