\documentstyle[12pt,twoside]{article}
\input{defs-6045}
%\input{defs-6045-1989}
\def\implies{\mbox{ implies }}
\renewcommand{\mid}{:}
\def\Ic{\mathify{\{c,I\}}}
\def\Kt{K_{\rm tot}}
\def\TIME{{\rm TIME}}
\def\NTIME{{\rm NTIME}}
\def\GSAT{{\rm GSAT}}
\def\jump#1{^{(#1)}}
\def\parens#1{\left( #1 \right)}

\begin{document}


\handout{Supp-reading-2}{Supplementary Reading Material-2}{April 6, 1992}

\bigskip

These are  lecture notes from a previous year 
about complexity theory, the last topic that we will cover
in this class. They were prepared 
 by Professors
Bui, Cosmodakis, Leiserson,  Meyer, Sipser, Sur and Weiss.
We will   supplement them  with additional material as the class
progresses.
\bigskip






\section{Introduction}

Turing-decidable languages correspond to problems which can in
principle be solved on a computer, {\it when we assume that sufficient
time and space are available to complete the computations.} Ignoring
this crucial question of computational efficiency is appropriate for a
theory explaining the limitations of computation: the assertion that
some problem is computationally {\it unsolvable} is even more
convincing when we adopt an unrealistically generous notion of what
computational solvability means.  On the other hand, the pure fact
that a problem {\it is} computationally solvable, i.e., that a
language is recursive, carries no information about its feasibility.
A quantitative refinement of computability theory known as
``complexity theory'' aims to address such questions about how hard it is to
solve computationally solvable problems.  The major contributions of
complexity theory are methods for determining and classifying the
complexity of a large variety of computational problems from
engineering, economics, language theory, logic, algebra, and other
areas.

In these notes we will develop the theory of one particular familiar measure
of efficiency, namely {\it time}---measured by the number of steps in a
computation.

\section{Time-bounded Turing Machines}
\label{time-bound-sec}

We pick one simple general computer model, namely, multitape multihead
Turing machines, henceforth called TM's, in terms of which to define
``steps'' and hence computational complexity.  In
Section~\ref{complexity-hypo} we will develop a hypothesis similar to
the simulation thesis, which states that this choice of computer model
does not have a significant effect on the complexity of a problem.

In these notes, the {\em complexity} of a language will be measured by a
function that maps the {\it length\/} of a language membership question
(TM input) to an upper bound on the number of steps it takes to
compute the correct answer to the question (TM output).  The dependence
on the length of the question is necessary, since it obviously takes a
long time just to listen to a very long question, even if the question
is easy to answer.  In general, a TM deciding a language must under our
input/output conventions take $n+2$ steps just to read an input of
length $n$ and print out ``YES'' or ``NO''.  It is distracting to
highlight this detail, so henceforth when using a function, $T$, as a
time bound, we shall take ``time $T(n)$'' actually to mean ``time
$\max(n+2,\lceil T(n)
\rceil)$.''

\begin{definition}
Let $L \subseteq \Sigma^*$ be a language, $M$ a TM, and $T$
a function from $\Nat$ to $\Nat$.  We say that {\it $M$ decides $L$ in time $T$}
if $M$ decides $L$, and $M$ halts on input $w$ in at most $\max(|w|+2,\lceil
T(|w|) \rceil)$ steps for each $w \in \Sigma^*$.  If $f$ is a function from
$\Sigma^*$ to $\Delta^*$, we define {\it $M$ computes $f$ in time $T$}
similarly.  
\end{definition}

Using the preceding definition, it is possible to classify languages
based on whether the time specified by a particular function is
sufficient for the language to be decided by a TM.  In particular, we
have the following definition.

\begin{definition}
Let $L \subseteq \Sigma^*$ be a language and $T$
a function from $\Nat$ to $\Nat$
$$\TIME(T) := \{L:\; L \hbox{ is decided by some TM in time } T\}.$$
\end{definition}

Since the definition of ``$M$ decides $L$'' treats ``Yes'' and ``No''
answers symmetrically, we immediately observe 

\begin{lemma}
 $L\in\TIME(T)$ if and only if $\overline{L} \in
\TIME(T)$.
\label{co-time}
\end{lemma}

Observe that the definition of $\TIME$ requires only that $T(|w|)$ be an
upper bound on the number of steps needed to decide whether $w$ is in
a language.  In particular, one should note that if for two functions $T$
and $T'$, from $\Nat$ to $\Nat$, $T'(n) \leq T(n)$, for all $n\in \Nat$,
then $L\in \TIME(T')$ implies
$L\in \TIME(T)$

The exact number of steps needed to compute solutions to problems is
sensitive to minor details in the formulation of computer models,
input/output conventions, and problem representation.  Even for 
multitape multihead Turing machines, 
the number of steps in computations can be slightly altered by routine
reprogramming tricks.  Consequently, we shall rarely try to obtain {\it
precise\/} upper bounds for the time requirements of algorithms.
Instead, we shall focus on the approximate growth rate of the length
of computation with respect to the input length.  The following
$O$ and $\Omega$ notation 
is useful in talking about approximate growth rates.

\begin{definition} Let $f$ and $g$ be functions from $\Nat$ to $\Nat$.
We say that $f$ is $O(g)$ (read ``$f$ is big-oh of $g$'') if there is a
constant $c>0$ and an integer $n_0$ such that $f(n) \leq c \cdot g(n)
\hbox{ for all } n \geq n_0$.
Similarly, 
we say that $f$ is $\Omega(g)$ if there is a constant $c > 0$
and an integer $n_0$ such that
$f(n) \geq c \cdot g(n) \hbox{ for all } n \geq n_0$.
\end{definition}

Note that $f$ is $O(g)$ if and only if $g$ is $\Omega(f)$.
Some other important facts about $O$ are:

\begin{itemize}
\item if $p_1$ and $p_2$ are univariate
polynomials with positive leading coefficients and are
of degree $d_1$ and $d_2$, respectively,
then $p_1 = O(p_2)$ if and only if $d_1 \leq d_2$,

\item $n^\epsilon = O(n^\delta)$ if and only if $\epsilon
\leq \delta$.

\item  $p(n)= O(c^{n^\epsilon})$, for all
$c>1, \epsilon>0$, and polynomials $p$.

\item  $\lceil \log_a(n) \rceil=
O(\lceil \log_b(n) \rceil)$
for all $a,b>1$,

\item  $p(\log n)=O(n)$ for any polynomial $p$.   
\end{itemize}

A feature of our choice of the TM model is that sloppiness in time
bounds, up to constant factors, is inconsequential.  To see this,
consider the following.  Given any TM, $M$, we can construct another TM,
$M'$, whose alphabet consists of pairs of symbols from $M$'s alphabet.
Now, by using one of its own steps to simulate two of $M$'s, $M'$ can
simulate $M$ but run in half the time.  Similarly, we can double the
speed of $M^\prime$ to get quadruple ``speed-up'', etc.  Observe,
however, that before $M'$ can start its double speed simulation, it must
compress the input to half its length, so $M'$ really runs in time
$O(n)+T(n)/2$.  The details are given in L\&P (Theorem 7.1.1).
Formally, we have:

\begin{theorem}{\bf (Linear Speedup Theorem) } Suppose that $L \in$
TIME$(T)$ and $\epsilon>0$.
Then $L \in \TIME(T^\prime)$ 
where $T^\prime(n) = n + 2 + \lceil \epsilon T(n) \rceil$.
\end{theorem}

\noindent From this theorem we easily obtain,

\begin{corollary}{\bf (Speedup corollary)} If $L$ is decidable in time
$T^\prime = O(T)$ then $L$ is decidable in time $T$, provided that
$T(n) \geq (1+\epsilon) n + 2$ for some $\epsilon > 0$ and all $n$.
\end{corollary}

In effect, the speedup corollary states that the use of approximate
growth rates does not effect the complexity classification of a
language.

Another convenient consequence of our choice of multitape
multihead TM's is that
two TM's can be simulated in parallel by a TM with a number of
tapes and heads equal to the sum of those of the simulated TM's.  In
particular,

\begin{lemma} If $L_1, L_2 \in $ TIME$(T)$ then $\ L_1 \cup
L_2$, $L_1 \cap L_2 \in$ TIME$(T)$.
\end{lemma}


\subsection{A Simple Example}
Consider the problems of recognizing composite (nonprime)
integers represented respectively in unary or binary notation.  Namely,

\begin{eqnarray*}
\mbox{UCOMPOSITE} &:=& \{ I^n:\; n \mbox{ is composite }\}\\
\mbox{COMPOSITE} &:=& \{w \in \{0,1\}^* : w \mbox{ is the binary
representation of a composite integer}\}
\end{eqnarray*}

COM\-POS\-ITE $\in \TIME(O(n^2 2^{n/2}))$ since a TM which tries to divide
the input by each of the $2^{n/2}$ binary numbers of length less than or
equal to half the length of the input runs in this time.

On the other hand, UCOMPOSITE $\in \TIME(O(n))$.  A TM which does the job
simply converts the unary input into binary in time $O(n)$, and then proceeds
like the TM for COMPOSITE working on a binary input of length $\log n$.

Notice that this example illustrates a consequence of measuring
complexity relative to input length: the way we code numbers, graphs, or
similar objects affects the complexity of languages that represent sets of
such objects.  From the point of view of general computability theory we
would ignore the distinction between UCOMPOSITE and COMPOSITE, but from
the point of view of complexity theory, they are very different:
UCOMPOSITE is ``easy'', while the above time bound classification of
COMPOSITE indicates that it may be a computationally expensive language
to decide.  (There are better bounds on the complexity of COMPOSITE,
however.)

\section{ Some Sample Decision Problems}
\label{problems-sec}

In this section, we list all of the decision problems that we will be
considering in our study of complexity.  The variety of problems is an
indication of the broad applicability of the concepts of complexity
theory.  In most cases, the definition of the problem suggests a
straightforward decision procedure and corresponding time complexity
class; we indicate those briefly here.  In later sections, we will
classify most of the problems more carefully.

\begin{itemize}

\item UCOMPOSITE.
This is in $\TIME(2n)$, as described in section 2.

\item MULTIPLY $:= \{(u,v,w) : u,v, $ and $w$ are binary representations of
integers such that $u * v = w\}$.
This is in $\TIME(n^2)$.

\item $\hbox{PATH} := \{(G,a,b) : G$ is a graph containing vertices
$a$ and $b$ such that there is a path from $a$ to $b\}$.  The graph
may be encoded, for example, by assigning a unique integer to each
vertex and listing all edges followed by all vertices. 
Using a breadth-first search starting from $a$, it is possible to
show that $\hbox{PATH} \in \TIME(n^2)$.

\item EMPTYREG $:= \{ E : E $ is a regular expression and $L(E) = \emptyset
\}$.  Just convert the regular expression to an NFA and see if there
is any accepting path to a final state.  This can be done in
$\TIME(n^2\log^2n)$, viz., the NFA state graph has $O(n)$ edges, where
$n = |E|$, and can be encoded as a string that is of length $O(n\log
n)$.  The path search takes time proportional to the square of the
length of the encoding.

\item NEREG $:= \{ E : E $ is a regular expression and $L(E) \neq \Sigma^* \}$.
This can be done by converting the expression into a NFA with at most
$3n$ states where $n=|E|$, converting the NFA to a DFA with at most
$2^{3n}$ states, and then seeing if any nonfinal state can be reached from
the start state of the DFA.  This can be done in $\TIME((2^{3n})^2)$, which is
equal to $\TIME(16^n)$.

\item GSAT $:= \{\phi:\; \phi$ is a satisfiable Boolean formula$\}$.
A Boolean formula may be encoded, for example, as a string over the
alphabet $\{),(,\vee,\wedge,\rightarrow,\equiv,\neg,I\}$, where
strings of $I$'s are used to encode variable names.  
 It is easy to see that GSAT $\in \TIME(n^22^n)$, 
using an algorithm that checks all possible truth assignments.

\item 3CNF $:= \{ \phi : \phi $ is in 3 conjunctive normal form (we will define
this later) $ \}$. This can be done in $\TIME(2n)$.

\item 3SAT $:= \{ \phi : \phi $ is a satisfiable formula in 3CNF$\}$. This can
be done in time equal the time for 3CNF plus the time for GSAT.

\item $\hbox{CLIQUE} = \{(G,k):\; G$ is a graph which has a clique
of size $k\}$.  A {\it clique}\/ is a subset $C$ of the vertices of $G$ such
that every pair of distinct vertices in $C$ are connected by an edge
of $G$.
An algorithm that checks all possible sets
of $k$ vertices of $G$ suffices to show that
$\hbox{CLIQUE} \in \TIME(n^32^n)$.
\end{itemize}

These time estimates are always upper bounds and might, as for
COMPOSITE, be improvable. However, nothing substantially better has
been found for GSAT or CLIQUE. 
In fact, it is widely believed that no
significantly better time bounds are likely to exist for these problems.  
Later in the course, we shall investigate why.

Other problems we will study include, 

\begin{itemize}

\item NEREGSQ $:=\{ E: E $ is a regular expression with squaring where $
L(E) \not =  \Sigma^* \}$.

\item EMPTYEXREG $:=\{ E: E $ is an extended regular expression (\ie , with
complement) where $ L(E) = \emptyset \}$.

\item $\hbox{HAMC} := \{G:\ G $ is a graph 
which has a hamiltonian circuit$\}$.
A {\it hamiltonian circuit}\/ of a graph $G$ is defined to be
a circuit that visits each vertex of $G$ exactly once. 
This will be discussed later in the course.

\item TSP (the ``traveling salesperson problem'')
$:= \{(C,b) : C$ is a cost matrix, $b$ is a
nonnegative integer, such that $G$ (the presumed underlying graph) has
a hamiltonian circuit of cost at most $b\}$. 
This will be discussed later in the course.

\item $\hbox{VC} = \{(G,k): G \hbox{ is a graph with a vertex cover of
size } k\}$. 
A {\it vertex cover} of a graph $G=(V,E)$ is defined to be a
subset $V'$ of $V$ such that if $(a,b)$ is an edge of $G$, then either
$a\in V'$ or $b \in V'$.  The size of a vertex cover is the number of
vertices in it.

\end{itemize}

\section{Feasible Problems, and Polynomial Time}
\label{complexity-hypo}

Let $$ {\cal P} = \bigcup_{k\geq 1} \TIME(n^k).$$ We propose this class,
$\cal{P}$, of {\it polynomial time decidable problems} as a useful
theoretical formulation of the notion of ``feasibly computable''
problems.  A simple rationale for this proposal is that for any problem
(language) in $\cal{P}$, two membership questions whose lengths are
within a constant factor of one another can be answered in numbers of
steps that are also within a constant factor of one another, regardless
of the absolute lengths of the questions.  For example, if a language
were in $\TIME(n^2)$, then for any two strings 
$w$ and $w'$ where $|w'| =
2|w|$ membership of $w'$ can be decided in 4 times the number of steps
needed to decide membership of $w$.  This contrasts, say, with
exponential growth functions like $2^n$ for which deciding membership of
$w'$ could require $2^{|w|}$ times the number of steps needed to decide
membership of $w$.

Of course, if a problem in $\cal{P}$ can only be solved in time
$10^{10^9}n^{10^7}$ we hardly regard it as feasible, so these claims
should be taken with a grain of salt.  However, in practice, it has
often been the case that once a polynomial-time algorithm has been
found for a problem, the necessary insights have been obtained to
bring the degree of the polynomial down to a small integer.  But the
real significance we attach to the borderline between $\cal{P}$ and
non-polynomial-time problems is in the case of {\it in}feasibility
results.  As with the general notion of decidability, when we have
demonstrated that a problem is {\it not\/} in $\cal{P}$, our judgment
that the problem is {\it not feasible} is convincing because of the
generosity of our notion of feasible.

(Even here, an objection can be made that although an algorithm can be
exponential in the worst case, it still can perform well on virtually
all instances encountered in practice, as is generally acknowledged to
be the case, for example, with the simplex algorithm for linear
programming.  Of course, the same objection can be raised about our
notion of undecidability.  Namely a problem might be undecidable, but
somehow easy in all practical cases.  Still, it seems fair to say that
establishing an undecidability or infeasibility result forces one to
question and analyze more deeply the basis for pragmatic
optimism.)


In the case of the general theory of computability the
use of Turing machines as a computer model was justified because
the choice didn't really matter: mutual simulations between variant
Turing machine models, between TM's and other computer models such
as random access machines, or even between completely different
computability paradigms such as general grammars or functions defined
by recursive equations, revealed that one model could be replaced by
another.  In particular, the class of decidable languages remained the
same regardless of which kind of general computer we used to do the
deciding.

For complexity theory the Turing machine model must be reexamined,
because Turing machines are obviously less efficient than models with
random access storage or parallelism, for example, so the complexity
of a problem will vary with the computer used to solve the problem.
Nevertheless, the measure of complexity of a problem remains
reasonably stable as we change models, and in particular, the class,
$\cal P$, of feasibly decidable problems remains the same over a large
class of computer models. 

For example, it is not hard to program a one-tape, one-head Turing
machine to simulate our multihead TM's with only
quadratic time loss: the one tape TM takes $m^2$ of its steps to
simulate what a multihead TM does in $m$ steps.  Thus,

\begin{lemma}
 If $L$ is in $\TIME(T)$,
then $L$ can be decided in time $O(T^2)$ by a one-tape, one-head Turing
machine.
\end{lemma}

But $T=O(p)$ for some polynomial $p$ iff $T^2=O(q)$ for some polynomial $q$,
so the class $\cal{P}$ would be the same even if one-tape Turing machines were the
basic model.  In general, we can formulate 

\noindent {\bf The Complexity Modeling Thesis:} The informal notion of
``bounded-size computation step'' is captured to within a polynomial
by the precise notion of computation step of a multitape multihead
Turing machine.  In particular, if $L$ can be decided in $T$ bounded-size
steps of some effective computational procedure, then $L \in \TIME(p
\circ T)$, for some polynomial $p$.
(Recall that $(p \circ T)(n) = p(T(n))$.)
%[[[Q: Is this circle the proper Latex notation for function composition?

The thesis is supported by a series of results about mutual simulations
like that of the lemma above.  For example, TM's simulate random-access
machines (RAM's) with only polynomial time loss, providing we remember
to count each RAM operation properly.  The need for a proper count
arises because in our model of RAM's, single RAM instructions may
involve words or integers of unbounded size, so it's not appropriate to
count one instruction execution as one step.  (Suppose the instruction
required setting to zero some register whose {\it address\/} was
billions of bits long.)  We ought to count each RAM instruction as a
number of steps proportional, say, to the sum of the lengths of the
contents and addresses of all registers involved in the instruction.  In
fact, as long as the $m^{\hbox{th}}$ RAM instruction executed in a
computation only involves words of length bounded by a polynomial in
$m$, the complexity modeling thesis still holds even if we {\it do}
count each RAM instruction as only one step.  This is the case, for
example, if the RAM instructions involved only comparing, copying, or
adding the contents of registers containing numbers in binary notation.
But the complexity modeling thesis fails if there is a unit-cost
multiplication instruction, since $n$ squarings of a register containing
the integer 2 yield a register containing a number of length $2^n$.

%[[[Students will never understand this without more background.  Alex]]]
%[[[It can be explained more in lecture.  
%I also tried adding some explanation - check if this is accurate -
%I don't really know the conventions offhand.  N.]]]

Similarly, TM's can simulate finite dimensional systolic arrays of
(bounded size) finite automata with only polynomial time loss.  (A {\em
systolic array} is a network of components similar to DFA's, but
interconnected by input and output channels - the output of one
component can be the input of one or more others.  The original input
arrives at a particular initial component.  The entire network operates
in lock step.)  Observe, however, that TM's cannot be expected to
simulate tree-connected systolic arrays with only polynomial loss
because on the $m^{\hbox{th}}$ step of a tree-connected systolic array
computation a number of cells exponential in $m$ may change state.

So we cannot be quite as casual about our choice of computer model for
complexity theory as we could for general computability theory, but once
we are aware of the {\it proviso}'s above, we can commit to our TM
model confident of the invariance of our conclusions about what is, and
especially what is {\em not}, feasibly computable.

Among the problems listed in section~\ref{problems-sec} UCOMPOSITE,
PATH, MULTIPLY, 3CNF and EMPTYREG are in ${\cal P}$.  The COMPOSITE
problem has not been proven to be in $\cal P$, but if a well-known and
widely believed number-theoretic hypothesis of Riemann is true, then
COMPOSITE is also in $\cal P$.

The problems GSAT, 3SAT, VC, CLIQUE, HAMC, and TSP are generally believed
{\it not}\/ to be in $\cal P$, although this fact is not definitely
known.  We shall see, however, that any one of them is in $\cal P$
if and only if they all are.

In section~\ref{squaring-sec}, we shall show that NEREGSQ $\not \in \cal P$.  In fact, there
is an $\epsilon > 0$ such that NEREGSQ $\not\in \TIME(2^{\epsilon
n})$.  EMPTYEXREG is even harder.

\section{The Time Bounded Halting Problem}

In this section, we give our first example of a hard, but decidable,
problem.  This example arises from the same kind of diagonal
construction we used to obtain undecidable problems.  Namely, for
$T:\Nat\rightarrow\Nat$, let $$K_1^T:=\{x:\;M_x\hbox{ is a TM which
accepts input }  x\hbox{ in }\leq T(|x|)\hbox{ steps}\}.$$ First we
prove that $K_1^T$ cannot be decided within time T; and then we show
that (under a technical assumption) $K_1^T$ can be decided within time
that is not much greater than T, namely, within time $n T^2(n)$.

\newcommand{\koc}{\mathify{\complement{K_{1}^{T}}}}

\begin{theorem}  
$K_1^T\not\in\TIME(T)$.\hfill\break
\end{theorem}

\begin{proof} 
Choose any string $x_0$ such that \(M_{x_0}\) decides \koc.  So, for all $x$, 
$$x \in \koc \iff  M_{x_0} \mbox{ accepts } x $$

By definition of \koc, we also have for all $x$, 
$$ x \in \koc \iff M_x \mbox{ does {\bf not } accept } x \mbox{ in }
\leq T(|x|) \mbox{ steps.} $$
Now, let $x$ be $x_0$, and conclude that $$ M_{x_0} \mbox{ accepts } x_0
\iff M_{x_0} \mbox{ does {\bf not} accept } x_0 
\mbox{ in } \leq T(|x_{0}|) \mbox{ steps } $$
It follows immediately that \(M_{x_0}\) does not accept $x_0$ in less
than \(T(|x_0|)\) steps, though it does accept
\(x_0\) 
in a larger number of steps.

Hence, no program which decides \koc halts in at most \(T\) steps on all inputs.
By Lemma~\ref{co-time}, $K_1^T$ is also not in $\TIME(T)$.
\end{proof}

\begin{corollary} $K_1^{2^n}\not\in\cal P$.\hfill\break
\end{corollary}

\begin{proof}Since $p=O(2^n)$ for any polynomial
$p$, if $K_1^{2^n}$ were in $\cal P$, then, by the Speed-up Corollary,
it would also be in $\TIME(2^n)$, a contradiction. 
\end{proof}

Now $K_1^T$ is obviously decidable if $T$ is computable: given input $M$
of length $n$, simply compute $T(n)$ and then simulate $M$ on input $M$
for $T(n)$ steps.  In general, $M$ may have more heads, tapes, or tape
symbols than the simulating machine, so the time to simulate $T(n)$
steps of $M$'s computation is larger than $T(n)$.  A straightforward
simulation works out to require $O(nT^2(n))$ steps.  (This can be
improved, but it the bound sufficient for our purposes in these notes.
Notation:  $T^k(n):=(T(n))^k$.)  Observe, however, that we must also add
the time to compute the bound $T(n)$ itself.  Here is where it is
convenient to add a technical assumption.  Nearly all the functions,
$T$, that we ever want to use to count steps are computable in time
$O(T)$.  Thus we make the following definition.

\begin{definition}
$T:\;\Nat\rightarrow\Nat$ is a 
{\it step-counting function\/} if there is a TM which computes
$T$ in time $O(T)$, using unary representation of integers for input and
output.\footnote{ This definition differs
from the one in L\&P (Def{.} 7.4.3).  However, with a few
programming tricks like linear speed-up, the two definitions
can be proved equivalent for $T(n)\geq(1+\epsilon)n$.}
\end{definition}

It turns out that the functions most commonly used for time bounds
are step counting functions.  For example, the class of step counting
functions contains the identity and successor functions, and is
closed under addition, multiplication, exponentiation and composition.

\begin{theorem}  For any step counting function
$T(n)$,
$$\TIME(T)\psubset\TIME(nT^2(n)).$$
\end{theorem}

(We use $\psubset$ for {\it proper}\/ subset.)

\begin{proof}Inclusion is obvious.  That the inclusion
is proper is guaranteed by the fact that $K_1^T$ is decidable
in time $O(nT^2(n))+O(T(n))=O(nT^2(n))$. 
\end{proof}

For example, 
$$\TIME(n)\psubset\TIME(n^3),$$
and
$$\TIME(n^2)\psubset\TIME(n^5).$$
Since $n^k=O(2^{n^\epsilon})$ for any $k>1,\epsilon>0$, we have
$$\TIME(n^k)\psubset\TIME(2^{n^\epsilon})$$
for any $k>1,\epsilon>0$.  In particular, 
$${\cal P}\psubset\TIME(2^{n^\epsilon})$$
for any $\epsilon>0$.

\noindent {\bf Exercise:}  Justify this last conclusion.  (After all, if,
for every $n$, $A_n\psubset B$, it does not follow that 
$\cup_n A_n \psubset B$; consider $A_n=\{1,\ldots,n\}$ and
$B=\{n:\;n\geq1\}$.)

In addition, we remember from calculus that
$n=O(2^{\sqrt{n}})$ and $\sqrt{n} = O (\delta n)$ for any $\delta>0$, so 
$$n\left(2^{n^{0.5}}\right)^2\ = 
O(2^{n^{0.5}})^3=
O\left(\left(2^3\right)^{n^{0.5}}\right)=
O\left(8^{\sqrt{n}}\right) =
O\left(8^{\delta n}\right) =
O\left({(1+\delta')}^n\right)$$
for some $\delta'$ which approaches $0$ as $\delta$ approaches
$0$.  Therefore, for any $\epsilon>0$, $${\cal
P}\psubset\TIME(2^{\sqrt{n}})\psubset\TIME({(1+\epsilon)}^n).$$

\section{ Polynomial-time Reducibility}

The complexity theoretic notion corresponding to many-one reducibility is
polynomial-time reducibility.

\begin{definition} Suppose $A \subseteq \Sigma^*$ and $B \subseteq \Delta^*$.  
We say that {\it $A$ is polynomial-time reducible to $B$} (written $A
\leq_p B$) if and only if there exists a function $f : \Sigma^* \rightarrow
\Delta^*$ {\em computable in polynomial time} such that for all $w\in
\Sigma^*$, $w \in A \iff f(w) \in B$.  We say that $A$ and
$B$ are polynomial-time equivalent, written $A \equiv_p B$, iff $A
\leq_p B$ and $B \leq_p A$. 
\end{definition}

The following are some useful properties of
$\leq_p$.

\begin{theorem} If $A \leq_p B$ and $B \leq_p C$ then $A
\leq_p C$.
\end{theorem}

\begin{proof} \LP, Lemma 7.5.1. 
\end{proof}

\begin{theorem} If $A \leq_p B$ and $B \in \TIME(T)$ for
some nondecreasing $T$, then $A \in \hbox{TIME}(p + T \circ p)$ for some
polynomial $p$.
\end{theorem}

\begin{proof}If $f$ is a function reducing $A$ to $B$ in
polynomial time and $M_B$ is a TM which decides $B$ in time $T$,
then the following TM, $M_A$ decides $A$:

``On input $w$, compute $f(w)$ and then simulate $M_B$ on
input $f(w)$, producing the same output as $M_B$.'' 

\noindent The time taken by $M_A$ on input $w$ is the time 
to compute $f(w)$, plus the time to simulate $M_B$ on $f(w)$.
Let $p(|w|)$ be the time needed to compute $f(w)$.
Since $|f(w)| \leq p(|w|)$ and $T$ is nondecreasing, the simulation
of $M_B$ requires time
$\leq T(p(|w|)$.  Consequently, the time taken by $M_A$ on input $w$ is
$p(|w|) + T(p(|w|)$
\end{proof}

\begin{corollary} If $A \leq_p B$ and $B \in \cal P$ , then $A
\in \cal{P}$.
\label{corollary-AinP}
\end{corollary}

That is, membership in $\cal P$ inherits downward under $\leq_p$.

\begin{proof} If $B \in \cal P$, then $B \in \TIME(n^k)$ for some $k \in \Nat$.
Since $n^k$ is nondecreasing, $A \in \hbox{TIME}(p + p^k)$.  But $p +
p^k$ is a polynomial, so $A \in \cal{P}$.
\end{proof}

We can use $\leq_p$ to prove negative results just as we used
$\leq_m$ to prove undecidability results.  For example, we
proved that the general halting problem $K_0$ was undecidable by
reducing $K_1$ to it.  We can do something similar with the general
time-bounded halting problem:
$$K_0^T := \{ \langle M,x \rangle : \hbox{ Turing machine } M \hbox{ accepts }
x \hbox{ in } \leq T(|x|) \hbox{ steps}\}.$$
Clearly $K_1^T \leq_p K_0^T$ by a straightforward reduction.  
Since $K_1^{2^n} \not\in {\cal P},$ we conclude $K_0^{2^n}
\not\in {\cal P}$. 

We can draw other types of conclusions as well.  For example, we can
conclude that if $L_1 \not\in \TIME(2^{2^n})$, then $L_1 \not\leq_p L_2$
for any $L_2
\in \TIME(2^{n^k})$. 


\section{$\leq_p$-Hardness and Completeness}
For a class of languages, $\cal C$, we will be interested in the {\it
hardest\/} languages in $\cal C$.  

\begin{definition} Let $\cal C$ be a class of languages and
$L$ a language. Then $L$ is $\cal C$-{\em hard} with respect to
$\leq_p$ if $L^\prime \leq_p L$ for every $L^\prime \in \cal C$.  $L$ is
$\cal C$-{\em complete} with respect to 
$\leq_p$ if $L$ is $\cal C$-hard with respect to
$\leq_p$ and, moreover, $L \in \cal C$.
\end{definition}

Informally, deciding a language which is $\cal C$-hard 
with respect to $\leq_p$
is as ``difficult'' (up to composition with a polynomial) as solving
any language in $\cal C$.

\begin{lemma} If $L_1, L_2$ are $\cal C$-complete with respect to
$\leq_p$, then $L_1 \equiv_p L_2$, and hence $L_1 \in \cal P$ iff 
$L_2 \in \cal P$ iff ${\cal C} \subseteq {\cal P}$. 
\end{lemma}

\begin{proof} Exercise.
\end{proof}

Henceforth we shall simply say $\cal C$-hard instead of $\cal
C$-hard with respect to $\leq_p$; similarly for $\cal C$-complete.

\begin{lemma} If $L$ is $\TIME(2^n)$-hard, then $L \not \in
\cal P$.
\label{hard-notP}
\end{lemma}

\begin{proof} 
By the definition of $\TIME(2^n)$-hard, $A \in \TIME(2^n)$ implies $A
\leq_p L$.  So if $L \in P$, then $A \in \cal P$ by
Corollary~\ref{corollary-AinP}.  This implies that $\TIME(2^n)
\subseteq {\cal P}$, which contradicts
the result that ${\cal P}\psubset \TIME((1+\epsilon)^n)$ for any
$\epsilon > 0$.
\end{proof}

Even when we do not know the precise complexity of problems,
polynomial time reductions may be informative.  For example:

\begin{definition}  A {\it vertex cover} of a graph $G=(V,E)$
is a subset $V'$ of $V$ such that if $(a,b)$ is an edge of $G$, then
either $a\in V'$ or $b \in V'$.  The size of a vertex cover
is the number of vertices in it.
\end{definition}

\noindent For example $\{2,3\}$ is a vertex cover of the graph in Figure 2.

Now we can define VC as follows.

$$\hbox{VC} = \{(G,k): G \hbox{ is a graph with a
vertex cover of size } k\}\ .$$

\begin{theorem} VC  $\equiv_p$ CLIQUE.
\end{theorem}

\begin{proof}Note that $C \subseteq V$ is a clique in a graph
$G=\langle V,E\rangle $ iff $V - C$ is a vertex cover of the
complementary graph $\overline{G} =\langle V,V^2-E\rangle $, as the
reader can check directly from the definitions.  
\end{proof}

It is not known whether VC and/or CLIQUE is in $\cal P$.  We shall say
more about this later in the course.  But we can certainly now say

\begin{corollary} VC  $\in \cal P$ iff CLIQUE $\in \cal P$. 
\end{corollary}

\section{A Provably Infeasible Problem from Automata Theory}
\label{squaring-sec}

We have learned procedures for deciding whether a regular expression
denotes $\Sigma^*$.  That is, if we define
$$\hbox{NEREG} ::= \{E : \ E \hbox{ is a regular expression over alphabet
$\Sigma$ and } L(E) \not= \Sigma^*\},$$
then NEREG is a decidable language.

Let us introduce a formal
``squaring'' operation for regular expressions; namely, we will
say that if $E$ is a regular expression, 
then so is $(E)^2$ and $L\left((E)^2\right) =_{\rm def} L(E)\cdot L(E)$.
The languages expressible by regular expressions with squaring
are precisely the regular languages, for we can always take
a regular expression with squaring and write an equivalent
regular expression without squaring, using the fact that
$L\left((E)^2\right)=L(E\cdot E)$.
Thus, letting
$$\hbox{NEREGSQ} ::= \{E : E \hbox{ is a regular expression with squaring
over alphabet $\Sigma$ and } L(E) \not = \Sigma^* \},$$
we conclude that
$$\hbox{NEREGSQ} \equiv_m \hbox{NEREG},$$
and since NEREG is decidable so is NEREGSQ.

On the other hand, squaring can make significant difference in the {\it
length\/} of a regular expression.  For example, $$(\cdots((0 \cup 1
\underbrace{)^2)^2 \cdots)^2}_n$$
is a regular expression with squaring of length $O(n)$ denoting the set
$\{0,1\}^{2^n}$, but the size of the smallest regular expression
without squaring for this set is at least $2^n$.  So, as with the
example of UCOMPOSITE and COMPOSITE from section~\ref{time-bound-sec},
we would expect the distinction between NEREG and NEREGSQ to be
important in complexity theory.

In this section we will prove that NEREGSQ $\not\in {\cal P}$.  (This
is not known to be the case for NEREG.)

NEREGSQ is our first example of a ``natural'' problem which is provably
infeasible. The only other problems we have seen which are provably
infeasible were constructed by diagonalization to have this property.
%Conceivably, all the ``real'' problems one might encounter could be in
%{$\cal P$}, even though there exists problems which require more than
%polynomial time.  The result of this section proves that this is not the
%case.

We will actually prove a stronger fact:

\begin{theorem}  NEREGSQ is TIME$(2^n)$--hard.
\label{neregsq}
\end{theorem}

From Lemma~\ref{hard-notP}, we know that there is a language $L \in
\TIME(2^n)$ which is not in $\cal P$.  By Theorem~\ref{neregsq}, $L \leq_p$
NEREGSQ, so it certainly follows that NEREGSQ $\not\in \cal P$.

\begin{proof} 
Let $L$ be any language such that $L\subseteq\Sigma^*$ and
$L\in\TIME(2^n)$.  For some $k$, the language $L$ will be decidable in
time $\leq 2^{kn}$ by a {\it one-tape, one-head\/} Turing machine, $M=
(K,\Sigma,\delta,s)$.

We can succinctly describe a configuration $(q,u,a,v)$ of $M$ as a
string, $uqav$.  We define the {\it computation words} for $M$ on
input $w$, to be certain strings over the alphabet $\Sigma_c = \Sigma
\cup \{\$\} \cup K \cup \{h\}$ 
(where $\Sigma, \{\$\} K,$ and $\{h\}$ are all assumed to be disjoint
sets).  Specifically, computation words for $M$ on input $w$, are of the
form

$$\$C_1\$C_2\$...\$C_l\$ $$

%[[[The letter n is already used for something else.]]]

\noindent where 
\begin{enumerate}
\item $C_1$ is of the form $\#ws(\#^j)$, where s is the start state and $j>0$, 
\item $C_i$ is of the form $u_iq_ia_iv_i(\#^{j_i})$ for all $i, 1 < i \leq l$,
where $u_i \in \Sigma^*$, $q_i \in K$, $a_i \in \Sigma$, $v_i \in \Sigma^+$ and
$j_i \geq 0$,
\item $(q_i, u_i, a_i, v_i) \vdash_M (q_{i+1}, u_{i+1}, a_{i+1}, v_{i+1})$ 
for each $i, 1 \leq i \leq l-1$, and
\item all the $C_i$'s have the same length, namely $|w| + j + 2$.
\end{enumerate}
The substrings of a computation word enclosed between \$'s 
are called {\it configuration words}.  
The configuration words do not include the \$ characters.

Note that $M$ accepts $w$ if and only if there is a
computation word for $M$ on $w$
such that $C_l$ is of the form $ \$h1\#^{j_l}\$ $.


We want to reduce $L$ to NEREGSQ.  To do this, we transform any string
$w \in \Sigma^*$ to an extended regular expression, $E_w$, such
that $L(E_w)=\Sigma_c^*-\{$the accepting computation word of $M$ on $w$
(if any) whose configuration words are of length exactly $n+3+2^{kn}\}$.
In particular, if there is no such computation word for $w$, then
$L(E_w)$ equals $\Sigma_c^*$.  Now suppose $w \in L$.  Then,
because of the assumed time bound for $M$, there is an accepting
computation word of the indicated form, so $E_w \in \hbox{NEREGSQ}$.  On
the other hand, if $w$ is not in $L$, then there is no such computation
word at all for $M$ on $w$, so that $E_w$ is not in NEREGSQ.  In
short, $$w\in L \hbox{ iff } E_w\in\hbox{ NEREGSQ}.$$  Finally, we will also show
that the construction of $E_w$ from $w$ can be done in time polynomial
(even linear) in $|w|$, so $L\leq_p$ NEREGSQ.

To understand how to construct $E_w$, first observe that whether a
string $z$ is a computation word for $M$ on input $w$ can be
checked using a
{\it legality}\/ function for $M$, 
$legal_M: {\Sigma_c} \times {\Sigma_c} \times {\Sigma_c} \times {\Sigma_c}
\rightarrow \Sigma_c$.  
We use this function to ensure that adjacent configuration words,
$C_i = a_1\cdots a_n$ and $C_{i+1} = b_1\cdots b_n$, are consistent
with the transition function of $M$.  In particular, we check the
local condition $b_{j+1} = legal_M(a_j,a_{j+1},a_{j+2},a_{j+3})$, for
all $j$.

\noindent More precisely, we define the computation words of $M$
in terms of the legality function for $M$ as follows.
We say that $z = z_1\cdots z_{|z|}$ is a computation word of $M$
exactly if it satisfies the following conditions.
\begin{enumerate}
\item the string $\$\#ws\#^i\$$ is a prefix of $z$ for some $i > 0$, and,
letting $m=|\$\#ws\#^i|$,

\item for all $j$, $|z| \geq j > m$,\ \ $z_j = {\it
legal}_M(z_{j-m},z_{j-m+1},z_{j-m+2},z_{j-m+3})$, and

\item $\$\#\;{\rm 1}\;h\#^{m-4}\$$ is a suffix of $z$.
\end{enumerate}


We find it helpful to define an operator $\widehat{\hbox{\ }}$ that
can be applied to any {\it string}\/ $x\in\Sigma_c^*$ to yield a
{\it regular expression}\/, $\widehat{x}$, for which
$L(\widehat{x})=\{z\in\Sigma_c^*:\; x$ is {\it not}\/ a prefix of
$z\}$.  For example, $\widehat{abb}$ will be a (particular) regular
expression denoting the set of all strings in $\Sigma_c^*$ which do
{\it not}\/ begin with $abb$; any such string either has fewer than
three characters, or has at least three characters, but the first
three characters are not $abb$.  For the general case, we define
$\widehat{x}$ inductively, as follows:

\begin{enumerate}
\item $\widehat{e}=\emptyset$: $e$ is a prefix of {\it any}\/ string,
so $\emptyset$ is the set of strings that $e$ is not a prefix of, which is
itself a regular expression.

\item $\widehat{\sigma} = (\Sigma_c - \sigma) \Sigma_c^* \cup \{e\}$, 
for any $\sigma\in\Sigma_c$:
the set of strings not having $\sigma$ as a prefix is just the set
of all strings that start with some character other than $\sigma$,
together with the empty string.

\item $\widehat{\sigma x} = \widehat{\sigma} \cup \sigma\widehat{x}$,
for any $\sigma\in\Sigma_c$ and $x \neq e$:
the set of strings that do not begin with $\sigma x$ is the set of
strings that do not start with the character $\sigma$, together with
those that do start with $\sigma$, but whose remainder does not start with
$x$.
\end{enumerate}


Note that the right hand sides of the equations in (2) and (3)
above are not really regular expressions, 
but rather abbreviations for regular expressions.
For example, $\Sigma_c$
abbreviates $(\sigma_1\cup\sigma_2\cup
\cdots\cup\sigma_{|\Sigma_c|})$.  You should observe that the
length of the regular expression $\widehat{\sigma x}$ is
just some constant (depending on $M$) plus the the length of the
regular expression $\widehat{x}$.  Thus the length of the
regular expression $\widehat{x}$ is $O(|x|)$.  Furthermore, this
regular expression is readily computed from $x$ 
in polynomial time.

How does this $\widehat{\hbox{\ }}$ operator that maps strings to
regular expression relate to reducing $L$ to NEREGSQ?  Recall that we
are assuming that $L$ is decidable by a 1-tape TM, $M$, in time
$2^{kn}$.  Thus $M$'s configuration words on $w$ need not have length
exceeding $2^{kn}+n+3$.  So $M$ accepts $w$, where $|w|=n$, iff $M$ has
an accepting computation word whose configuration words are of length
exactly $2^{kn}+n+3$.  And $M$ has such a computation word, $z$, iff
\begin{enumerate}
\item $z$ begins with $\$\#ws\#\#^{2^{kn}}\$$,

\item successive configuration words follow according to
the legality function for $M$, and

\item $z$ ends with $\$\#\;{\rm Y}\;h\#^{2^{kn}+n}$.
\end{enumerate}

Using our $\;\widehat{\hbox{\ }}\;$
operator, we can write a regular expression
with squaring that denotes the language over $\Sigma_c$ for which
these three conditions do {\it not}\/ hold:
\begin{enumerate}
\item $z$ does not begin correctly: either the initial four
characters are wrong, or there are the wrong number of \#'s to the right
of the head in the leftmost configuration word, 
or the initial configuration word in $z$ does not terminate with
a \$:
$$\widehat{\;\;\$\#ws\;\;} \cup\; (\$\#ws(\#\cup e)^{2^{kn}}\widehat{\#})
 \;\cup\; \$\#ws\#\#^{2^{kn}}\widehat{\$}.$$

\item some move is wrong: 
 $$\bigcup_{a,b,c,d\in\Sigma_c}\;\Sigma_c^*abcd\Sigma_c^{2^{kn}+n+1}
  \left(\Sigma_c-{\it legal}_M(a,b,c,d)\right)\Sigma_c^*.$$

\item $z$ doesn't end correctly.  This is handled analogously
to (1).
\end{enumerate}
The expression $E_w$ is the $\cup$ of the regular expressions with
squaring from (1), (2), and (3).  Again, the formulas exhibited above
are actually just descriptions of regular expressions with squaring:  we are
implicitly using the expressive power of the squaring operator
when we write things such as $\Sigma_c^{2^{kn}+n+1}$.  This would
correspond to a very long string without the squaring operator,
but by using the squaring operator repeatedly, this can
be written as 
$$(\cdots((\Sigma_c\underbrace{)^2)^2\cdots)^2}_{kn}\;
\underbrace{\Sigma_c\Sigma_c\cdots\Sigma_c}_{n+1},$$
which is of length $O(n)$.  The entire expression, $E_w$, is
of length $O(n)$.  

Now we have that any language $L\in\TIME(2^n)$ reduces to NEREGSQ
via a polynomial time reduction.
That is, $w\in L \Longleftrightarrow E_w\not=\Sigma_c^*$.  This means
that NEREGSQ is $\TIME(2^n)$--hard. 
\end{proof}


\end{document}
