One Hat Cyber Team

View File Name : CHAPT23.TEX

\section{Representations of two-dimensional real Lie algebras, their
non-linear
transformations and semilinear relations}

\subsection{Representations of two-dimensional real Lie algebras and
their non-linear transformations by bounded operators}

1. Lie algebras $(IV_0)$, $(IV_1)$, $(IV_2)$, as like as relations
$(V_0)$, $(V_1)$ $(VI_1')$ can be treated from the common point of view.
Namely, all these relations are partial cases of the relation
\begin{equation}\label{deformedlie}
[A,B] = iP_2(A)
\end{equation}
where $P_2(A)$ is a real quadratic polynomial.

If $A= A^*$, $B=B^* \in L(H)$, then their study can be done using the
following theorem.

\begin{theorem} (D. Kleincke, F. V. Shirokov). If $P$ and $Q$ are
bounded operators, and $[P,[P,Q]]=0$, then the operator $[P, Q]$ is
quasi-nilpotent, i.e.,
\[
\lim_{n \to \infty} \sqrt[n]{\|[P, Q]^n\|} =0.
\]
\end{theorem}

\begin{proof}
The Kleinecke-Shirokov theorem follows, for example, from the fact
formula $\ad_P^nQ^n= n!\, (\ad_PQ)^n$, where $\ad_P(X) = PX - XP$ is a
bounded operator in $L(H)$, and $\|\ad _P\| \le 2 \|P\|$. Then
\[
\sqrt[n]{\|[P,Q]^n\|} \le \frac 2{\sqrt[n]{n!}} \, \|P\|\cdot \|Q\| \to
0, \quad n \to \infty.
\]
\end{proof}

In Section \ref{sec233} below we will give a generalization of this
theorem.

Now we consider bounded self-adjoint operators
satisfying~\eqref{deformedlie}.

\begin{proposition}
The irreducible pairs, $A$, $B$, of bounded self-adjoint operators which
satisfy relation~\eqref{deformedlie} are one-dimensional, and they are:
$A=\lambda$, $B=\mu$, where $(\lambda, \mu) \in M = \{(\lambda, \mu \in
\mathbb{R}^2 \mid P_2(\lambda) =0\}$.

An arbitrary bounded pair has the form
\[
A = \int_M \lambda \, dE(\lambda, \mu), \quad B = \int_M \mu\,
dE(\lambda, mu),
\]
where $E(\cdot, \cdot)$ is a resolution of the identity on $M$.
\end{proposition}

\begin{proof}
Indeed, \eqref{deformedlie} implies $[A, [A,B]]=0$, and due to the
Kleincke-Shirokov theorem, the operator $[A,B]$ is quasi-nilpotent. But
since $[A,B]$ is skew-adjoint, it yields $[A,B]=0$. Then $P_2(A)=0$, and
the statement follows from the spectral theorem for a pair of commuting
self-adjoint op[erators.
\end{proof}

\begin{remark}
The latter proposition imlies that if the polynomial, $P_2(\cdot)$ has
no real roots, then there are no bounded self-adjoiint pairs, which
satisfy \eqref{deformedlie}. in particular, there are no bounded pairs
which satisfy CCR (relation $(IV_1)$), or $[A,B] = i(A^2 +I)$ (relation
$(V_1)$).
\end{remark}

\begin{remark}
Relation $(V_2)$ $[A,B] = i(A^2 +B)$ also can be rewritten in an
equivalent form $[A, A^2 +B] = i(A^2 +B)$, which can be reduced by a
non-degenerate non-linear change of variables, $\tilde A = A^2 +B$,
$\tilde B = B$, to the form $[\tilde A, \tilde B] =i\tilde A$. Since
$\tilde A$ and $\tilde B$ are also bounded self-adjoint operators, then
by the Kleincke-Shirokov theorem we have $[\tilde A, \tilde B]=0$, and
$[A,B=0$, which yields $A^2 =-B$.

Irreducible representations of relation$(V_2)$ are one-dimensional and
they are $A=\lambda$, $B=\mu$, $(\lambda, \mu \in M_{(V_2)} = \{
(\lambda, \mu) \in \mathbb{R}^2 \mid \lambda^2 = \mu\}$.

An arbitrary pair of bounded self-adjoint operators satisfying relation
$(V_2)$ has the form
\[
A = \int_{M_{(V_2)}} \lambda\, dE(\lambda, \mu), \quad A =
\int_{M_{(V_2)}}
\lambda\, dE(\lambda, \mu),
\]
where $E(\cdot, \cdot)$ is a resolution of the identity on $M_{(V_2)}$.
\end{remark}

\begin{remark}
In the study of representations of relations from families $(IV)$ and
$(V)$, we could select finite-dimensional level separately, and instead
of the Kleincke-Shirokov theorem, which was proven in 1956, apply its
finite-dimensional variant, which is the following much more early
(1935) theorem. We formulate it in the simplest case.

\begin{theorem} (N. Jackobson) If $\dim H < \infty$, and $[P,[P,Q]]=0$,
then the operator $[P,Q]$ is nilpotent.
\end{theorem}

However, in relations $(VI)$ and $(V)$,  we do not select
finite-dimensional level separately, since no new representations arise
when passing from finite-dimensional case to bounded one.
\end{remark}

\begin{remark}
Of course, considering unbounded operators, for which the
Kleinecke-Shirokov theorem fails, one obtain much more consistent
representation theory of relations $(IV)$ and $(V)$ (see Chapter III).
\end{remark}

\subsection{Pairs of operators connected by semilinear relations}

In Sections 2.3.2---2.3.5 we study a wide class of
so-called semilinear relations which connect pairs of bounded
operators $A=A^*$, $B\in L(H)$.  They appear as a generalization
of relations (\ref{}) and (\ref{}) studied in  previous sections.
We prove many Kleinicke-Shirokov type theorems
(2.3.3).  For general semilinear relations we describe
properties of irreducible representations (2.3.4). All
irreducible representations are classified up to unitary
equivalence for semilinear ${\cal F}_4$-relations (2.3.5).

 1. Consider bounded operators $A=A^*$,
$B\in L(H)$ which satisfy relation of the form:
\begin{equation}\label{sr}
\sum_{i=1}^nf_i(A)Bg_i(A)=h(A),
\end{equation}
where $f_i(\cdot)$, $g_i(\cdot)$, $h(\cdot)$,  $i=1,\ldots,n$ are complex
bounded Borel mappings defined on ${\Bbb R}$ or some subset $D$,
$\sigma(A) \subset D$. Relation (\ref{sr}) is called semilinear
and the operators $A$, $B$ the representation of (\ref{sr}).

If $f_i(\cdot)$, $g_i(\cdot)$,
$h(\cdot)$, $i=1,\ldots n$, are polynomials, then the operators
$A$, $B$, $B^*$ define
a representation of the
$*$-algebra $\frak A$ with three generators $a=a^*$, $b$, $b^*$ satisfying
the relation $\sum_{i=1}^nf_i(a)bg_i(a)=h(a)$.
This algebra is the quotion-algebra of the free $*$-algebra with three
generator
${\Bbb C}<a=a^*, b,b^*>$ with respect to the two-sided $*$-ideal
generated by the element $\sum_{i=1}^nf_i(a)bg_i(a)-h(a)$.

Unless otherwise stated we assume here $f_i$, $g_i$, $h$ to be defined on the
whole ${\Bbb R}$. The general case can be easily derived from this one.

\begin{remark} \rm
As before if the operators $A=A^*$, $B$ are unbounded then it is necessary
to define the meaning of operator equality (\ref{sr}).
We investigate the question on
``correct'' definition of relation (\ref{sr}) with unbounded operators
for some special relations in Ch.~III.
\end{remark}

2. Let us establish a connection between representations of relation
(\ref{sr})
 and the corresponding homogeneous relation:
\begin{equation}\label{hom}
\sum_{i=1}^nf_i(A)Bg_i(A)=0.
\end{equation}

If the function $\phi (t)=h(t)/\sum_{i=1}^nf_i(t)g_i(t)$ is bounded on the
spectrum of an operator $A=A^*$, then the pair of operators $A$,
$B=\phi (A)$ is a particular solution of nonhomogeneous relation
(\ref{sr})
and the general solution of (\ref{sr}) with the fixed operator $A$
consists
of all pairs
($A$, $B+\phi(A)$) where ($A$, $B$) satisfies (\ref{hom}). Let
$$
\tilde\phi(t)=\left\{
\begin{array}{ll}
h(t)/\sum_{i=1}^nf_i(t)g_i(t), &\mbox{if}\
\sum_{i=1}^nf_i(t)g_i(t)\ne 0,\\
0,&\mbox{ otherwise}.
\end{array}\right.
$$

\begin{proposition}
If $(\ref{sr})$ has a (bounded) solution $(A,B)$, then there exists
 $C>0$ such that
$$
|h(t)|\le C|\sum_{i=1}^nf_i(t)g_i(t)|, \quad t\in \sigma(A)
$$
Any solution $A$, $B$ of
$(\ref{sr})$ has the form $B=B'+\tilde\phi(A)$ where ($A$, $B'$) is  a
representation of $(\ref{hom})$.
\end{proposition}
\begin{proof}
Let $A$, $B$ satisfy (\ref{sr}) and  let
$\psi:L(H)\rightarrow {\frak A}$ be the conditional expectation on
the $W^*$-algebra $\frak A$ of all operators which commute with the
operator $A$.
Applying $\psi$ to both hand-sides of equation (\ref{sr}), we obtain
$$
\sum_{i=1}^nf_i(A)\psi (B)g_i(A)=h(A).
$$
>From this we conclude that ($A$, $\psi(B)$) is a solution of (\ref{sr})
and
$$
\psi (B)\sum_{i=1}^nf_i(A)g_i(A)=h(A).
$$
Thus, $\psi(B)=\tilde\phi(A)$
on  the  image  of   the   operator
$\sum_{i=1}^n f_i(A)g_i(A)$.
Since $\psi(B)$ is a bounded operator, there exists $C>0$ such that
for every $t\in\sigma(A)$ and  $\sum_{i=1}^nf_i(t)g_i(t)\ne 0$ we have
$|h(t)|\le C|\sum_{i=1}^nf_i(t)g_i(t)|$. On the other hand,
$\ker \sum_{i=1}^nf_i(A)g_i(A)\subset \ker h(A)$ which implies that the
inequlity holds for every $t\in\sigma(A)$, and
($A$, $B-\tilde\phi(A)$) is a
representation of relation (\ref{hom})
\end{proof}

It is easy to prove that the correspondence between the irreducible
representations of (\ref{sr}) and (\ref{hom}) preseves unitary
equivalence.
So, while we study bounded representations,
we can restrict ourselves by considering only  homogeneous
relations (\ref{hom}).

 3. We associate with  semilinear relation (\ref{hom})

a) the characteristic function:
$$
\Phi(t,s)=\sum_{i=1}^nf_i(t)g_i(s),\qquad (t,s\in {\Bbb R})
$$

b) the characteristic binary relation:
$$
\Gamma=\{(t,s)\mid \Phi(t,s)=0\}\subset {\Bbb R}^2
$$

c) an oriented graph ($D$, $\Gamma$) where an edge
$\edgeright{t}{s}$, $t$, $s\in D$ belongs to the graph if and only
if $\Phi(t,s)=0$.

 If no confusions arise we shall write simply $\Gamma$ for the
graph ($D$, $\Gamma$) and call it the graph of
relation (\ref{hom}).  Any susbset $M\subset {\Bbb R}$ determines
a subgraph $\Gamma|_M$ such that its vertices are points of $M$
and its edges are edges of $\Gamma$ which connect points of $M$.

If $\Phi(t,s)=0$ is equivalent to $\Phi(s,t)=0$, the
graph $\Gamma$ together with the edge $\edgeright{}{}$ also contains the
edge $\edgeleft{}{}$. In this case we shall consider
the graph as  non-oriented.

In what follows $\Gamma_s$ denote the set
$\{(t,s)\mid \Phi(t,s)=0, \Phi(s,t)=0\}\subset {\Bbb R}^2$ or the
corresponding non-oriented graph.

Consider  some examples of semilinear relations.

1) Relations
$ad_A(B)=[A,B]=AB-BA=0$ and $(ad_A)^n(B)=[A,\ldots[A,B]\ldots]=0$ have
charecteristic functions of the form
$\Phi(t,s)=t-s$ and $\Phi(t,s)=(t-s)^n$ respectively and define the same
graph  all connected components of which are following:
$\ear {\lambda}$.

2) Characteristic  functions corresponding to the relations
$ad_{A,-1}(B)=\{A,B\}=AB+BA=0$  and
$(ad_{A,-1})^n(B)=\{A,\ldots\{A,B\}\ldots\}=0$ are
$\Phi(t,s)=t+s$ and $\Phi(t,s)=(t+s)^n$ respectively. As
before, they define the same graph with connected components of
the form:  $$ \ear {0},\qquad \edge {\lambda}{-\lambda},\quad
({\footnotesize\lambda>0}).  $$

3) Let $ABA=\alpha B$, $\alpha\in{\bf R}$. Then $\Phi(t,s)=ts-\alpha$ and
all connected components of the corresponding graph are of the form:

if $\alpha\ne 0$
$$
\edge {\alpha\lambda^{-1}}{\lambda},\quad({\footnotesize
\lambda^2\ne\alpha,0})\qquad \onepoint {0},\qquad
\ear{\pm\sqrt{\alpha}},\quad ({\footnotesize\alpha>0}), $$

if $\alpha=0$, then any vertex $\lambda\in{\Bbb R}$ is connected
with $0$ by the edge $\edge{\lambda}{0}$.

4) Any connected component of the graph corresponding to the relation
$$A^2B-(q+q^{-1})ABA+BA^2=0,\quad q\in{\Bbb
R}\setminus\{-1,0,1\},$$ is an infinite chain:
$$\qquad\chainl{q^{-1}t}{t}{qt}$$

\subsection{Kleinecke-Shirokov type theorems}

We begin with study of structure of
operators $A=A^*$, $B$ satysfying (\ref{hom}), in particular, the
connection
between  spectral properties of the operator $A$ and the structure
of the operator $B$.  It turns out that for a broad class of
semilinear relations the characteristic binary relation
corresponding to them completely defines solutions of (\ref{hom}).
This fact provide many  Kleinicke-Shirokov type
theorems.

1. We start by studying  finite-dimensional
representations of semilinear relations. Note that the same arguments work
in more genaral case when
the operator $A$ in a representation has discrete spectrum. So
let $\sigma (A)=\{\lambda_1,\ldots, \lambda_m\}$ and
$H_{\lambda_j}$ be the eigenspaces of $A$ corresponding to
$\lambda_j$.  With respect to the decomposition
$H=H_{\lambda_1}\oplus \ldots \oplus H_{\lambda_m}$, the operator
$B$ can be written in the form of block matrix:  $$
B=(B_{ln})_{l,n=1}^m $$

\begin{proposition}\label{pr1} For
operators $A$, $B$ to define a representation of relation
$($\ref{hom}$)$ it is  necessary and sufficient that the block
 matrix $B=(B_{sn})_{s,n=1}^m$ is supported by $\Gamma|_{\sigma
(A)}$ (i.e., $B_{sn}=0$ for $(\lambda_s, \lambda_n)\notin
\Gamma)$.  \end{proposition}

\begin{proof}
The statement follows immediatly from the equality
$$
(\sum _{i=1}f_i(A)Bg_i(A))_{kj}=
\Phi (\lambda_k,\lambda_j)B_{kj},\quad k,j=1,\ldots,m.
$$
\end{proof}

2.
A pair $A=A^*$  $B=B^*$ is a representation of relation (\ref{hom}) if and
only if
the block-matrix $B=(B_{ij})_{i,j=1}^n$ is supported by
$$
\Gamma_s|_{\sigma(A)}=\{(t,s)\in\sigma(A)\times\sigma(A)\mid
\Phi(t,s)=\Phi(s,t)=0\} $$ In fact, if $A=A^*$,
$B=B^*$ is a representation of (\ref{hom}), then $A$, $B$ satisfy
additionally the relation
$$
\sum_{i=1}^n\overline{g_i}(A)B\overline{f_i}(A)=0.
$$
Hence the
block-matrix $B=(B_{ij})_{i,j=1}^m$ is supported by
$(\Gamma\cap\Gamma^*)|_{\sigma(A)}$, where
$\Gamma^*=\{t,s)\in {\Bbb R}\times{\Bbb R} \mid \Phi(s,t)
=\sum_{i=1}^ng_i(t)f_i(s)=0\}$ which gives the required statement.

%3.
%Proposition~\ref{pr1} allows us to consider a representation of
%relation (\ref{hom}) as a representation of a corresponding
%oriented $*$-graph such that all points are selfadjoint (see
%\cite {Kr,R}). We recall that a representation of a $*$-graph
%($V$, $G$) is a  mapping assigning to every vertex $\lambda\in V$
%a Hilbert space $H(\lambda)$, and to every edge ($\lambda$,$\mu$)
%an operator $T(\lambda, \mu):H(\lambda)\rightarrow H(\mu)$. A
%subrepresentation of a given representation is definded by a
%collection of subspaces $K(\lambda)\subset H(\lambda)$ such that
%$T(\lambda,\mu)K(\lambda)\subset K(\mu)$.  A representation which
%does  not have non-trivial subrepresentations is called
%irreducible. Similarly, one defines the notion of
%indecomposability.  Representations ($T(\cdot,\cdot)$,$H(\cdot)$)
%and ($T'(\cdot)$, $H'(\cdot)$) are unitarily equivalent if there
%exist unitary operators $R(\lambda):H(\lambda)\rightarrow
%H'(\lambda)$ such that $R(\mu)T(\lambda,
%\mu)=T'(\lambda,\mu)R(\lambda)$.
%
%The problem of unitary classification of irreducible collections
%$A$, $B$, $B^*$ (or indecomposable collections $A$, $B$ which is
%just the same) with spectrum $\sigma(A)\subset M$ is equivalent
%to that of indecomposable representations of the graph
%$\Gamma|_M$ (or irreducible representations of the $*$-graph
%$\tilde {\Gamma}|_M$ which, together with the edge ($\lambda$,
%$\mu$), contains the adjoint edge $(\lambda, \mu)^*$ of opposite
%orientation).
%
%To a restriction imposed on the operator $B$ there corresponds a
%restriction on a form
%of representations of the graph $\Gamma$. For example, to
%representations of (\ref{hom}) with selfadjoint $B$ there
%corresponds a representation of the non-oriented graph $\Gamma_s$
%(the edge ($\lambda$, $\mu$) belongs to $\Gamma_s$ if and only if
%$\Gamma$ contains the edges ($\lambda$, $\mu$) and ($\mu$,
%$\lambda$)). In representations of such a graph, to every edge
%($\lambda$, $\mu$), $\lambda \ne \mu$ there corresponds a pair of
%adjoint operators, and to every loop one selfadjoint operator.
%

4. Now we will try to remove the condition of discreteness of $\sigma(A)$
in
Proposition~\ref{pr1}.
For this purpose we will consider more general situation.

Let $M$, $N$ be normal operators acting on Hilbert spaces $H_M$, $H_N$,
respectively, and let $E_{M}(\cdot)$, $E_N(\cdot)$ be their spectral
measures.
\begin{definition}\label{def1}
We say that a subset ${\cal F}\subset{\Bbb C} \times{\Bbb C} $\quad
$(M,N)$-supports an operator $B:H_N\rightarrow H_M$ if
$$
E_M(\alpha)BE_N(\beta)=0
$$
for any pair ($\alpha,\beta$) of Borel sets such that
$(\alpha\times\beta)\cap{\cal F}=\emptyset$.
\end{definition}
It is not difficult to prove that there exists the smallest closed set
${\cal F}$
supporting $B$ (take the complement to the union of all open
$\alpha\times\beta$).
We will denote this set by $\mbox{supp}_{M,N}(B)$. It is clear that
$\mbox{supp}_{M,N}(B)\subset\sigma(M)\times\sigma(N)$; in particular, it
is
a subset of ${\Bbb R} \times{\Bbb R} $ when $M$ and $N$ are selfadjoint.
We shall also write $\mbox{supp}_M(B)$ instead of
$\mbox{supp}_{M,M}(B)$.

\begin{theorem} \label{th1-semi}
If $A$, $B$ is a representation of relation $(\ref{hom})$, then
$\mbox{\rm supp}_A(B)\subset \Gamma$,
where $\Gamma$ is the binary relation corresponding to $(\ref{hom})$.
\end{theorem}
\begin{proof}
Let $\alpha\times \beta$ do not intersect $\Gamma$. Denote by $\mu$ the
scalar spectral
measure of the operator $A$.
By Luzin's theorem, for every $\varepsilon>0$, there exist compact
sets $\alpha'\subset \alpha$ and $\beta'\subset \beta$ such that
$\mu(\alpha\setminus \alpha')<\varepsilon$, $\mu(\beta\setminus
\beta')<\varepsilon$,
and the functions $f_i'=f_i|_{\alpha'}$,
$g_i'=g_i|_{\beta'}$ are continious. Put $P=E_{A}(\alpha')$,
$Q=E_{A}(\beta')$, $B'=PBQ:QH\rightarrow PH$. The operator of
multiplication:
$$
\Delta: X\rightarrow \sum_{i=1}^nf_i'(A)Xg_i'(A)
$$
acts on the space  $L(QH, PH)$.
The spectrum of the operator $\Delta$ belongs to the set
$$
\{\sum_{i=1}^nf_i(\llll)g_i(\mu):\llll \in \alpha',\mu \in
\beta'\}= \Phi(\alpha'\times \beta') $$
By the condition of
the theorem, $\Phi(\alpha'\times \beta')$ does not contain zero,
hence the operator $\Delta$ is invertible. Since $\Delta(B')=0$, we
have that $B'=0$, i.e., $E_A(\alpha')BE_A(\beta')=0$. Letting
$\varepsilon \rightarrow 0$, we obtain  that $E_A(\alpha)BE_A(\beta)=0$.
\end{proof}

In a similar way, more general results can be proved:
\begin{theorem} \label{th2-semi}
If $M$, $N$ are normal operators and
\begin{equation}\label{p20'}
\sum_{i=1}^nf_i(M)Bg_i(N)=0,
\end{equation}
then
$$\mbox{\rm supp}_{M,N}(B)\subset \Gamma,$$
 where $\Gamma=\{(t,s)\in {\Bbb C} \times{\Bbb C} \mid
 \Phi(t,s)=\sum_{i=1}^n
f_i(t)g_i(s)=0\}$.
\end{theorem}

5. For any $F\subset {\Bbb C} \times{\Bbb C} $ let us denote by ${\frak
M}(F)$
 the set of all operators supported by $F$. If necessary, we shall
write more explicitly: ${\frak M}_A(F)$ or ${\frak M}_{M,N}(F)$.
Similarly, we shall  usually denote by $\Delta$ (instead of
$\Delta_{M,N}$ or $\Delta_A$) the multiplication operator $$
X\rightarrow\sum_{i=1}^nf_i(M)Xg_i(N)
$$
on $L(H_N,H_M)$.

By analogue with Proposition~\ref{pr1} it would be naturally to expect
that the equality
\begin{equation}\label{p20}
{\frak M}(\Gamma)=\ker\Delta.
\end{equation}
holds.
Theorem~\ref{th1-semi} shows that ${\frak M}(\Gamma)\supset\ker\Delta$.
We shall see that the inverse inclusion is true under some
restrictions on the smoothness of the functions $f_i$, $g_i$, and
it is not true in the general case.

Let us establish some additional results. Denote  by $pr_1$, $pr_2$ the
projections onto the components in $\sigma(M)\times\sigma(N)$. Let
$S=\sigma(M)\times\sigma(N)\cap\Gamma$.
\begin{lemma}\label{l1}
Let $g_i\in\mbox{\rm Lip}\sigma(M)$, $k=1,\ldots,n$. If
$pr_1S=\sigma(M)$, then
$$ ||\Delta||\le \sum_{i=1}^n||f_i||
||g_i||_{Lip}\mbox{\rm diam}\sigma(N) $$ where
$||f_i||=sup\{f_i(t)\mid t\in\sigma(M)\}$.  \end{lemma}
\begin{proof}
Set $C=2\sum_{i=1}^n||f_i|| ||g_i||_{Lip}\mbox{diam}\sigma(N)$ and
fix $s_0\in \sigma(N)$. Then $$
\Delta(B)=\sum_{i=1}^nf_i(M)Bg_i(s_0)+\sum_{i=1}^nf_i(M)B(g_i(N)-g_i(s_0)1
).
$$
Since $\sigma(g_i(N)-g_i(s_0)1)=\{g_i(t)-g_i(s_0):t\in
\sigma(N)\}$, we have that $$ ||g_i(N)-g_i(s_0)1||\le
||g_i||_{Lip}\mbox{diam}\sigma(N).  $$ Thus,
$$
||\sum_{i=1}^nf_i(M)B(g_i(N)-g_i(s_0)1)||\le \frac{1}{2}C||B||.
$$
Furthermore,
$\sigma(\sum_{i=1}^ng_i(s_0)f_i(M))=\{\Phi(t,s_0):t\in
\sigma(M)\}$.  By the condition of the theorem,  for any
$t\in\sigma(M)$ there exists $ s=s(t)\in \sigma(N) $ such that
$\Phi(t,s)=0$. Therefore,
\begin{eqnarray*}
|\Phi(t,s_0)|=|\Phi(t,s_0)-\Phi(t,s(t))|= \\
|\sum_{i=1}^n(g_i(s_0)-g_i(s(t)))f_i(s)|\le \frac{1}{2}C.
\end{eqnarray*}
Hence, $||\sum_{i=1}^ng_i(s_0)f_i(M)||\le \frac{1}{2}C$.
Thus,
$$||\sum_{i=1}^nf_i(M)Bg_i(s_0)||\le \frac{1}{2}C||B||,$$ and
$||\Delta (B)||\le C||B||$
\end{proof}
\begin{lemma}\label{l2}
Let $E(\cdot)$ be a projection-valued measure on a compactum $K$, let
$\alpha_1,
\ldots, \alpha_N$ be Borel subsets of $K$ such that the intersection of
any
$m+1$ subsets $\alpha_i$ is empty . Then
$$
\sum_{j=1}^N||E(\alpha_j)\xi||^2\le m||\xi||^2
$$
for any vector $\xi$.
\end{lemma}
\begin{proof}
Since $E(\alpha_j)=\int_K \chi_{\alpha_j}(t)dE(t)$, where
$\chi_{\alpha_j}$
is the characteristic function of $\alpha_j$, we have that
$$
\sum_{j=1}^NE(\alpha_j)=\int_K(\sum_j\chi_{\alpha_j}(t))dE(t)\le
m\int_KdE(t)=m1.
$$
Therefore,
$\sum_{j=1}^N||E(\alpha_j)\xi||^2=\sum_{j=1}^N(E(\alpha_j)\xi,\xi))
\le m||\xi||^2$. \end{proof}
\begin{theorem} \label{cth2}
Let one of the following conditions hold:

(a) $g_i$, $i=1,\ldots,n$ are polynomials;

(b) $g_i\in\mbox{\rm Lip}\sigma(N)$,  $i=1,\ldots,n$, and the
Hausdorff dimension of $\sigma(M)$ is less then $2$.

 Then ${\frak M}(\Gamma)=\ker\Delta$.
\end{theorem}
\begin{proof}  Let $B\in
{\frak M}(\Gamma)$, then $\mbox{supp}_{M,N}B\subset S$.  For any
$\beta\subset \sigma(N)$ we shall denote by $\tilde{\beta}$ the
set
$pr_1(pr_2^{-1}(\beta))$. It is easy to show that
if $\beta$
is closed then so is $\tilde \beta$,
and that
$pr_1(pr_2^{-1}(\cup_{k=1}^{\infty}\beta_k)=\cup_{k=1}^{\infty}\tilde
{\beta_k}$, for any sequence $\{\beta_k\}$. Hence $\tilde{\beta}$
is a Borel set for any ${F}_{\sigma}$-set $\beta$ (here we
consider only such $\beta$).  Since
$(\sigma(M)\setminus\tilde\beta)\times \sigma(N)$ does not
intersect $S$, one has $(1-E_M(\tilde{\beta}))BE_N(\beta)=0$.

Let $\varepsilon>0$ and let
$\displaystyle\sigma(N)=\cup_{j=1}^K\alpha_j$, where
$\alpha_i\cap\alpha_j=\emptyset$,
$\mbox{diam}\alpha_j<\varepsilon$ and $K=K(\varepsilon)$ is
supposed to be the least possible for all such decompositions.
Then
\begin{equation}
\Delta(B)=\sum_{j=1}^K\Delta(B)E_N(\alpha_j)=
\sum_{j=1}^KE_M(\tilde\alpha_j)\Delta(B)E_N(\alpha_j).
\end{equation}
Set $\gamma=\{t\in \sigma(M):\sum_{i=1}^nf_i(t)g_i(N)=0\}$.
Then
\begin{equation}\label{p21}
E_M(\gamma)\Delta=0.
\end{equation}
In fact, for any $Y\in L(H_N, H_M)$, we have
$E_M(\gamma)\Delta(Y)=\sum_{i=1}^nE_M(\gamma)f_i(M)Yg_i(N)$
and
\begin{eqnarray*}
&&(E_M(\gamma)\Delta(Y)\xi,
\nu)=\sum_{i=1}^n(Yg_i(N)\xi,
E_M(\gamma)(f_i(M))^*\nu)=\\
&&=\sum_{i=1}^n\int_{\gamma}(Yg_i(N)\xi,
\overline{f_i(t)}dE_M(t)\nu)= \int_{\gamma}(dE_M(t)\nu,
\sum_{i=1}^nYg_i(N)f_i(t)\xi)=0,
\end{eqnarray*}
for any $\xi\in
H_N$, $\nu\in H_M$.

Relation (\ref{p21}) implies
$\Delta (B)=\sum_{j=1}^KE_M(\hat{\alpha}_j)\Delta (B)E_N(\alpha_j)$,
where $\hat{\alpha}_j=\tilde\alpha_j\setminus\gamma$. Hence,
\begin{eqnarray}
|(\Delta(B)\xi, \nu)|=|\sum_{j=1}^K(E_M(\hat{\alpha}_j)
\Delta (B)E_N(\alpha_j)\xi, E_M(\hat{\alpha}_j)\nu)|\le \nonumber \\
\le(\sum_{j=1}^K||E_M(\hat{\alpha}_j)\Delta (B)E_N(\alpha_j)\xi||^2)^{1/2}
(\sum_{j=1}^K||E_M(\hat{\alpha}_j)\nu ||^2)^{1/2}.\label{p22}
\end{eqnarray}
Since $\hat{\alpha}_j\subset pr_1((\hat\alpha_j\times\alpha_j)\cap
S)$, we can apply Lemma~\ref{l1} to the operators
$NE_N(\alpha_j)$, $ME_M(\hat{\alpha}_j)$. Thus, $$
||E_M(\hat{\alpha}_j)\Delta (B)E_N(\alpha_j)||\le
C||B||\mbox{diam}\alpha_j
\le C||B||\varepsilon,
$$
$$
||E_M(\hat{\alpha}_j)\Delta (B)E_N(\alpha_j)\xi||\le
C||B||\varepsilon ||E_N(\alpha_j)\xi||,
$$
$$
\sum_{j=1}^K||E_M(\hat{\alpha}_j)\Delta (B)E_N(\alpha_j)\xi||^2
\le C^2||B||^2\varepsilon^2||\xi ||^2,
$$
since the sets $\alpha_j$ are mutually disjoint.

Let first all $g_i(\cdot)$ be polynomials and let $m$ be the
greatest degree of the polynomials $g_i(\cdot)$.  Then, for any
$t\in\sigma(M)\setminus\gamma$, the equation $\Phi(t,s)=0$ does
not have more then $m$ roots. Thus, $t$ can not belong to more
than $m$ sets $\hat{\alpha}_j$  because for any $t\in
\hat{\alpha}_j$ there exists $s(t)\in \alpha_j$ such that
$\Phi(t,s(t))=0$, and the sets $\alpha_j$ are disjoint.  Applying
Lemma~\ref{l2}, we can conclude that
$\sum_{j=1}^K||E_M(\hat{\alpha}_j)\nu||^2\le m||\nu||^2$.  Hence,
$|(\Delta (B)\xi, \nu)|\le C||B||\varepsilon ||\xi||m^{1/2}||\nu
||^2$.
Letting $\varepsilon\rightarrow 0$, we obtain  $\Delta (B)=0$.

Let now the second condition hold. We can estimate the second
factor in the right-hand side of (\ref{p22}) as follows:  $$
(\sum_{j=1}^K||E(\hat{\alpha}_j)\nu||^2)^{1/2}\le (K||\nu ||^2)^{1/2}=
K^{1/2}||\nu ||
$$
Thus $||\Delta(B)||\le C||B||\varepsilon K^{1/2}$
($K=K(\varepsilon)$ depends on $\varepsilon$).  Since $\mbox{dim}
\sigma(M)<2$, $K(\varepsilon)=o(\varepsilon^{-2})$, this implies
$\Delta(B)=0$. \end{proof}
\begin{remark}\rm
The second
restriction on $\mbox{dim}\sigma(M)$ in Theorem~\ref{cth2} could
be removed, but the proof would become much more
complicated, so we omit this generalization.  \end{remark}

6. As a corollary from Theorem~\ref{cth2} we obtain the following
Kleinicke-Shirokov type theorems.

\begin{theorem} \label{c1}
If two polynomial semilinear relations have the same graph, then
their representations coinside.  \end{theorem}

\begin{example}
By Theorem~\ref{c1}, the semilinear relations $(ad_{A,q})^n(B)=0$ and
$ad_{A,q}(B)=AB-qBA=0$ have the same representations by bounded operators
$A=A^*$, $B$, since the characteristic functions $\Phi(t,s)=t-qs$ and
$\Phi(t,s)=(t-qs)^n$ corresponding to the first and the second
relations respectively define the same graph.
In particular, for $q=1$ we obtain  that bounded operators
  $A=A^*$, $B$ satisfy the relation
  $(ad_A)^n(B)=[A,\ldots[A,B]\ldots]=0$ if and only if
  $A$, $B$ commute, and for $q=-1$, a pair of bounded
  operators $A=A^*$, $B$ is a solution of the relation
  $(ad_{A,-1})^n(B)=\{A,\ldots\{A,B\}\ldots\}=0$ if $AB=-BA$.
  \end{example}

7. As corollaries from Theorem~\ref{cth2} we shall also give some
  results, where the form of characteristic binary relation
 corresponding  to semilinear relation (\ref{hom}) is more
important then the form of the functions $f_i$, $g_i$.
\begin{theorem}\label{c2} Let $f$, $g$ be bounded Borel functions.
A pair of bounded operators $A=A^*$, $B$ satisfies the equation $$
f(A)B=Bg(A) $$ if and only if $\mbox{\rm supp}_A(B)\subset\Gamma$,
where $\Gamma=\{(t,s)\mid f(t)=g(s)\}$.  \end{theorem}
\begin{proof} Set $M=f(A)$, $N=g(A)$.  Since
$E_M(\alpha)=E_A(f^{-1}(\alpha))$ for any Borel set $\alpha$, the
condition $\mbox{supp}_A(B)\subset\Gamma$ is equivalent to the
following one $$ \mbox{supp}_{M,N}(B)\subset\{(t,s)\mid t=s\}.  $$
Theorem~\ref{cth2} gives the required statement. \end{proof}

In a similar way one can show that  Theorem~\ref{cth2} implies
the Fuglede--Putnam--Rozenblum theorem (see 2.4).
In particular, if bounded operators $A=A^*$, $B$ satisfy  the relation
$AB=qBA$, where $|q|=1$, then $AB=\overline qBA$, since
the sets $\{(t,s)\in{\Bbb R}\times{\Bbb R}\mid t=qs\}$  and
$\{(t,s)\in{\Bbb R}\times{\Bbb R}\mid t=\overline qs\}$ coincide.

\begin{remark}\rm
Sets of the form $\{(t,s)\mid f(t)=g(s)\}$, where $f$, $g$ are bounded,
Borel
 functions are called rectangular sets. For them, much more stronger
 result is true: an operator supported by a rectangular set $F$ satisfies any
relation whose characteristic binary relation  contains $F$.
The proof is very complicated and we
restrict ourselves to a more special result.
\end{remark}

\begin{theorem}\label{gr}
Let $F$ be the graph of a function, that is, $F=\{(t,s)\mid
s=\varphi(t)\}$, where
$\varphi$ is a bounded, Borel function. If $\mbox{\rm supp} _A(B)\subset
F$, then
the pair $A$, $B$ gives a representation of any relation whose
binary relation contains $F$.
\end{theorem}
\begin{proof} Let $F\subset\Gamma=\{(t,s)\mid\sum_{i=1}^nf_i(t)g_i(s)=0\}$.
Since
$$
\mbox{supp}_A(B)\subset \{(t,s)\mid s=\varphi(t)\}\subset
\{(t,s)\mid g_i(s)= g_i(\varphi(t))\},
$$
we have
$Bg_i(A)=g_i(\varphi(A))B$, $i=\overline{1,n}$, by Theorem~\ref{c2}. On
the
other hand,
$$\sum_{i=1}^nf_i(t)g_i(\varphi(t))=0$$ for any $t\in \sigma(A)$,
hence $\sum_{i=1}^nf_i(A)g_i(\varphi(A))=0$. It follows that
$$
\sum_{i=1}^nf_i(A)Bg_i(A)=\sum_{i=1}^nf_i(A)(Bg_i(A)-g_i(\phi(A))B)=0.
$$
\end{proof}

\begin{corollary}\label{c4}
Let $A=A^*$, and let  there  exist a decomposition of $\sigma(A)$ into
Borel
sets $P_i$, $\sigma(A)=\cup_{i}P_i$, such that each $P_i\times
P_j\cap\Gamma$,
$i,j=1,2,\ldots$, is the graph of a function. Then $\ker
\Delta_A={\frak M}_A(\Gamma)$.
\end{corollary}
\begin{proof}
It follows from Theorem~\ref{gr} that $E_A(P_i)\Delta
(B)E_A(P_j)=0$ for any $i,j$, hence  $\Delta (B)=0$.\end{proof}

\begin{remark}\rm
Let us note that if $\Gamma$ is such that  the set $\{\mu\in D\mid (\llll,
\mu)\in \Gamma\}$ is
finite or countable for any $\llll \in D$, then the condition of
Corollary~\ref{c4} is satisfied.
\end{remark}

8. As it was noticed, the converse of Theorem~\ref{th2-semi} is
false in the general case. Here we construct an example that
simultaneously solve the Fuglede-Weiss problem. Namely, we show that
there exist a pair of bounded operators $A=A^*$, $B$ and
continuous functions $f_i$, $g_i$ such that
$$
\sum_{i=1}^mf_i(A)Bg_i(A)=0,\ \
\mbox{but} \ \ \sum_{i=1}^m\overline{f}_i(A)B\overline{g}_i(A)\ne 0
$$

\begin{example}\rm\label{ex}
Let $D({\Bbb R}^n)$ be the space of compactly supported infinitely
differentiable
functions on ${\Bbb R} ^n$, $D'({\Bbb R}^n)$ its dual space,
$FL^1({\Bbb R} ^n)$  the Fourier algebra,
$PM({\Bbb R} ^n)$ the space dual to $FL^1({\Bbb R} ^n)$ (the space
of pseudomeasures), $F$  the Fourier transform, $\varphi*\psi$ the
convolution of two functions in $D({\Bbb R} ^n)$, $\tilde
\varphi(x)=\overline{\varphi(-x)}$.

Consider the following polynomial in six variables
$$
p(x_1,\ldots, x_6)=x_1^2+x_2^2+x_3^2-1+i(x_4^2+x_5^2+x_6^2-1)
$$
Clearly, there exist polynomials $s_i$, $r_i$, $i=1,\ldots,m$, such
that $p(x-y)=\sum_{i=1}^ms_i(x)r_i(y)$ for any
$x,y\in {\Bbb R} ^6$. Let $u,v\in D({\Bbb R} ^n)$ and $a_i=us_i$,
$b_i=vr_i$.
Obviously, $a_i$, $b_i\in D({\Bbb R}^6)$. Consider in the space
$L^2({\Bbb R} ^6)$ the
operators $A_i=M_{a_i}$, $B_i=M_{b_i}$, $i=1,\ldots,m$ ($M_f$ is
the operator of multiplication by the function $f$). Since the Fourier
transform
of a pseudomeasure $\Phi$ belongs to $L^{\infty}({\Bbb R} ^6)$,
the operator $T=F^{-1}M_{F\Phi}F$ is well defined in the
space $L^2({\Bbb R}^6)$.
Furthermore a direct computation shows that
\begin{equation}
(\sum_{i=1}^mM_{b_i}TM_{a_i}\varphi,\psi)=<p\Phi,u\varphi*
\tilde{\overline{v}\psi}>
\end{equation}
for any $u,v,\varphi,\psi\in D({\Bbb R}^6)$, where $<\cdot,\cdot>$ is the
pairing of
the spaces $D({\Bbb R}^6)$, $D'({\Bbb R}^6)$
(since $PM({\Bbb R} ^6)\subset D({\Bbb R} ^6)'$,
$p\Phi\in D'({\Bbb R}^6)$ for any
polynomial $p$
in six variables).
Since the set
$L=\{u\varphi*\tilde{\overline{v}\psi}\mid \varphi,
\psi,u,v\in D({\Bbb R}^6)\}$ is dense in $D({\Bbb R}^6)$,
the existence of a
pseudomeasure $\Phi$ such that $p\Phi=0$ and $\overline{p}\Phi\ne0$,
would imply the existence of functions $u$, $v$ such that
\begin{eqnarray*}
(\sum_{i=1}^mM^*_{b_i}TM^*_{a_i}\varphi, \psi)=
<\overline{p}\Phi,
\overline{u}\varphi*\tilde{v\psi}>\ne 0\quad \mbox{for
some $\varphi,\psi\in D({\Bbb R}^6)$}, \\
(\sum_{i=1}^nM_{b_i}TM_{a_i}\varphi,\psi)=<p\Phi,u\varphi*\tilde{\overline
{v}\psi}>=0
\quad \mbox{for all}\quad \varphi,\psi\in D({\Bbb R}^6).
\end{eqnarray*}
Hence, for the operator $A_i$, $B_i$, $T$ constructed relatively
to $\Phi$, $u$, $v$, we  obtain
\begin{equation}\label{prp}
\sum_{i=1}^mB_iTA_i=0 \qquad \mbox{and} \qquad
\sum_{i=1}^mB_i^*TA_i^*\ne 0 \end{equation}
Since any finite,
commutative family of normal operators can be realized as a family
of continuous functions of one selfadjoint operator, one can find
a selfadjoint operator $A$ and continuous functions $f_i$, $g_i$
such that $B_i=f_i(A)$, $A_i=g_i(A)$, and (\ref{prp}) is
equivalent to the following $$ \sum_{i=1}^mf_i(A)Tg_i(A)=0
\qquad\mbox{and}\qquad
\sum_{i=1}^m\overline{f}_i(A)T\overline{g}_i(A)\ne 0
$$
Thus, it remains to prove the existence of a pseudomeasure $\Phi$ such
that $p\Phi=0$, $\overline{p}\Phi\ne 0$. In order to do this, we consider
the $\delta$-measure
$\mu_0$ of the surface $S^2$ in
${\Bbb R} ^3$. Let $\mu=\mu_0\times\mu_0$. It is easy to show that
$$
|F\mu(\llll)|=|\int_{{\Bbb R} ^6}e^{i(\llll,x)}d\mu(x)|=O(|\llll |^{-1})
$$
as $|\llll |\rightarrow +\infty$ in ${\Bbb R} ^6$.

\noindent Since $F(\frac{\partial}{\partial x_i}f)=\llll _iFf$,
the Fourier transform $F(\frac{\partial\mu}{\partial x_i})$ belongs to
$L^{\infty}({\Bbb R} ^6)$, so that $\frac{\partial\mu}{\partial x_i}\in
PM({\Bbb R} ^6)$. The pseudomeasure $\frac{\partial\mu}{\partial x_i}$ has
a compact support. Thus $L\mu\in PM({\Bbb R} ^6)$ for any first-order
differential operator $L$ with smooth coefficients. Let
$$
L=(1+i)x_4\frac{\partial}{\partial x_1}-(1-i)x_1\frac{\partial}{\partial
x_4},
\quad\mbox{and} \quad \Phi=L\mu.
$$
It is easy to see that $Lp=0$ and $L\overline{p}=4(1+i)x_1x_4$, which
implies that for
any function $\varphi$ in $D({\Bbb R}^6)$,
\begin{eqnarray*}
<p\Phi,\varphi>=<L\mu,p\varphi>=<\mu,L(p\varphi)>=<\mu,pL(\varphi)>=0, \\
<\overline{p}\Phi,\varphi>=<\mu,L(\overline{p})\varphi>+<\mu,\overline{p}L
\varphi>=
4(1+i)<x_1x_4\mu,\varphi>.
\end{eqnarray*}
Thus, $p\Phi=0$ and $\overline{p}\Phi=4(1+i)x_1x_4\mu\ne 0.$
\end{example}
\begin{remark}\rm
Instead of one operator $A=A^*$ one can consider a family of commuting
selfadjoint operator operators  ${\Bbb A}=(A_k)_{k=1}^m$ which are
connected
with a bounded operator $B$ by a semilinear relation of the form
\begin{equation}\label{mul}
\sum_{i=1}^n f_i({\Bbb A})Bg_i({\Bbb A})=0,
\end{equation}
where  $f_i({\Bbb A})$, $g_i({\Bbb A}$ are Borel bounded functions
of the family ${\Bbb A}$. As before, one can associate to relation
(\ref{mul}) the set $\Gamma=\{(t,s)\in {\Bbb R} ^m\times{\Bbb R}
^m \mid \sum_{i=1}^n f_i(t)g_i(s)=0\}$. Support of the operator
$B$ ($\mbox{supp}_{\Bbb A}B\subset \sigma ({\Bbb
A})\times\sigma({\Bbb A})$) with respect to the family ${\Bbb A}$
is defined in the same way as in Definition~\ref{def1} but with
the joint resolution of identity $E_{\Bbb A}(\cdot)$ for the family
${\Bbb A}$ instead of $E_{M}(\cdot)$ and $E_{N}(\cdot)$.  If
${\frak M}_{\Bbb A}(\Gamma)$ is the set of all bounded operators
whose support belongs to $\Gamma$ and $\ker\Delta_{\Bbb A}$ is the
operator of multiplication $X\rightarrow \sum_{i=1}^n f_i({\Bbb
A})Bg_i({\Bbb A})$, then using the similar
arguments given in 2.3.3, 4, one can prove that $\ker\Delta_{\Bbb
A}\subset{\frak M}_{\Bbb A}(\Gamma)$. But the converse inclusion
is not true already for $m$ greater than 2 and polynomials $f_k$,
$g_k$. The corresponding example can be derived using similar
arguments given in Example~\ref{ex}.  \end{remark}

\begin{remark}\rm
As has been shown above, an operator $B$, $A$-supported by the set
$\Gamma=\{(t,s)\mid\sum_{i=1}^nf_i(t)g_i(s)=0\}$, might not  satisfy the
corresponding
semilinear relation $\sum_{i=1}^nf_i(A)Bg_i(A)=0$ if the functions
$f_i(\cdot)$,
$g_i(\cdot)$ are not smooth. But this is true  if the operator $B$ is
pseudointegral. Let us recall the definition of
a pseudointegral operator.

Let $\mu$ be the scalar spectral measure of selfadjoint operator $A$,
let $H=\int_{\sigma(A)}^\oplus l_2(N(\llll))d\mu(\llll)$,
$N(\llll)\in{\bf N} \cup\{\infty\})$, be the spectral resolution
of  $A$, and let $m$ be a bounded measure on ${\Bbb R} \times
{\Bbb R} $ such that the projections of $m$ on  both coordinates
are majorized by $\mu$. The measure $m$ is said to be  regular.
Since $l_2(N)$ for $N\in {\Bbb N}$ can be embedded into
$l_2(\infty)$, we can define
the bilinear form
$$
(\vec{x}, \vec{y})\rightarrow \int\int (\vec{x}(t),\vec{y}(s))dm,
$$
where $(\vec{x}(t),\vec{y}(s))$ is the scalar product in $l_2(\infty)$.
If $m$ is regular,
then the bilinear form defines some bounded operator $T_m$.
The operator $T_m$ is called a ``pseudointegral operator constructed
relatively
to $m$''.

It was shown by Arveson that the pseudointegral operator $T_m$ is
supported by any pseudoclosed set on which the measure $m$ is
concentrated.
(The set $E\subset{\Bbb R} \times{\Bbb R} $ is called pseudoclosed if its
complement is a union of countably many measurable sets of the form
$X\times Y\subset{\Bbb R} \times{\Bbb R} $).

Let $\Gamma$ be the characteristic binary relation corresponding to
semilinear relation (\ref{hom}),
and $m$ a regular measure with $\mbox{supp}\  m\subset\Gamma$. Then,
$A$, $T_m$ is a representation of relation (\ref{hom}). Indeed,
\begin{eqnarray*}
&&(\sum_{i=1}^nf_i(A)T_mg_i(A)\vec{x},\vec{y})=\sum_{i=1}^n(T_mg_i(A)\vec{
x},
\overline{f_i}(A)\vec{y})=\\
&&=\sum_{i=1}^n\int\int_{\Gamma}g_i(t)f_i(s)(\vec{x}(t),\vec{y}(s))dm=
\int\int_{\Gamma}\Phi(t,s)(\vec{x}(t),\vec{y}(s))dm=0.
\end{eqnarray*}

A subset $F\subset{\Bbb R} \times {\Bbb R} $ is called  marginally null
with
respect to a measure $\mu$ (or with respect to a class of measures
equivalent to $\mu$) if
$F\subset
(\alpha\times {\Bbb R} )\cup({\Bbb R} \times \beta)$, where $\mu(\alpha)=
\mu(\beta)=0$.

In what follows we will need the Arveson theorem on null sets:
\begin{theorem}\label{A} A pseudoclosed set carries a regular
measure $m$ if and only if it is not marginally null with respect
to $\mu$.  \end{theorem} \end{remark}

\subsection{Irreducible  re\-pre\-sen\-ta\-tions of se\-mi\-li\-near
re\-la\-tions}

Now we study irreducible representations of semilinear relations, i.e.,
irreducible family of operators $A=A^*$, $B$, $B^*$ satisfying
$$\sum_{i=1}^nf_i(A)Bg_i(A)=0.$$
It is clear that if $f_i$, $g_i$ are polynomials then such family of
operators defines an irreducible representation of the
$*$-algebra   generated by $a=a^*$, $b$, $b^*$ and the relation
$\sum_{i=1}^nf_i(a)bg_i(a)=0$.
In what follows we shall mean by representation of semilinear relation
a triple ($A=A^*$, $B$, $B^*$) satisfying the semilinear relation instead
of pair ($A=A^*$, $B$), if it does not lead to any confusion.

1. We begin with  finite-dimensional re\-pre\-sen\-ta\-tions
and establish a connection between irreducible representations
of (\ref{hom}) and the corresponding graph.
\begin{proposition}\label{connec}
If a family ($A$, $B$, $B^*$)  is irreducible and define a
finite-dimensional representation of (\ref{hom}), then the corresponding
graph $\Gamma|_{\sigma(A)}$ is
connected. For every finite connected subgraph ($D$, $\Gamma|_{D}$),
there exists an irreducible representation $A$, $B$, $B^*$ with
$\sigma(A)=D$.
\end{proposition}
\begin{proof} In fact, if $\Gamma|_{\sigma(A)}=\Gamma_1\cup\Gamma_2$ is
a union of two connected subgraphs then, by Theorem~\ref{th1-semi},
the spectral subspaces $A$ corresponding to vertices of $\Gamma_1$ and
$\Gamma_2$ are invariant with respect to $B$. This shows that $A$, $B$
$B^*$
is reducible.

Let $\Gamma|_{\sigma(A)}$ be connected. The family of
operators $A$, $B$, $B^*$ defined by
\begin{eqnarray*} A=\left( \begin{array}{lll} \llll _1 &        &
0\\ & \ddots &  \\ 0      &        & \llll _m \end{array} \right),
\qquad
 \{\lambda_1,\ldots,\lambda_m\}=D,\quad \llll _i\ne \llll _j,
\mbox{for} \ i\ne j,\\
B=(b_{ij})_{i,j=1}^m
\qquad
b_{ij}=\left\{
\begin{array}{ll}
 0, & ( \llll_i,\llll_j)\notin \Gamma|_D\\
 1, & ( \llll_i,\llll_j)\in \Gamma|_D
\end{array} \right.
\end{eqnarray*}
is irreducible. Indeed, the relation $[C,A]=0$ implies that $C$ is
diagonal
 ($C=\mbox{diag}(c_1,\ldots,c_2)$). From $[C,B]=0$, it follows that
$c_kb_{kl}=b_{kl}c_l$. Since $\Gamma|_{\sigma(A)}$ is connected, we have
that
there exists a permutation $(l_1,\ldots,l_m)\in S_m$ such that
$(\lambda_{l_k},\lambda_{l_{k+1}})\in\Gamma|_{\sigma(A)}$, hence
$b_{l_k,l_{k+1}}=1$. This implies that $c_1=\ldots=c_m$, and $C=c_1I$.
Moreover, we have $\sigma(A)=D$.
\end{proof}

We give also the reformulation of this statement for the symmetrical case.
\begin{proposition}
If $A=A^*$, $B=B^*$ is an irreducible finite-dimensional representation of
$(\ref{hom})$,
then the graph $\Gamma_s|_{\sigma(A)}$ is
connected. For every finite connected subgraph ($D$,
$\Gamma_s|_{\sigma(A)}$),
there exists an irreducible pair $A=A^*$, $B=B^*$  satisfying
$(\ref{hom})$
such that $D=\sigma(A)$.
\end{proposition}
The proof is the same as the one given above, but with $\Gamma_s$ instead
of
$\Gamma$. Note that the constructed operator $B$ is selfadjoint because
the graph $\Gamma_s$ is symmetrical.

2. In what follows, we investigate irreducible
representations of (\ref{hom}) on a Hilbert space  $H$, where $H$
is not assumed to be finite-dimensional. First, we prove an
analogue of the theorem on connectedness of the  graph
(Proposition~\ref{connec}). For this purpose, we recall some
definitions.

\begin{definition}
A subset $F\subset{\Bbb R}\times {\Bbb R}$ is called  marginally null
with
respect to  a measure $\mu$ (or a class of measures equivalent to $\mu$)
if
$F\subset
(\alpha\times {\Bbb R})\cup({\Bbb R}\times \beta)$, where $\mu(\alpha)=
\mu(\beta)=0$.
\end{definition}
Note that to every spectral measure there corresponds a class  of scalar
measures equivalent to it (a spectral type).

Let, further,
\begin{eqnarray*}
&\Gamma(M)=\{y\mid\exists x\in M: (x,y)\in \Gamma\},\\
&\Gamma^{-1}(M)=\{x\mid\exists y\in M: (x,y)\in \Gamma\},
\end{eqnarray*}
and let $M^{c}$ be the complement of the set $M$. In what
follows we shall assume that $\Gamma(M)$ and $\Gamma^{-1}(M)$
are Borel for any Borel set $M$. The concepts defined below
generalize those of quasiinvarience and ergodicity of a measure.

\begin{definition} We call a set $M\subset{\Bbb R}$ left-invariant
(right-invariant) with respect to $\Gamma$ and a measure $\mu$ if
the set $M^{c}\times M\cap\Gamma$ $($respectively $M\times
M^{c}\cap\Gamma$$)$ is marginally null; $M$ is invariant if it is
left- and right-invariant. A measure $\mu$ is called (left-,
right-) quasiinvariant with respect to $\Gamma$ if the set
$\Gamma(M)\cup\Gamma^{-1}(M)$ $($respectively $\Gamma^{-1}(M)$,
$\Gamma(M)$$)$ is of non-zero measure for every Borel set $M$ such
that $\mu(M)\ne 0$, $\Gamma(M)$, $\Gamma^{-1}(M)\in {\frak
B}({\Bbb R})$. A spectral type $\mu$ is called (left-, right-)
ergodic with respect to $\Gamma$ if any (left-, right-) invariant
set is a $\mu$-null set or has the $\mu$-null complement.
\end{definition}

Note that for any operator $A=A^*$ there exists the trivial representation
($A$,0,0) of the relation (\ref{hom}). Therefore, it is natural to
consider
representations without trivial parts.
\begin{definition}
We call a representation $A$, $B$, $B^*$
$*$-complete if for any non-zero spectral subspace $W$ of the
operator $A$ one of the operators $B$ and $B^*$ is not equal to
zero on $W$.
\end{definition}
It  is easy to show that any irreducible representation of
dimension greater than 1  is $*$-complete.
\begin{proposition}\label{pr4}
If a representation $(A$, $B$,
$B^*)$ of $(\ref{hom})$ is $*$-complete, then the spectral measure
of the operator $A$ is quasiinvariant with respect to $\Gamma$.
Conversely, if the spectral measure of $A$ is quasiinvariant, then
 there exists a bounded operator $B$ such that  $(A$, $B$, $B^*)$
is a $*$-complete representation of $(\ref{hom})$.
\end{proposition}
\begin{proof}
If $E_A(M)\ne 0$ but $E_A(\Gamma^{-1}(M))=0$ and
$E_A(\Gamma(M))=0$ then $$BE_A(M)= E_A(\Gamma^{-1}(M))BE_A(M)=0$$
and $$B^*E_A(M)=E_A(\Gamma(M))B^*E_A(M)=0,$$
hence $B|E_A(M)H=0$
and $B^*|E_A(M)H=0$. Thus, the representation ($A$, $B$, $B^*$) is
not $*$-complete.

Conversely, let $A$ be a selfadjoint operator, $\mu$ its spectral scalar
measure, and $U$ the space of measures which are regular with
respect to $\mu$ and concentrated on $\Gamma$. Then $U$ contains
a measure $m$ such that all measures from $U$ are absolutely
continuous with respect to $m$ (in $U$ there is a countable dense
subset $\{\mu_n:n\ge 1\}$, and we can put $m=\sum_n\frac{1}
{2^n}\mu_n$).  Consider now a representation $A$, $T_m$, where
$T_m$ is a pseudointegral operator constructed relatively to $m$.
Let $\mu$ be quasiinvariant, $\mu (M)\ne 0$, and assume that
$T_mE_A(M)=0$ and $T_m^*E_A(M)=0$.  Then $m({\Bbb R}\times M)=0$
and $m(M\times {\Bbb R})=0$.  Hence the sets ${\Bbb R}\times M$,
$M\times {\Bbb R}$ do not carry measures from $U$. By
Theorem~\ref{A} ${\Bbb R}\times M\cap\Gamma$ is
marginally null, i.e., ${\Bbb R}\times
M\cap\Gamma\subset(M_1\times{\Bbb R})\cup({\Bbb R}\times M_2)$,
where $\mu(M_1)=\mu(M_2)=0$, hence $\mu(\Gamma^{-1}(M\setminus
M_2))=0$ and $\mu$ is not left-quasiinvariant. The same arguments
shows that $\mu$ is also not right-quasiinvariant which imply a
contradiction.  \end{proof}
\begin{theorem}\label{therg}
If a representation ($A$, $B$, $B^*$) of relation (\ref{hom})
is irreducible, then the spectral measure of $A$ is ergodic with
respect to $\Gamma$. For any ergodic measure $\mu$ there exists an
irreducible  representation $A$, $B$, $B^*$ of relation
(\ref{hom}) such that $\mu$ is a spectral scalar measure of $A$.
\end{theorem}
\begin{proof} Let ($A$, $B$, $B^*$) be an irreducible representation of the
relation (\ref{hom}).
If the spectral type of $A$ is not ergodic, then there exists
$M\subset{\Bbb R}$ such that $E_A(M)\ne 0$, $E_A(M)\ne 1$, and
the sets $\Gamma\cap(M^{c}\times M)$, $\Gamma\cap(M\times M^c)$ are
marginally null sets. Hence,
$\Gamma\cap
(M^{c}\times M)\subset(M_1\times{\Bbb R})\cup({\Bbb R}\times M_2)$, where
$\mu(M_1)=\mu(M_2)=0$. From this it follows  $E_A(M^{c})BE_A(M)=
E_A(M^{c}\setminus M_1)BE_A(M\setminus M_2)=0$,
i.e., the spectral subspace  $E_A(M)H$  is invariant with respect
to $A$, $B$. In the same way one can show that $E_A(M)H$ is invariant
with respect to $A$, $B^*$.
This contradicts the irreducibility of $A$, $B$, $B^*$.

Conversely, let $\mu$ be an ergodic measure on ${\Bbb R}$, $A$ the operator
of
multiplication by independent variable in $H=L^2(\mu)$, $B$ the
pseudointegral
operator just as in the previous theorem. Then the family $A$, $B$, $B^*$
is
irreducible.
Indeed, if $L=E_A(M)H$ is invariant with respect to $A$, $B$, $B^*$, then
$E_A(M^c)BE_A(M)=0$ and $E_A(M^c)B^*E_A(M)=0$. From the definition of
pseudointegral operator it
follows that $m(M^c\times M)=0$ and $m(M\times M^c)=0$. Hence, by
Theorem~\ref{A}, $M^c\times M\cap
\Gamma$, $M^c\times M\cap\Gamma$ are marginally null and we can
conclude that  $M$ is invariant. This contradicts the
ergodicity.  \end{proof}

3.
Let ($A$, $B$) satisfy the following semilinear relation:
\begin{equation}\label{mk}
AB=BF(A),
\end{equation}
where $F(\cdot)$ is a bounded Borel mapping defined on ${\Bbb R}$.
It is easy to show that in this case the invariance of a set
$\Delta\in {\frak B}({\Bbb R})$ with respect to  $\Gamma$ and a
measure $\mu$ means its invariance with respect to the mapping
$F(\cdot)$, i.e., there exist $\alpha_1$, $\alpha_2$,
$\mu(\alpha_i)=0$, $i=1,2$ such that
$\mu(F^{-1}(\Delta\setminus\alpha_1)\setminus\Delta)=0$ and
$\mu(F(\Delta\setminus\alpha_2)\setminus\Delta)=0$.  The
ergodicity of the spectral measure with respect to $\Gamma$ means
the ergodicity of the measure $E_A(\cdot)$ with respect to
$F(\cdot)$ (i.e., for any $F(\cdot)$-invariant Borel set
$\Delta\subset{\Bbb R}$, either $E_A(\Delta)=0$ or
$E_A(\Delta)=I$).  Therefore, by Theorem~\ref{therg}, we have
that, if ($A$, $B$, $B^*$) is an irreducible representation of
($\ref{mk}$), then  $E_A(\cdot)$ is ergodic with respect to
$F(\cdot)$.

4. In  analysis of the representations of (\ref{mk}), the behaviour of
the dynamical system
$({\Bbb R}, F)$ play a central role.
The structure of representations of (\ref{mk}) depends on the structure of
the orbits
of the corresponding dynamical system (see 2.5 for details). Here we
give the corresponding
results for general semilinear relations.
We begin with the following definition.

\begin{definition}
A set $E$ is called $\Gamma$-invariant if
$E\times E^c\cap\Gamma=\emptyset$ and $E^c\times E\cap\Gamma=\emptyset$.
A minimal $\Gamma|_M$-invariant set $O_M(E)$
containing the set $E$
is said to be a trajectory (semitrajectory) of the set
$E\subset M$ with respect to $\Gamma|_M$.
\end{definition}
The concept of a trajectory generalizes that of an orbit for
dynamical systems. The following result is an analogue of the theorem on
connectedness of the graph supporting an irreducible
finite-dimensional representation.
\begin{theorem} Let $M$ be a
compact set.

(a) If there is $x\in M$ such that the trajectory $O_M(\{x\})$
 is dense in $M$ then $M$ is the spectrum
of an irreducible representation.

(b) If $M$ is the spectrum of irreducible
representation of (\ref{hom}), then the trajectory $O_M(G\cap M)$
is dense in $M$ for any open set $G$, where
$G\cap M\ne\emptyset$.
\end{theorem}
\begin{proof}
 Let $O_M(\{x\})$ be dense in $M$. Then there is a sequence
$\{\lambda_n\}_{n=1}^{\infty}\subset O_M(\{x\})$ dense in $M$.
 Let now $\mu $ be a measure concentrated on
 $\{\lambda_n\}_{n=1}^{\infty}$,
and $\mu(\lambda_n)\ne 0$ for any $\lambda_n$. Then $\mu$ is ergodic.
In fact, let $S$ be invariant with respect to $\Gamma$, and $\mu(S)\ne 0$,
$\mu(S^c)\ne 0$. Then $S\cap\{\lambda_n\}_{n=1}^{\infty}\ne\emptyset$.
>From this it follows that $S\cap O_M(\{x\})\ne \emptyset$. If
$x\in S$ then $O_M(\{x\})\subset S$, hence $\mu(S^c)=0$, which
contradicts the assumption.  Assume that $x\not\in S$, hence $x\in
S^c$. Since $S$ is invariant with respect to $\Gamma$, so is the
complement of $S$ and $O_M(\{x\})\cap S^c\ne\emptyset$.  From the
condition $x\in O_M(\{x\})\cap S^{c}$ we obtain $O_M(\{x\})\cap
S^c=O_M(\{x\})$.  Hence $O_M(\{x\})\subset S^c$, and $\mu(S)=0$,
which give a contradiction.

2. Let $G$ be an open set, $G\cap M\ne \emptyset$, $\overline{O_M(G\cap
M)}\ne M$,
and $E_A(\cdot)$ be the spectral measure of the operator $A$ with
spectrum $M$. Then $E_A(G)\ne 0$ and there exists a compact set
$F\subset G\cap M$, $E_A(F)\ne 0$. The set $O_M(F)$ is a Borel
set.  Since $\overline{O_M(F)}\ne M$, $E_A(O_M(F))\ne 1$. Thus,
$E_A(\cdot)$ is not ergodic with respect to $\Gamma|_M$ because
$O_M(F)$ is an invariant set.  \end{proof} \begin{definition} A
set $\tau$ is said to be a measurable section  of ($D$, $\Gamma$)
if $\tau\subset D$ is a Borel set and every trajectory
$O_D(\{x\})$ with respect to $\Gamma|_D$ intersects $\tau$ at
exactly one point.  \end{definition} \begin{proposition} Let ($D$,
$\Gamma$) has a Borel section. Then, for every  indecomposable
representation of (\ref{hom}) with $\sigma(A)\subset D$, there exists a
(unique)
trajectory
$O_D(\{x\})$ of full spectral measure: $E_A(O_D(\{x\}))=I$.
\end{proposition}
\begin{proof}
Let $\tau$ be a measurable section of ($D$, $\Gamma$).
Assume that $\mu $ is not concentrated at any orbit $O_D(\{x\})$. Let us
prove that
there exists a partitions of $\tau$ into two sets
$\tau=\tau_1\cup\tau_2$, $\tau_1\cap\tau_2=\emptyset$, such that
$\mu(O_D(\tau_1))>0$, $\mu(O_D(\tau_2))>0$. Consider the trajectories
$T_i=O_D^i(\tau_i)$, $i=1,2$ for any two sets $\tau_i$ satisfying the
conditions
$\tau=\tau_1\cup\tau_2$, $\tau_1\cap\tau_2=\emptyset$. By the definition
of
a measurable section,
$T_1\cap T_2=\emptyset$ and $T_1\cup T_2=D$. Assuming that for any such
decomposition $D=T_1\cup T_2$ one of the values $\mu(T_1)$ and $\mu(T_2)$
 is equal to zero, we can find a decreasing sequence $\{T^k\}$ such that
 $E_A(T^k)=I$ for any $k$ and $\cap T^k=O_D(\{x\})$ for some $x\in\tau$.
 From the last argument we can conclude that $\mu$ is concentrated on an
 orbit which contradicts the assumption.

 Let $\tau=\tau_1\cup\tau_2$ be the required decomposition, i.e.
 $\mu(O_D(\tau_i))>0$, $i=1,2$.  Since both sets $O_D(\tau_1)$ and
 $O_D(\tau_2)$
 are invariant with respect to $\Gamma|_D$, the existence of the
 decomposition implies a contradiction  with
ergodicity of the measure $\mu$. Thus, $\mu$ is concentrated
on the trajectory of some point $x$. \end{proof}

If there is no measurable section for the graph ($D$, $\Gamma$) of
semilinear
relation (\ref{hom}), then the structure of representations with bounded
operators
($A$, $B$, $B^*$) is
more complicated: there might exist
irreducible representations such that the spectrum of the operator
$A$ is not discrete.

\begin{remark}\rm
The same theorems are valid for the
representation with $B=B^*$, but with $\Gamma_s$ instead of $\Gamma$.
\end{remark}

\subsection{Representations of semilinear ${\cal F}_4$-relations}

The problem of  describing  all irreducible
representations of semilinear relations up to unitary equivalence
might be very difficult.
Complexity of description depends on the structure of the
corresponding graph.
\begin{theorem}
$1$. If all connected
components of the graph $\Gamma$ corresponding to a semilinear relation
are  of the form:
$$\onepoint {},\qquad \edgeright {}{}$$
then any irreducible representation ($A$, $B$, $B^*$) of the
relation is one- or two-dimensional:

(i) $A=\lambda$, $B=0$, $\lambda\in{\Bbb R}$;

(ii) $A=\left(
\begin{array}{cc}
\lambda_1&0\\
0&\lambda_2
\end{array}\right)$,
$B=\left(
\begin{array}{cc}
0&0\\
b&0
\end{array}\right)$, where the triple ($\lambda_1$, $\lambda_2$,
$b$) belongs to the set $K_1=\{(\lambda_1, \lambda_2, b)\in {\Bbb R}^3\mid
%\sum_{i=1}^nf_i(\lambda_1)g_i(\lambda_2)=0,
\Phi(\lambda_1,\lambda_2)=0,\lambda_1\ne\lambda_2,
b>0\}$.

$2$. If all connected
components of the graph $\Gamma_s$
are of the form
$$\onepoint {},\qquad\ear{},\qquad \edge {}{} $$
then any irreducible representation ($A$, $B=B^*$) of the relation
is one- or two-dimensional:

(i) $A=\lambda$, $B=0$, $\lambda\in{\Bbb R}$;

(ii) $A=\lambda$, $B=b$, where ($\lambda$, $b$) belongs
to the set $K_2=\{(\lambda, b)\in {\Bbb R}^2\mid \Phi(\lambda,\lambda)=0\}$;
%\sum_{i=1}^nf_i(\lambda)g_i(\lambda)=0\}$;

(iii) $A=\left(
\begin{array}{cc}
\lambda_1&0\\
0&\lambda_2
\end{array}\right)$,
$B=\left(
\begin{array}{cc}
0&b\\
b&0
\end{array}\right)$, where the triple ($\lambda_1$, $\lambda_2$,
$b$) belongs to the set $K_3=\{(\lambda_1, \lambda_2, b)\in {\Bbb R}^3\mid
%\sum_{i=1}^nf_i(\lambda_1)g_i(\lambda_2)=0,
\Phi(\lambda_1,\lambda_2)=\Phi(\lambda_2,\lambda_1)=0,\lambda_1\ne\lambda_2,
b>0\}$.

\end{theorem}
Note that if $f_i$, $g_i$ are polynomials, then the algebra
${\Bbb C}<a,b\mid \sum_{i=1}^nf_i(a)bg_i(a)=0>$ is an ${\cal
F}_4$-algebra.
\begin{proof} Under the above condition we have
that $\Gamma$ is the graph of a bijective mapping
$\phi:N_1\rightarrow N_2$, where
$N_1\cap N_2=\emptyset$
(i.e. $\Gamma=\{(t,s)\in N_2\times N_1\mid t=\phi(s)\}$), and
$\Gamma_s$ is the
graph of a bijective mapping $\phi:N\rightarrow N$
(i.e. $\Gamma=\{(t,s)\in N^2\mid t=\phi(s)\}$).
Moreover, $\phi$ is measurable under the assumption that
$\Gamma(M)$ and $\Gamma^{-1}(M)$ are Borel sets for any Borel $M$.
Therefore, by
Corollary~\ref{cth2}, if $(A,B)\in L(H)$ is a representation of a
relation with  such graph, then ($A$, $B$) satisfy the relation
$AB=B\phi(A)$, and the spectrum of the operator $A$ belongs to
$N_1\cup N_2$ in the first case, and to $N$ in the second one.
Clearly, the operators $A$, $BB^*$ commute.

Let us consider the first case. We have
$B^2=0$ and, therefore, $\ker B\ne 0$. Denote by $H_1$ the
subspace $ \ker B\cap\ker B^*$. It is easy to show that $H_1$ is
invariant with respect to $A$, $B$, $B^*$ and  all
irreducible representations defined on $H_1$ are one-dimensional
and given by $(i)$.  Assume now that $\ker B\cap\ker
B^*=\emptyset$.  Set $H_0=\ker B\cap(\ker B^*)^{\perp}$.  Since
$H_0\subset E_A(\phi(N_1))H$, $B^*H_0\subset E_A(N_1)H$ and
$N_1\cap\phi(N_1)=\emptyset$, the subspaces $H_0$, $B^*H_0$ are
orthogonal. Moreover, $H_0$,
$B^*H_0$ are invariant with respect to $A$, $BB^*$, which imply that given
$\Delta\in {\frak B}({\Bbb R}^2)$, $E(\Delta)H_0\oplus
B^*E(\Delta)H_0$ is invariant with respect to $A$, $B$, $B^*$,
where $E(\cdot)$  is the joint resolution of the identity for
the commuting pair of operators $A$, $BB^*$ restricted to $H_0$.
>From this it follows that $\Delta$ is concentrated in one point if
$A$, $B$, $B^*$ is an irreducible family. Thus for such
family of operators there exists a joint for $A$, $BB^*$
eigenvector $e\in H_0$, and $\{e,B^*e\}$ define an orthogonal basis
of the representation space. The corresponding irreducible
representation is given by $(ii)$.

In the case $A=A^*$, $B$ we have that the operators $A\phi(A)$, $A+\phi(A)$
commute with $A$, $B$, and hence due to irreducibility, they are multiples of the identity, i.e. $A\phi(A)=a_1I$ and $A+\phi(A)=a_2I$. Then $A^2-a_2A+a_1I=0$,
and so the spectrum of $A$ is $\sigma(A)=\{\lambda_1,\lambda_2\}$, where
$\lambda_1$, $\lambda_2$ are roots of the equation
$\lambda^2-a_2\lambda+a_1=0$. Hence the spectrum of $A$ is discrete as soon as
($A$, $B$) is irreducible. In addition, $B^2$ commute with $A$, $B$ and
is a multiple of the identity
 $B^2=b^2I$. If $b\ne 0$ and $e_{\lambda_1}$ is an eigenvector of $A$, then
$e_{\lambda_1}$, $Be_{\lambda_1}$ define an invariant subspace, moreover
$Be_{\lambda_1}$ is an eigenvector with the eigenvalue
$\phi(\lambda_1)=:\lambda_2$.
Therefore, by irreducibility, the operators $A$, $B$ can be at most
two-dimensional. If $\phi(\lambda_1)\ne\lambda_1$ (i.e.
$\Phi(\lambda_1,\lambda_1)\ne 0$)
then normalazing the orthogonal basis $e_{\lambda_1}$, $Be_{\lambda_1}$ we get an orthogonal basis in which operators $A$, $B$ are of the form $(iii)$.
For $\phi(\lambda)=\lambda$ one has that $A$, $B$ commute and hence we can
choose a joint eigenvector $e_{\lambda,b}$, which
define an inavariant subspace. From this one can  conclude that the
corresponding irreducible representation is one dimensional and given by $(i)$
or $(ii)$.
 \end{proof}

If the  graph $\Gamma$ corresponding to a semilinear relation
contains the subgraphs:
a) $\ear {\llll}$ or b) $\twovec $ (and with any other orientation)
and the graph $\Gamma_s$ contains the subgraphs:
c) $\earedge{}{} $, or d) $\twoedge{}{}{} $
then the problem of describing all irreducible representations ($A$, $B$,
$B^*$) and ($A$, $B=B^*$) respectively is getting very complicated (the
corresponding $*$-algebra is wild). We refer the reader to 2.9 and
2.10 for the precise
definition of $*$-wild algebras and proof of the above fact to 2.9 and
2.10.