Transitive operators and a problem of halmos

C. F

OIAS

, I. B. J

UNG

, E. K

O

,

&

C. P

EARCY ABSTRACT. In this note we show that a (still open) problem raised by Halmos in [1] is related to the general invariant subspace prob-lem for operators on Hilbert space (and may be equivalent to it).

1. INTRODUCTION

LetH denote a separable, infinite dimensional, complex Hilbert space, and write

L(H )for the algebra of bounded linear operators onH. In the now classical and very influential paper [1], P.R. Halmos set forth ten open problems concerning operators inL(H ). Of these ten, today only one problem remains open, namely: (P) If T is an invertible operator in L(H ) and T has a nontrivial invariant

subspace (n.i.s.), mustT−1_{have a n.i.s. also?}

A similar problem mentioned in the discussion of Problem (P) in [1] that Halmos was also fond of advertising is as follows:

(R) IfT ∈ L(H )andT2has a n.i.s., mustT have a n.i.s. too?

The authors of this article have spent much time trying to resolve both Prob-lems (P) and (R) affirmatively with only modest success (cf. [3], [4], and [2]).

The purpose of this note is to make some observations which suggest the possibility that Problem (R) may be equivalent to the general invariant subspace problem for operators inL(H ). (It should be noted at once that if one changes the context to infinite dimensional complex Banach spacesX, then examples due to Read (cf. [7], [8]) show that at least for someX, Problem (R), as well as problem (H) below, has a negative answer.) To proceed, let us recall that an operatorT in

L(H )is said to be transitive if its only invariant subspaces are (0)andH, and consider the following problem (which is the general invariant subspace problem for operators inL (H )):

(H) Is every operator inL (H )nontransitive?

Of course if the answer to Problem (H) is affirmative, then Problem (R) has an affirmative answer too. Therefore Problem (R) is relevant only if the set of

119

transitive operators is not empty. In this note, therefore, we will operate under the assumption that the answer to Problem (H) is “no”.

Our principal result is the following.

Theorem 1.1. If the answer to Problem (H) is negative, then either the answer to

problem (R) is also negative or every transitive operatorT inL(H )has the following property:

(†) there exist quasiaffinitiesV1, V2 ∈ L(H ) (i.e., kerVi = (0) = kerVi∗, i = 1,2)and operatorsG1, G2∈ L(H )such that

(1.1) V1G21= T V1, G22V2= V2T , and V2V1

G1= G2 V2V1

.

The proof of Theorem1.1will gradually emerge from the discussion below, which concludes with the proof of Theorem5.1. We begin, however, by making some remarks about a property (††) closely related to Property (†), but defined for a classJ of operators inL(H )as follows:

(††) for everyT inJ there existGi,i =1,2,and quasiaffinities Vi, i = 1,2,in

L (H )such that (1.1) holds.

It is worthwhile to note that the class of normal operators trivially has Property (††) since every normal operator has a normal square root, but, as we show below (Example6.3), the classL(H )does not have Property (††).

Thus letT be a transitive operator inL(H ), and without loss of generality suppose thatkT k = β ≤1. Consider the operatorA = AT ∈ L(H ⊕ H )defined as

(1.2) A = 0 1H

T 0

! .

ClearlyA2 = T ⊕ T, and a short calculation shows that A2 is not transitive. In particular, the subspaceM = {x ⊕ x:x ∈ H }is a n.i.s. forA2. Thus ifAabove were also transitive, then a negative answer to Problem (H) would imply that also Problem (R) has a negative answer. Therefore, we will try to determine, in this context, whetherAis transitive or, at least, may be transitive.

2. PRELIMINARIES

As usual, Lat(A)will denote the lattice of all invariant subspaces of an operator

A. Also, the domain of a linear transformationJ(not necessarily defined on all of

H) will be denoted byDJ,while its range will be written asRJ. Our discussion begins with the following lemma.

Lemma 2.1. IfT is a transitive operator inL(H ),A = AT is as in (1.2), and

M ∈Lat(A) \(0), H ⊕ H , thenMmust have the form

(2.1) M =x ⊕ Jx:x ∈ DJ

,

(a) the domainDJofJis a dense linear manifold inH; (b) the rangeRJofJis a dense linear manifold inH;

(c) Jis a closed, perhaps unbounded, injective, linear transformation;

(d) JDJ= RJ⊂ DJandT DJ⊂ DJ; and (e) forx ∈ DJ,T x = J2xandT Jx = JT x.

Proof. SinceA2 = T ⊕ T andT is transitive, it is clear that if there exists a vector of the formx ⊕0 [0⊕ x] inM, thenM ⊃ H ⊕ 0
[M ⊃ 0
⊕ H], but neitherH ⊕ 0
nor 0
⊕ H belongs to Lat(A), and it follows easily thatM

must have the form (2.1). SinceD−

J, R−J ∈ Lat(T ), we obtain (a), (b), and (c).

Finally, a short calculation gives (d) and (e). ❐

We remark immediately that Lemma 2.1carries no implication concerning how many suchJexist corresponding to a fixedAas in (1.2). Let us now expound a simple case in which the operatorAin (1.2) cannot be transitive, even though

T is transitive.

Proposition 2.2. With the notation as in Lemma2.1, and under the additional assumption that there exists an operatorT1∈ L H

such thatT12= TandT1belongs

to the norm-closed algebra Rat T
generated by the rational functions ofT (with poles off the spectrumσ T
ofT ), we have that for everyJas in Lemma2.1,J = T1 or

J = −T1, and thus

M±=x ⊕ ±T1x

:x ∈ H

are exactly the nontrivial invariant subspaces ofA (soAis not transitive).

Proof. Note first thatT1 must be transitive with T, and letJ be any linear transformation as in Lemma2.1. SinceT = T1

2

withT1∈Rat T

,T1J ⊂ JT1, i.e.,T1DJ⊂ DJandT1Jh = JT1hfor allh ∈ DJ. Hence

T1
2 h = T h = J2h, h ∈ DJ, and (2.2) 0= J2_{h − T} 1
2 h = J + T1
J − T1
h, h ∈ DJ.

But sinceT1is bounded, it is easy to see that bothJ + T1 andJ − T1are closed linear transformations. Hence ker J + T1

and ker J − T1

are (closed) subspaces ofH and from (2.2) we get that at least one of these kernels is not 0
. But such a kernel is invariant underT, and sinceT is transitive, it must be the whole space

H, so eitherJ = T1orJ = −T1. ❐

For a discussion of the case in which T1 is merely a square root of T, see Section3.

Proposition 2.3. If in Proposition2.2we assume thatT is also invertible, then

AT is similar toT1⊕ −T1

Proof. We first show that

(2.3) H ⊕ H = graphT1+˙ graph −T1

,

meaning thatH ⊕H is the algebraic sum of the “disjoint” subspaces graph ±T1

which do not make angle 0 with one another. To verify (2.3), lethn⊕ T1hn

andkn⊕ −T1

kn

be sequences of unit vectors in graphT1and graph −T1

, respectively, and suppose that

hn⊕ T1hn

− kn⊕ −T1kn

→ 0.

Then hn− kn → 0 and T1 hn + kn
→ 0. The first of these relations shows that T1 hn− kn
→ 0, and “adding” this last relation to the second clearly gives T1hn →0. Hence hn →0 sinceT1is invertible withT, which contradicts the fact that thehn⊕ T1hn are unit vectors, and shows that graph

T1∩graph −T1

= 0
and that the angle between these spaces is not 0. Suppose now thatu ⊕ v is orthogonal to the (closed) subspace graph T1+graph −T1

. Then u, h+v, T1h  =0, u, h−v, T1h  =0, h ∈ H ,

so clearlyu =0= v,and this establishes (2.3). Moreover,

A h ⊕ ±T1h

= ±T1h
⊕ T h, h ∈ H . DefineY± h ⊕ ±T1h

= h for allhinH. ThenY± : graph(±T1) → H are invertible and Y±A h ⊕ ±T1h

= ±T1h = ±T1Y± h ⊕ ±T1h

, that is, Y±A|_{graph ±T} 1
= ±T1Y±, whenceA|_{graph ±T} 1

_{is similar to±T1andAis similar toT1⊕ −T1
.} ❐

Looking at (e) in Lemma2.1leads one to ask whether the assumption thatJ

is unbounded and the closure ofJ2_{is a bounded operatorZinL H
forcesJto} be bounded. We show in Section 6 with some examples that this need not be the case, even ifZ =1H.

Remark 2.4. A bit of thought shows that the facts presented in this section

together with their proofs are valid even whenH is only a complex Banach space. However, in case H is, indeed, a Hilbert space, the operators J appearing in Lemma2.1enjoy an interesting supplementary property, as follows.

Proposition 2.5. LetATbe associated with a(not necessarily transitive)operator

T ∈ L(H )as in (1.2), and letMbe an invariant subspace ofAT of the form (2.1)

whereJ also satisfies (a) and (c) of Lemma2.1. ThenJ also satisfies (d) and (e) of Lemma2.1, as well as the following:

(f ) Jand(its adjoint) J∗are the closures of their restrictions toDJ∩ DJ∗.

Proof. The only fact that needs proof is (f ). To this end, we recall a basic

result of von Neumann [6, Theorem 13.28]; namely, that any closed linear trans-formationX with domainDX dense inH is the closure of its restriction to the domainDX∗_XofX∗X. We apply this result withX = Jas in the statement. Since

JDJ⊂ DJ(via (d)), we obviously have thatJDJ∗_J⊂ D_J∩ D_J∗. Thus

L = {x ⊕ Jx:x ∈ DJ∩ DJ∗} ⊃ L₀= {x ⊕ Jx:x ∈ JD_J∗_J}

= {Jy ⊕ J2_{y:y ∈ D}

J∗_J} = {Jy ⊕ T y :y ∈ D_J∗_J}.

If for somex0∈ DJ, we have(x0⊕ Jx0) ⊥ L, then(x0⊕ Jx0) ⊥ L0, and hence

x0, Jy  = −T∗_Jx 0, y 

, y ∈ DJ∗J. Due to the result of von Neumann mentioned above, this equality is valid fory ∈ DJ. In particular, this implies that

x0∈ DJ∗. Thusx0⊕Jx0∈ Land thereforex0⊕Jx0=0. This shows thatL−= graphJ, and thus thatJis the closure ofJ|DJ∩DJ∗. With respect toJ

∗_{, observe} that

M⊥= {(−J∗y) ⊕ y :y ∈ DJ∗},

is invariant under(AT)∗, and the proof is concluded as above with minor

modi-fications. ❐

Note that sinceM =graphJ andAM =graph J2, the proof above implies that

L−

0 = (AM)−, M L−0 ⊂ L, and thus thatL ⊃ L0+ (M L−0).

3. MORE ON SQUARE ROOTS

In this section we consider the case in which a transitive operatorT ∈ L(H )has a square rootR inL H
that is not necessarily in Rat(T ). In this caseA is not transitive, since, in particular,

(3.1) M±=x ⊕ ±Rx:x ∈ H

are n.i.s. forA. ButT andRhave some supplementary properties that are worth mentioning.

Lemma 3.1. WithT ∈ L H
transitive, letRdenote the collection of all square roots ofT that belong toL H
. Then either

(b) Rhas exactly two elements,±Rfor someR ∈ L H
, or

(c) the cardinal number ofR is at least 4, and for every pair R1, R2 ∈ R, R1 is

quasisimilar toR2(i.e., there exist quasiaffinitiesXandY such thatR1X = XR2

andY R1= R2Y ).

Proof. If neither (a) nor (b) holds, then there existR1, R2 ∈ R such that

R1+ R2 6= 0 6= R1− R2. Moreover, R1 and R2 are transitive with T, so, in particular,

Lat(R1) ∩ Lat(R2) = {(0), H }. This, together with the following interwining relations:

R1(R1+ R2) = (R1+ R2)R2,

(R1+ R2)R1= R2(R1+ R2),

R1(R1− R2) = (R1− R2)(−R2),

(R1− R2)R1= (−R2)(R1− R2),

shows that bothR1+ R2andR1− R2 are quasiaffinities (cf. [5] for details), and thus that all of the operators inRare quasisimilar one to another. ❐

Returning to the operatorA = AT associated with a transitiveT (with square rootR) in L(H )as in (1.2), we observe that, with R and M± as in (3.1), the restrictions A|M± are transitive too. Indeed, supposeM ⊂ M+ andAM ⊂ M.

Then, by Lemma 2.1, there exists J = J(A, M) with the properties enunciated there such that M =graph J. ButJ = R|DJ is bounded, and sinceJ is closed

andDJ is dense in H, we getJ = R andM = M+. However, in the present case, unlike that of Proposition2.2,M±may not be the only nontrivial spaces in Lat(A). The reason is that here, ifN ∈Lat(A)andN =graphG, withGas in Lemma2.1, we do not know thatGcommutes with R. Therefore we can only conclude that

(G + R)Gx = R(G + R)x,

(G − R)Gx = (−R)(G − R)x, x ∈ Dj.

Since ker(G ± R)and{Ran(G ± R)}−all belong to Lat(T ), we must have ker(G ± R) =0 , Ran(G ± R) −= H ,

so the (obviously closed) linear transformationsG ± R are “perhaps unbounded quasiaffinities”.

4. ELIMINATION OFUNBOUNDEDOPERATORS

Returning to the context of Section 1, with T ∈ L H
transitive, A = AT ∈

L H ⊕ H
as in (1.2), andJas in Lemma2.1, for subsequent use we define (4.1) V = 1+ J∗_J
−1/2

, W = J 1+ J∗_J
−1/2

Lemma 4.1. [6, Theorem 13.27] WithV andW as in (4.1), V andW are contractions inL(H ),Vis Hermitian,RV = DJ, and

(4.2) V k 2+ W k 2= k 2, k ∈ H .

AlsoVis invertible(inL H
)if and only ifJis bounded, i.e.,J ∈ L H
.

We next define the operatorΩ:H → Mby

(4.3) Ωx = V x ⊕ W x = V x ⊕ JV x, x ∈ H ,

whereV andW are given by (4.1). Then, by (4.2),

Ωx 2

= V x 2+ W x 2= x 2, x ∈ H ,

and thereforeΩis an isometry. HoweverRV = DJ. ThusΩis a unitary operator fromH toM. We now defineG = Ω∗ A|M
Ω = Ω∗AΩ. Clearly G ≤ A ≤ 1 and W = JV = V G. (4.4) Thus, by (4.2), _k 2 = V k 2+ W k 2= V k 2+ V Gk 2, k ∈ H , (4.5) and also T V = J2V = V G2, (4.6) so _G2 h 2= V G2h 2+ V G3h 2= T V h 2+ T V Gh 2 (4.7) ≤ T 2 V h 2+ V Gh 2
, h ∈ H .

Hence G2 ≤ T = β. From (4.5) we also infer that

(4.8a) 1= V∗V + G∗V∗V G.

It follows from (4.8a) and (4.6) that

and subtracting this last equality from (4.8a), we obtain 1− G∗_{G = V}∗ ₁₋ T∗T
V, whence (4.8b) 1− G∗G ≥ 1− β
V∗V . Thus 1− β
= 1− β
V∗V + 1− β
G∗V∗V G ≤1− G∗_{G + 1− β
G}∗_V∗_{V G} =1− βG∗G − 1− β
G∗(1− V∗V )G,

whence, by rearranging, we obtain

(4.8c) (1− β)G∗(1− V∗V )G ≤ β(1− G∗G).

In order to introduce equivalent forms of the just established relations which in-volve the norm ofH, we introduce the defect operator of a contractionC, namely

DC = 1− C∗C

1/2

. With this notation the relations (4.8a)–(4.8c) can be rewrit-ten as _{DV Gh} 2 = DGh 2+ DVGh 2= V h 2, h ∈ H , (4.9a) _DGh 2 ≥ 1− β
V h 2, h ∈ H , (4.9b) 1− β
DVGh 2≤ β DVh 2, h ∈ H , (4.9c)

respectively. Observe that (4.9a), (4.9b) yield

_DVGh 2

≤ β V h 2, h ∈ H .

These relations readily yield the following.

Proposition 4.2. Ifβ = T
<1, then for every bounded sequencexn

∞ n=1

inH, the following assertions are equivalent:

(a) V xn →0, (b) DGxn →0,

(c) DVGxn →0.

Note that if T is as above and G = 1 then there exists xn

∞

n=1 in H,

_{xn = 1, such that Gxn →1. In this case DGxn → 0,and consequently V xn →0. Hence 0∈ σ V}

. Conversely, for the sameT ,if 0 ∈ σ V
,then there exists a sequence xn

∞

n=1 in H of unit vectors such that V xn → 0, whence DGxn →0, too. This means that 1− Gxn 2 = DGxn 2 →0, i.e.,

_{G =1.}

Proposition 4.3. LetT ∈ L(H )be transitive with T = β ≤1, and suppose

thatA = AT is as in (1.2). Then A2 is not transitive, and, moreover, if A is not

transitive, then:

(a) there existsV ∈ L(H )such that 0≤ V ≤1H and kerV =ker 1H− V
= 0
,

and

(b) there existsG ∈ L(H )such that G2 ≤ T _{,T V = V G}2_{, and (4.8a) holds.}

In caseβ <1, we also have:

(c) 0∈ σ V
if and only if G =1.

In the converse direction we have the following.

Proposition 4.4. With the notation as in Proposition4.3, suppose that

(a)0 there existsV0inL(H )such that kerV0=0 =ker V0
∗,

(b)0 there existsG0inL H
such thatT V0= V0 G0
2, and

(c)0 for every bounded sequence xn

n∈N in H such that V0xn → 0 we have

_V0_G0_x n →0.

ThenAis not transitive. Proof. Consider the space

M0_{= {V}0_{x ⊕ V}0_G0_{x:x ∈ H }}−_.

Then properties (a)0and (b)0imply thatM0_{6= (0)andAM}0_{⊂ M}0_{. Finally,} prop-erty (c)0 obviously implies thatM0 _{is a graph (of a closed linear transformation)} and thusM0_{6= H ⊕ H. Consequently,Ais not transitive. ❐}

5. DUALCONSIDERATIONS

In this section we apply the discussion above to the adjointA∗ of the operatorA

in (1.2), and thereby obtain a sort of dual to Proposition4.3. The notation and usage will be the same as in Sections1and2. In particular, ifM ∈Lat A
, then by Lemma2.1, we obtainM = x ⊕ Jx :x ∈ DJ

, soM⊥ ₌  _−J∗_{y
⊕ y :}

y ∈ DJ∗ , andA∗M⊥ ⊂ M⊥. Thus, for anyy ∈ D_J∗, there existsy ∈ De _J∗

such thatT∗y ⊕ −J∗y
= −J∗ye
⊕ ey, and

T∗y = −J∗y,e y = −Je ∗y, T∗y = J∗2y.

Consequently, paralleling the construction in Section4, we define

Ω∗= −W∗z
⊕ V∗z z ∈ H , where

V∗= 1+ JJ∗
−1/2, W∗= J∗V∗⊇ VJ∗, (5.1a)

and we obtain

W∗∗= JV = V G. (5.1b)

As before,Ω∗:H , M⊥ is a unitary operator, and withG∗ := − Ω∗∗A∗Ω∗
∗, we have thatΩ∗G∗∗= A∗Ω∗, and

W∗G∗∗z

⊕ −V∗G∗z
= A∗ −W∗z
⊕ V∗z

= T∗_V

∗z
⊕ −W∗z
, z ∈ H . Hence,G∗W∗∗= V∗T andV∗G∗∗= W∗. ThusG∗V∗= JV = VGand

V∗T = G∗W∗∗= G∗ G∗V∗
, so V∗T = G2∗V∗, (5.2a) G∗V∗= VG, (5.2b) V∗JV = JV2= VGV . (5.2c)

Similarly, we obtain V J∗V∗ = V∗G∗∗V∗, whence V∗G∗V∗ = V∗JV = V GV. Therefore, by using also (5.2b) we get

(5.3) V∗V G = V∗G∗V∗= VGV = G∗V∗V .

We can now state our main result, from which Theorem 1.1 immediately follows.

Theorem 5.1. The operatorA = AT ∈ L H ⊕ H

given by (1.2) associated

with a transitive operator T in L H
is nontransitive if and only if Property (†)

holds.

Proof. The necessity of the conditions (†) was already established above, even with the supplementary properties that 0≤ V1, V2≤1 as well as

_x 2

= V1x 2+ V1G1x 2, x 2= V2x 2+ V2G2∗x 2

, x ∈ H .

Therefore it remains only to prove that Condition (†) is sufficient, which goes as follows. Consider the (obviously nonzero) subspaces

M1=  V1x1⊕ V1G1x1:x1∈ H − , M2=  −V2∗G∗2x2⊕ V2∗x2:x2∈ H − .

Note that by virtue of (1.1), V1x1⊕V1G1x1, −V2∗G∗2x2⊕V2∗x2  = −G2V2V1x1, x2  +V2V1G1x1, x2  =0,

soM2⊂ M⊥1 and thereforeM16= H ⊕ H. Moreover,

A V1x1⊕ V1G1x1
= V1G1x1⊕ T V1x1 = V1G1x1⊕ V1G21x1∈ M1, x1∈ H , soM1∈Lat A
. ❐

Remark 5.2. The operators V, G and V∗, G∗ associated with J and J∗, respectively, as in (4.1), (4.4), and (5.1a)–(5.1b), also satisfy, because of property (f ) in Proposition2.5, the following two conditions:

{V y ⊕ V Gy :V y ∈ RV∗}−= {V y ⊕ VGy :y ∈ H },

{(−V∗G∗∗y) ⊕ V∗y:V∗y ∈ RV}−= {−V∗G∗∗y ⊕ V∗y:V∗y ∈ H }, which shows thatRV∩ RV∗ is dense inH, a curious fact.

6. SOMEEXAMPLES

We give now some examples referred to in Sections1and2.

Example 6.1. Let the interval(0,1)be equipped with Lebesgue measure, and let Mf
t
= tf (t)a.e. in 0,1
, for allf ∈ L2 _{(0,1)
. Define}

J = M M

−1

0 M

! ,

on L2 (0,1)
⊕ RM, where, of course, RM is the range of M and M−1 is un-bounded. Consider

xn⊕ yn∈ DJ, xn⊕ yn→ x ⊕ y, J(xn⊕ yn) → u ⊕ v.

ThenMyn→ My,soMy = vandM−1yn→ u − Mx,soy ∈ DM−1, M−1y =

u − Mx,i.e.,x ⊕ y ∈ DJ, J(x ⊕ y) = u ⊕ v. ThusJis closed. Forx ⊕ y ∈ DJ, we get J(x ⊕ y) = (Mx + M−1y) ⊕ My∈ DJ andJ2(x ⊕ y) = (M2x +2·1Hy) ⊕ M2y. Hence J2= M 2 _2·1H 0 M2 !   DJ , andJ2_{is bounded.}

Example 6.2. Consider the interval(−1,1)equipped with Lebesgue measure, and consider two linear transformationsQ±onL2(−1,1)defined by

Q±f
(t) = 1∓ t 1± t

f (t),

for thosef in L2((−1,1))satisfying Q±f ∈ L2(−1,1), respectively. ThenQ± are closed linear transformations with dense domainsD+ andD−, respectively. Consider also the unitary operatorV ∈ L2(−1,1)defined by

V f (t) = f (−t), f ∈ L2 (−1,1)
,

and writeJ = V Q+, which is, of course, defined onDQ+. Then

f ∈ DQ± ⇐⇒ Z₁ −1  1∓ t 1± t 2 +1 ! f (t)2dt < ∞

and for suchf,

Jf (t) =  1+ t 1− t  f (−t) = Q−V f
(t). Thus, Z1 −1  1− t 1+ t 2 +1 !  _Jf
(t)2dt = Z1 −1 1+  1+ t 1− t 2! _{f (−t)}2 dt = Z₁ −1 1+  1− t 1+ t 2! _{f (t)}2 dt < ∞.

Hencef ∈ DQ+,soJ2f is well defined. Moreover,J2f = V Q+Q−V f = V2f =

f ,and thusJ2|DQ+ =1DQ+ even thoughJ|Q+ is unbounded.

Example 6.3. Herein we show that L H
does not have Property (††) de-fined in Section1. For this purpose we takeH = H2 _{T
, withTthe unit circle} inC(equipped with normalized arc length measure) andH2 _{T
the usual Hardy} subspace ofL2 T
, regarded as a space of analytic functions on the open unit disc

DinC. LetT = Tz be the (analytic) Toeplitz operator inL H2 T

with symbol

z, soTz is a unilateral shift of multiplicity 1. Suppose that there exist quasia ffini-tiesV1, V2and operatorsG1,G2inL(H )satisfying (1.1). Then, as was shown in the proof of Theorem5.1,

M:=V1h ⊕ V1G1h:h ∈ H

is a nontrivial invariant subspace forA = AT and thus forA2. Consequently, by virtue of the Beurling-Halmos-Lax theorem, we know thatMmust have one of the forms M = {θ1h ⊕ θ2h:h ∈ H } withΘ =  θ1 θ2  _{an inner function, or} M =     θ11 θ12 θ21 θ22    h k  _{:h ⊕ k ∈ H ⊕ H}    , withΘ =  θ11 θ12 θ21 θ22  _{an inner function.}

In the first case, sinceθ1⊕ θ2 ∈ M andA θ1⊕ θ2

= θ2⊕ Tzθ1

, there exists 06= h ∈ H2 _{T
such thatθ}

2 z
= θ1 z
h z
andzθ1 z
= θ2 z
h z
, soθ2 z
h z
2= zθ2 z
forz ∈ D. Ifθ26=0,z ≡ h z
2 , which is impossible in H2 _T
.

In the second case, consider the vectorw = −θ12

⊕ θ11∈ H ⊕ H. Clearly,

Θw =0⊕detΘ ∈ M\ 0
. On the other hand, from (1.1) we get G2V2

V1h =

V2 V1G1h

for h ∈ H2_{(T), whence, withu}

1 = V1h andu2 = V1G1h, we get

G2V2u1− V2u2=0 for allu1⊕ u2∈ M. Thus, from above, withu1=0,u2= detΘ, we obtainV2 detΘ

=0, which is impossible sinceV2is a quasiaffinity and detΘ 6=0. HenceL(H )does not have Property (††).

7. CONCLUDINGREMARKS

A careful perusal of the foregoing proofs shows that we have also established the following.

Theorem 7.1. LetA = AT be defined by (1.2) whereT is arbitrary inL(H ).

ThenAhas a nontrivial invariant subspaceMof the form (2.1) whereJ is a closed linear transformation defined on a dense domain DJ in H if and only if T has

Property (†).

Of course, ifT is transitive, then, as has already been observed (Lemma2.1),

Mmust be of the form described in Theorem7.1. Moreover, if in Theorem7.1

the role of invariant subspace is replaced by that of hyperinvariant subspace, the same conclusion holds. Indeed, we have the following.

Theorem 7.2. LetA = AT be defined by (1.2) whereT ∈ L(H )has no

non-trivial hyperinvariant subspace. ThenA has a nontrivial hyperinvariant subspace if and only ifT has Property (†).

Proof. As already noted in Theorem7.1, the basic observation in our consid-erations is the fact that anyM ∈Lat(A)has the formM =graphJ(as in Lemma

2.1). Since, as an easy calculation shows, the commutant{A}0_{has the form}

{A}0=      B C T C B  _{:B, C ∈ {T }}0    ,

it is easy to verify that if we suppose only thatT has no nontrivial hyperinvariant subspace and thatM ∈Hlat(A), all the conclusions of Lemma2.1remain valid for M. In fact,J has the supplementary properties thatBDJ ⊂ DJ andBJ =

JB|DJ for allB ∈ {T }0. ❐

Therefore it is natural to pose the following.

Problem 7.3. In caseT in Theorem 7.1is transitive, find additional relations between theViandGi, i =1,2,arising in Property (†).

Also, in connection with Example6.3, we ask another question.

Problem 7.4. Find a classJ of biquasitriangular quasiaffinities inL(H )that does not have Property (††).

Acknowledgement. The second and third authors were supported by a grant

(R14-2003-006-01000-0) from the Korea Research Foundation. Also the authors express their appreciation to the referee for a thorough and timely referee’s report.

REFERENCES

[1] P.R.HALMOS, Ten problems in Hilbert space, Bull. Amer. Math. Soc. 76 (1970), 887–933. [2] I.JUNG, E.KO, and C.M.PEARCY, Square roots of (BCP)-operators, Arch. Math. 82 (2002),

317–323.

[3] C.FOIAS, C.M.PEARCY, andB ´ELASZ. NAGY, Contractions with spectral radius one and

in-variant subspaces, Acta Sci. Math. 43 (1981), 273-280.

[4] C.FOIASand C.M.PEARCY, (BCP)-operators and enrichment of invariant subspace lattices, J. Operator Theory 9 (1983), 187–202.

[5] V.MATACHE, Operator equations and invariant subspaces, Matematiche (Catania) 49 (1994), 143–147.

[6] J.VONNEUMANN, Functional Operators, vol. II: The geometry of orthogonal spaces, Princeton Univ. Press, Princeton, 1950.

[7] C.READ, A solution to the invariant subspace problem, Bull. London Math. Soc. 16 (1984), 337– 401.

[8] , A solution to the invariant subspace problem on the space`1, Bull. London Math. Soc. 17 (1985), 305–317.

CIPRIANFOIAS:

Department of Mathematics Texas A&M University

College Station, TX 77843, U.S.A. E-MAIL: foias@math.tamu.edu

ILBONGJUNG:

Department of Mathematics College of Natural Sciences Kyungpook National University Daegu 702-701, Korea E-MAIL: ibjung@knu.ac.kr EUNGILKO:

Department of Mathematics Ewha Women’s University Seoul 120-750, Korea E-MAIL: eiko@ewha.ac.kr CARLPEARCY:

Department of Mathematics Texas A&M University

College Station, TX 77843, USA E-MAIL: pearcy@math.tamu.edu

KEY WORDS AND PHRASES: invariant subspace; transitive operator.

2000MATHEMATICSSUBJECTCLASSIFICATION: 47A15.

Received : March 27th, 2006; revised: May 19th, 2006. Article electronically published on January 29th, 2007.