Operator equations and subscalarity

We consider the system of operator equations ABA = A² and BAB = B². Let (A, B) be a solution to this system. We give several connections among the operators A, B, AB, and BA. We first prove that A is subscalar of finite order if and only if B is, which is equivalent to the subscalarity of AB or BA with finite order. As a corollary, if A is subscalar and its spectrum has nonempty interior, then B has a nontrivial invariant subspace. We also provide examples of subscalar operator matrices. Moreover, we deal with algebraicity, power boundedness, and quasitriangularity, using some power properties obtained from the operator equations.