Here G and H are Lie groups, and are lattices in one of those Lie groups, and C is a subset of the Lie group.
Detecting (non)compactness via accumulation at the identity
Conjugates by compact sets do not accumulate: The image of in is precompact iff the identity element is not an accumulation point of the conjugated lattice .
[[ some pictures to be inserted here ]]
Proof: iff we cannot find distinct s.t. for all , iff the translated orbit maps (covering maps!) given by are injective on U for every .
Now if our image is precompact, then we may cover it by finitely many pancakes (i.e. neighborhoods in diffeomorphic to each connected component of their preimage in the cover H); by this finiteness, and H-equivariance, we can find some small open neighborhood U so that the desired injectivity condition on our translated orbit maps holds.
If our image is not precompact, then given any small open neighborhood U we can find a sequence of elements in C escaping any compact translate . By an inductive / diagonal argument (involving larger and larger compact sets, each of which contains a previous ) we have a sequence with pairwise disjoint. Since has finite covolume, these cannot all have the same (nonzero) volume, and so (by fundamental domain considerations) not all of the translated orbit maps are injective on U.
Mahler’s compactness criterion is the special case where and .
This can be proven without using that is a lattice, by considering as the moduli space of unit-covolume lattices in (or: “wow, the lattices are back at a whole new level!”): then the result says that a set of unit-covolume lattices is compact iff it does not contain arbitrarily short vectors.
The forward implication is fairly immediate after a moment’s thought; the reverse implication follows by considering the shortest vectors which appear in the lattices of any given sequence from our set, observing that those vectors form bounded orbits, taking (sub)sequential limits in those orbits using compactness in , and finally passing back to (sub)sequential limits in the sequence of lattices.
Detecting (non)compactness with unipotents
Cocompact lattices have no unipotents: The homogeneous space is compact only if has no nontrivial unipotents.
One proof [of the contrapositive] follows from (the general version of) Mahler’s criterion above, via the Jacobson-Morosov Lemma which gives a map sending our favorite unipotent to a nontrivial unipotent . Call a the image under of our favorite hyperbolic element . Then accumulates to the identity, and so is not precompact by the first criterion above.
More is true if we have more structure on our lattices:
Godement: Suppose G is defined over a real algebraic number field F (e.g. ), and let be the ring of integers in F. The homogeneous space (in the case of , ) is compact iff has no nontrivial unipotents.
[[ some more pictures here ]]
Proof of the new (reverse) direction uses the arithmetic structure on . Again we prove the contrapositive. Suppose is not compact. By (Mumford’s generalisation of) the Mahler criterion above, this implies accumulates at the identity for some . By continuity, the characteristic polynomials of the elements in an accumulating sequence converge (in the sense of coefficients; this makes sense if we are working within some fixed ambient , for example) to .
But this implies, by matrix similarity, that the characteristic polynomials of the converge to the same; and now we use the fact that is an integer matrix to say that this implies the , for n sufficiently large, has characteristic polynomial , and hence is unipotent.