Greenmail is the buyback by a company of its own stock at an inflated price from an investor who is threatening to launch a takeover bid for the company. As part of the repurchase agreement, the investor promises not to buy any more company shares for a certain period of time. The outcome is that company management remains in charge, so the transaction is an anti-takeover mechanism.
The greenmail name comes from the amount of money (greenbacks) required to make the hostile party go away. An excise tax on greenmail profits has resulted in a sharp decline in the number of greenmail payments having been made in recent years.