Sam Zell‘s top-of-the-market buyout of Tribune Co. cashed out shareholders about a year before the media company tumbled into bankruptcy protection. Now, those former holders are bracing for a possible barrage of litigation aimed at clawing back more than $8 billion in payouts.

If successful, any litigation would represent an unprecedented legal development, some lawyers said. It would force shareholders to give back money gained from simply selling their shares into a corporate buyout offer, based on the theory that the deal was so fundamentally flawed that it amounted to a fraud that should never have happened. Under the best-case scenario, creditors likely would get back only between $2 billion and $3 billion, the amount necessary to make unsecured creditors whole.

While most lawyers said such cases are difficult to prove, former shareholders are girding for the worst, in some instances talking with lawyers and investors about plans to sock away money in anticipation of any litigation.

Creditors are going after the shareholders under a legal concept known as “fraudulent transfer.” The theory allows creditors to argue that the banks financing Tribune’s buyout and the shareholders who cashed out should have known the deal would destroy the company. As a result, the argument goes, the banks shouldn’t be allowed to recoup their loans and the shareholders should have to give back money they received.

In certain instances, merely demonstrating a company was insolvent at the time of a leveraged buyout can leave deal participants exposed.

Hedge fund Stark Investments, one of Tribune’s biggest investors prior to the 2007 buyout, already has begun setting aside money in anticipation of a settlement or judgment, said people familiar with the matter.

Others on Wall Street are hunting for new ways to profit from it.

Prime-brokerage executives at Deutsche Bank AG are in discussions about setting up an operation through the distressed-debt trading desk that would match buyers and sellers of bankruptcy claims related to the shareholder litigation, said a person familiar with the matter. While banks have set up such market-making operations for creditors’ claims, the possibility that shareholders could have to cough up payments expands the market for trading claims.

Deutsche Bank’s prime-brokerage business caters to dozens of hedge funds that could want to bet on the probability that claims against shareholders will succeed, an opportunity one bank executive called “creative thinking” by the bank.

In the lawsuit filed in Delaware bankruptcy court last year, Tribune creditors allege that Mr. Zell’s buyout was “among the worst in American corporate history.” The creditors’ complaint said an unnamed engineer of the deal likened it to “carrying a fat person up [Mount] Everest, hopefully it doesn’t kill us.” The lawsuit was filed by Tribune’s official committee of unsecured creditors, which serves as a watchdog in the bankruptcy case and represents bondholders and a variety of other creditors.

Tribune declined to comment. Jon Wasserman, general counsel for Mr. Zell’s Equity Group Investments, said the bankruptcy examiner found no evidence that Mr. Zell acted in bad faith.

Hedge funds, mutual funds, former Tribune executives and even family trusts and a large foundation are ensnared in the suit. The deal swelled the company’s debt to roughly $13 billion before Tribune was forced to seek bankruptcy protection.

Aurelius Capital Management LP, a large Tribune bondholder, recently said in bankruptcy court that it also could go after shareholders in state courts to disgorge payouts from the deal.

“Former Tribune shareholders likely face a daunting task in defending their receipt of proceeds from the mountain of debt used to finance this deal,” said Bill Welnhofer, managing director and head of restructuring at investment bank Robert W. Baird & Co. in Chicago. “The company’s business model had already started showing signs of weakening.”

The owner of the Chicago Tribune, Los Angeles Times, Baltimore Sun and several television stations used more than $11 billion in debt financing from several Wall Street banks to back the deal and pay off existing shareholders, money some creditors believe belongs to them.

A bankruptcy-court examiner last summer found it “highly likely” that Tribune was “rendered insolvent and without adequate capital” as a result of the deal. The examiner, Kenneth Klee, said some financial projections made by Tribune management as the deal neared closing were too rosy and bore the “earmarks of a conscious effort” to make the company’s financial condition appear better so the buyout could be completed. The examiner found Mr. Zell didn’t act in bad faith.

Tribune’s reorganization plan would shield the lending banks, including J.P. Morgan Chase & Co., Citigroup Inc., Bank of America Corp. and the former Merrill Lynch & Co., which is now owned by BofA, from any future litigation. In exchange, the banks have agreed to pay money to bondholders that those creditors wouldn’t normally receive in a bankruptcy. Shareholders and some former Tribune executives, though, remain exposed. The banks declined to comment.

Courts have generally held that shareholders aren’t liable in leveraged buyouts that later collapse unless the case involves actual fraud with evidence that shows clear intent, said Richard Levin, a bankruptcy lawyer at Cravath, Swaine & Moore LLP.

“It’s tough to prove actual intent because there usually is not a smoking gun,” he said. “Usually, these deals are driven by an intent to make a good business deal, not harm existing creditors.”

For Stark, the potential litigation exacerbates an already precarious position. In 2007, Stark managed $13 billion for investors. But it suffered investment losses in 2008 and some of the worst client withdrawals in the industry, leaving it with $3 billion now, with investors still waiting for several hundred million dollars more in redemptions that haven’t yet been paid, said people close to the matter.

Stark has told clients that the Tribune bankruptcy “could get messier,” and the hedge-fund firm is being conservative by holding back money, said one person familiar with the matter.

Other large former shareholders targeted by Tribune creditors include the Chandler family, among Tribune’s largest owners at the time of the buyout, and the McCormick and Cantigy foundations. The trio held at least a third of Tribune’s shares and reaped more than $2 billion in the buyout, according to creditors.

Tribune’s former chief executive, Dennis FitzSimons, also is targeted.

Representatives for the Chandlers didn’t respond to requests for comment. The foundations declined to comment. Mr. FitzSimons, also the McCormick foundation’s board chairman, declined to comment.

Write to Mike Spector at mike.spector@wsj.com, Jenny Strasburg atjenny.strasburg@wsj.com and Shira Ovide at shira.ovide@wsj.com