In February the two exchanges announced they would tie the knot with the combined group having joint headquarters in London and Toronto, with 20 trading platforms/markets across Europe and North America.
In a statement, TMX Group said a majority of shareholder votes cast by proxy prior to the proxy cutoff supported the merger resolution, however the two-third threshold required to approve the merger would not be achieved.
The scrapping of the merger opens the door to the rival bid made by a consortium of financial firms called the Maple Group Acquisition Corporation.
TMX Group said it would review other opportunities including the offer from Maple.
"The TMX Group management and board believe that the TMX-LSEG merger would accelerate our business strategy and create shareholder value, while enhancing the performance of Canada's capital markets," TMX Group chief executive officer Tom Kloet said.
"Although we will not join forces with LSEG, our business is strong and I have enormous confidence in the continued success of our company."
TMX has agreed to pay a $10 million expense fee to the LSE over the failed bid and a further $29 million fee if it accepts Maple's acquisition offer in the next 12 months.
The merged group would have included the smaller Alternative Investment Market in London and TSX Venture Exchange in Toronto, which cater to small-cap and early stage companies, particularly in the mining sector.
It would also have included Borse Italiana, owned by the LSE.
The combined Toronto and London stock exchanges would have been the number one listings venue in the world with 6700 total listings, as well as the top market in the world for mining companies.