In a statement at Downing Street she said it was unacceptable for either side in the discussions to make threats.
They also put on ice a special summit suggested for mid-November to seal a deal, saying it would only happen if there was real progress at the next European Union gathering in October.
Mrs May said she was ready to come forward with new ideas on unblocking the disagreement over future arrangements at the Irish border.
And she promised the people of Northern Ireland that, if there was no deal, the Government would "do everything in our power to prevent a return to a hard border".
Many have voiced opposition to her plans, which they said would bind Britain into much European Union regulation in return for free trade, and some would prefer a no-deal "hard Brexit" in March, despite warnings that would ravage the British economy.
"It was clear today that we need substantial progress by October and that we then aim to finalise everything in November", Merkel said.
The prime minister's economic plans, which would end free movement and keep frictionless trade in goods but not services, were given equally short shrift.
He said their manner "pushes us more into a position where we say the quicker we're out of this circus, the better".More news: Elon Musk reveals the first passenger SpaceX will send around the moon
Speaking of the Instagram post, he said: "I think that is quite insulting actually".
Mr Macron said Mrs May had tried to tell the leaders "take it or leave it" but added: 'The proposals in their current state are not acceptable.
"Theresa May's Brexit negotiating strategy has been a disaster", opposition Labour leader Jeremy Corbyn said.
"I actually think the Chequers proposal is not ideal but broadly represents the kind of compromising package that protects Britain's industrial base, that protects agriculture and represents a positive position to take into the negotiations", he said.
The situation was not helped by the emergence of a report that Trade Secretary Liam Fox was planning to slash food standards after Brexit to give the United Kingdom a competitive advantage and secure a USA trade deal.
May had hoped that Salzburg would strengthen her hand in the run up to her Conservative Party conference, the annual gathering of the party where prime ministers traditionally make a keynote speech on policy and the direction of the country.
May's speech followed the end of summit talks on Thursday which had initially been billed as a chance for European diplomats and leaders to offer the prime minister words of support and shore up backing for a Brexit deal at home.
"The referendum was the largest democratic exercise this country has ever undergone", said May, who has repeatedly ruled out a second vote following the original 2016 referendum.
"We need serious engagement on resolving the two big problems in the negotiations" - trade and the Irish border, she declared. To deny its legitimacy or frustrate its result threatens public trust in our democracy.