The Prime Minister phoned Mariano Rajoy to complain about the checks and suggestions that Spain could introduce further sanctions.
It is understood the two leaders agreed the need to calm the row, which has resurfaced in recent months due to a dispute over fishing grounds.
Downing Street said Mr Cameron had told Mr Rajoy he did not want the row to “damage our bilateral relationship” but also made clear he was standing firm.
On Twitter, the Prime Minister described the 15-minute call as “constructive” and wrote: “I made clear my concerns re Gibraltar and that our position on sovereignty won’t change.”
Number 10 said there had been a “real risk” the relationship between Britain and Spain would be affected if the situation at the border did not improve.
A spokesman said: “Mr Rajoy agreed that he did not want the issue to become an obstacle in the bilateral relations and that we needed to find a way to de-escalate the issue.
"As a next step, the Foreign Secretary should speak to Mr [Jose] Garcia-Margallo to discuss a way forward.
"In the meantime, Prime Minister Rajoy committed to reducing measures at the border. Both leaders agreed that there should be a solution to the fishing dispute."
However, a Spanish government statement about the phone call did not mention any concession over the border controls and firmly blamed Gibraltar for the row.
It said Mr Rajoy had explained the controls were “due to the fulfilment of the duty to control illegal trafficking” and that they were within the Schengen rules.
The statement added: “The head of the Spanish government reiterated his will to find, as soon as is possible, a solution to the current situation created by the Gibraltar authorities, which has produced a deep unease and a great concern because it hurts the environment and fishing activity.”
The call followed a formal protest by Britiain’s ambassador to Spain over “disproportionate” checks at the border.
Ambassador Giles Paxman visited Spanish foreign secretary Gonzalo de Benito to seek an official explanation for Spanish threats to levy a charge on vehicles and to close airspace.
A Foreign Office spokesman said: “Giles Paxman visited the Spanish secretary of state (for foreign affairs) Gonzalo de Benito to formally protest at the disproportionate border checks at the weekend.
"The ambassador re-iterated that the UK would stand shoulder to shoulder with Gibraltar in face of threats from Spain."
Spain has been accused by Gibraltar’s chief minister of “sabre-rattling” and behaving like North Korea after foreign minister Mr Garcia-Margallo announced proposals to levy border-crossing fees.
The Government has insisted there will be no compromise over the sovereignty of Gibraltar, and Foreign Secretary William Hague has vowed to stand “shoulder to shoulder” with its citizens.
A deal on ending the dispute, which dates back to the Treaty of Utrecht 300 years ago that ceded the territory to Britain, appeared close in 2002 after negotiations between then Europe minister Peter Hain and his Spanish counterpart.
Mr Cameron and Mr Rajoy last spoke about the situation in June before the latest deterioration in relations.
In recent days, Spanish ministers have raised the prospect of imposing a 50 euro (£43) levy on vehicles crossing the border and the possibility of closing airspace.
Mr Garcia-Margallo said the proceeds of a border fee could be used to help Spanish fishermen who have lost out because of damage to fishing grounds allegedly caused by Gibraltarian authorities following the creation of an artificial reef.
Gibraltar’s chief minister, Fabian Picardo, likened the comments coming out of Madrid to something from the Franco-era.