Cavusgolu and Pompeo met on the sidelines of a meeting of regional ministers in Singapore.
This comes after the US slapped sanctions on Turkish government officials earlier this week for their role in his capture.
"I made clear that it is well past time that pastor Brunson be free and permitted to return to the United States and that the others being held by Turkey also similarly be freed as well", Pompeo told reporters in Singapore.
Cavusoglu, however, said both the countries have agreed to work together.
"We have said since the beginning that threatening language and sanctions will not get any result", he said.More news: Trump Administration Aims to Freeze Fuel Standards, End Tougher California Rules
White House spokeswoman Sarah Sanders said that Brunson was a "victim of unfair and unjust attention by the government of Turkey" as she announced sanctions. The United States has also been seeking the release of three locally employed embassy staff detained in Turkey. But he said threats would not work.
The U.S. on Wednesday imposed sanctions on Interior Minister Suleyman Soylu and Justice Minister Abdulhamit Gul for not releasing American Pastor Andrew Craig Brunson, who faces terrorism charges in Turkey.
"Since the beginning we said that a solution can not be reached by using threatening language and sanctions", the minister added.
Mr. Brunson had denied all charges and his case has become a cause celebre for US evangelicals and the Trump administration. He was arrested in December 2016 following a failed coup on charges of "committing crimes on behalf of terror groups without being a member" and espionage.
Before his move home last week, Brunson, who led a Protestant church in the Aegean city of Izmir, had spent almost two years in jail.More news: Weinstein lawyers ask court to toss sex case, citing emails
The former prime minister's position echoes that of the Turkish government. Mr. Erdogan's government also alleges the Mr. Brunson has ties to a Turkish preacher now living in exile in the USA whom Ankara accuses of fomenting the coup.
But Brunson's case has resonated with U.S. President Donald Trump and more particularly with Vice President Mike Pence.
The Turks have also vowed to retaliate for the sanctions "without delay". Gulen denies the claim.More news: Thibaut Courtois tells Chelsea to sell him this summer