TRIPOLI, Libya (CNN) -- Libyan Prime Minister Ali Zeidan appealed for calm Thursday in his first public comments since he was freed after being abducted by militia gunmen for several hours.

In remarks to a Cabinet meeting broadcast on Libyan state TV, Zeidan said he did not want to see the situation escalate and urged Libyans to show "wisdom."

Zeidan's abduction from a luxury hotel early Thursday highlighted the security threat posed by militias that have run rampant in Libya since the revolution that ousted Moammar Gadhafi two years ago.

But the prime minister appeared to play down his kidnapping, characterizing it as an internal political problem.

"I want to reassure the foreigners inside Libya that this issue happened within the context of political Libyan disagreements and the foreigners are not being targeted," he said.

Immediately after Zeidan's release, Nouri Abusahmain, president of the Libyan General National Congress, said the prime minister was "in good shape" and "in good spirits" as he headed to his office.

His abduction prompted a slew of confusing reports.

Gunmen move in before dawn

The gunmen took Zeidan before dawn from the Corinthia Hotel in Tripoli and put him into a convoy of waiting cars, according to a hotel clerk who was not authorized to speak to the media.

The five-star hotel that Zeidan calls home is popular among government officials, some of whom reside there, including the justice minister.

The witness reported no gunfire during the incident and said the gunmen were respectful and "caused no trouble."

Zeidan's office initially called the abduction a "rumor" on its official Facebook page, but later posted an update that it was "coerced by kidnappers to deny the report."

His spokesman told CNN that the prime minister was kidnapped.

But the Operations Room of Libya's Revolutionaries, the militia that took him, said it merely detained him over financial and administrative corruption charges.

However, the Justice Ministry said there was no arrest warrant for Zeidan, calling the move a kidnapping. Abusahmain said the government was not aware of the charges and the prime minister was prepared to answer any questions.

The militia works with the Interior Ministry -- a not-altogether uncommon practice in Libya, which has tried unsuccessfully to rein in the many militia groups. Instead, various ministries have teamed up with them for their own needs, including providing security services.

U.S. Secretary of State John Kerry, on a visit to Malaysia, said he had spoken with U.S. Ambassador to Libya Deborah Jones and Washington as the situation in Libya evolved, and would stay in close touch.

"Our embassy personnel are secure. We're confident about our abilities to keep them in that security," Kerry said.

The situation underscores "something that we've been really focused on in these last months, which is building capacity in Libya," he added. It has been just over a year since U.S. envoy Christopher Stevens was killed in an attack on the U.S. Consulate in Benghazi, Libya.

British Foreign Secretary William Hague welcomed Zeidan's release. "We will work with Libyan government on ensuring the transition remains on track and insecurity is addressed," he tweeted.

Militias demand more autonomy

In the eastern part of the country, militias are demanding more autonomy from the central government and have severely constrained Libya's oil output, which is central to its export revenue.

Last month, Zeidan said the nation is trying to rebuild after decades under Gadhafi's rule. He shot down reports that Libya is a failed state.

"We are trying to create a state, and we are not ashamed of that," he told CNN's Christiane Amanpour. "The outside world believes that Libya is failing, but Libya was destroyed by Gadhafi for 42 years, and was destroyed by a full year of civil war. And that's why we are trying to rebuild it."

Part of Libya's rebuilding involves reconciliation and accountability, Zeidan said last month. Libya has an interim president, but the prime minister holds all executive powers.

Security an issue

Rights groups have said security remains a principal concern in Libya.

"The main problem affecting both justice and security is that armed militias still maintain the upper hand," Human Rights Watch said in a report last month. "They have various agendas -- financial, territorial, political, religious -- and operate with impunity two years after the Gadhafi regime ended. Successive interim governments have failed to assert control over these militias, preferring to contract them as parallel forces to the army and police."

In a statement issued Thursday, the rights group said it was "deeply troubled" by Zeidan's detention, which it described as a clear violation of Libyan law.

"The incident highlights the grave security conditions in Libya today. Hopefully it will refocus attention on the urgent need to strengthen both the country's security forces and its judicial systems," it said.

Recent attacks have added to the uncertainty in Libya.

Gangs of armed men have surrounded key ministries, including the Justice Ministry, trying to force out members of the democratically elected government.

Libyan Justice Minister Salah Marghani was forced to evacuate after armed militias surrounded his ministry in April.

Libyan intelligence services have warned that the country is becoming a haven for al Qaeda to regroup and regenerate itself.

Numerous weapons left over after Gadhafi's downfall are providing groups with different motivations to form their own militias, government officials said.

U.S. operation

On Saturday, U.S. forces swooped into Tripoli and seized a Libyan national indicted in the 1998 American Embassy bombings in East Africa.

Abu Anas al Libi is a suspect in the embassy attacks in Kenya and Tanzania, and American officials have described him as "one of the world's most wanted terrorists."

The militia group said the prime minister's capture had nothing to do with the arrest of al Libi. However, three days ago, it issued a warning that whoever collaborated with "foreign intelligence services" in the arrest of the terror suspect would be punished.

It's unclear if Zeidan played a role in the arrest of al Libi by U.S. forces.

But the Libyan government has decried the arrest of the terror suspect, and its national congress Tuesday demanded that the United States hand him over.

Human Rights Watch has also voiced concerns about the U.S. government's treatment of al Libi. "We repeat our call on the U.S. to grant al-Libi access to a lawyer, to charge him promptly before a judge," the group said.

CNN's Victoria Eastwood, Yousuf Basil and Jomana Karadsheh contributed to this report from Tripoli.