The inmates were found to be missing during a security sweep on Wednesday night, after two other detainees were caught attempting to flee.
A spokesman for the US military said the escape was “regrettable”.
The jail, formerly known as Camp Cropper, was turned over to Iraqi authorities in July, but US forces retain custody of about 200 inmates.
They are some of the most dangerous prisoners, including those with ties to terrorist groups or Saddam Hussein’s former regime.
The escaped men are “senior members of al-Qaeda”, according to Baghdad security spokesman Major General Qassim al-Moussawi.
“We have no idea if they fled outside Baghdad. We are searching for them,” Mr Moussawi said, adding that the four were considered dangerous.
“We are working to apprehend these individuals,” said Major General Jerry Cannon, the US deputy commanding General for Detainee Operations.
Camp Cropper, which houses around 1,500 detainees, was the last US prison in Iraq. Its handover ended a difficult period in which thousands of people were held without charges and some were abused by American jailers.
The escape is the second since the US transferred custody of the detention facility to the Iraqis. A week after the handover, four al-Qaeda-linked detainees awaiting trial on terrorism charges escaped from the Iraqi section of the prison.