Prisoner numbers must be cut to prevent further violence, the Parole Board chairman for England and Wales says.
Speaking after a riot at Birmingham prison, Nick Hardwick said there was only “a small window” in which to act.
The Ministry of Justice is recruiting an additional 2,500 prison officers to tackle prison violence.
But Mr Hardwick told the BBC the situation was “very grave” and it would be too long before the extra staff were trained and effective.
Mr Hardwick, formerly Her Majesty’s Chief Inspector of Prisons, told BBC Radio 4’s The World This Weekend programme that there were widespread fears of violence in the prison system.
HMP Birmingham, where inmates took over four wings and started fires on Friday, had not even been among the institutions causing the most concern, he said.
“The levels of violence and suicide and self-harm are not only increasing, but the rate at which they are increasing is accelerating.
“Successive ministers cannot say that they were not warned about this.”
Last month, Justice Secretary Liz Truss announced a £1.3bn investment in new prisons, including more prison officers, extra drug tests and greater autonomy for governors.
She has also promised a zero-tolerance approach to attacks on prison staff and said body-worn cameras would be provided at jails across the country.
But Mr Hardwick said the government must take urgent action to reduce the prison population of 85,000 to a more manageable figure.
“We are not prepared to pay for the size of the prison population that we now have, so the balance between the prison population and the number of staff that we’ve got is now unworkable,” he said.
“It would be a mistake to take emergency, reckless measures around the prison population. But if you don’t do things in a planned and sensible way then further down the track you may be forced into a much more difficult position.”
Alex Cavendish, a former prisoner turned academic who runs the Prison UK blog, also told the BBC he was concerned more prison officers was not the answer.
“I’m not convinced that they will be able to retain them, or that they will be able to get them deployed, trained effectively, onto wings in time to stop the next round of riots.
“And I am very, very much afraid that we are going to see more of this disturbance. Mercifully, at Birmingham nobody was seriously injured or killed, but that cannot be ruled out in the future.”