The group of international lawyers alleges that Mr Rumsfeld personally approved the use of torture to extract information from the prisoners.
Wolfgang Kaleck, the lawyer leading the attempt to bring the case, said former US Army Brig-Gen Janis Karpinski would be the "star witness".
Ms Karpinski was commander of US prisons in Iraq when several prisoners were abused by US soldiers at Baghdad's Abu Ghraib facility.
Mr Rumsfeld resigned on Wednesday following Republican losses to the Democrats in the US mid-term elections.
The US denies any torture has taken place at Guantanamo Bay and has defended its interrogation techniques.
Nov 14, 2006
Kicking Rumsfeld to the curb wasn't enough
Now the Center for Constitutional Rights is asking German authorities to sue the former Defense Secretary over Abu Ghraib and Guantanamo.