Since some answers were quite mixed up or even wrong, here is a summary once again. If you cancel the Riester contract, you have to pay back the subsidies and tax savings, and the capital gains must be taxed. Since you never declared it on your tax return, this is 175.--€/person/year plus child allowance. That amount is deducted immediately and you will have the money within 6 months. Since it is a home savings contract with low interest, you do not have to pay much additional tax on it in the next tax return.
If you withdraw money without canceling the contract, a housing promotion account is set up and the amount is credited with 2% interest annually. You must then pay tax on the final amount during retirement. If you no longer use your own apartment yourself, this is considered a use damaging the subsidy and is treated like a cancellation (see above). Exceptions only apply for a nursing home, but even there you can often own an apartment.