Are there contractually agreed deadlines?
The completed service phases will have to be paid for.
Keep in mind in your approach, especially if you perhaps live in a rural area, your reputation as a non-payer spreads quickly within the industry.
Your architect is not delivering because, like everyone else, he has too many orders. You won’t be able to exclude this with the next one, but you can be more meticulous with the (written fixed) deadlines. Deriving a reason for termination from delay alone is not enough; there must be contractual penalties, otherwise it’s toothless. Whether someone will sign that for you is another matter.
I would go through with it now. Delay must always be expected; this time it was only the architect, next will be the trades (or not ). Especially in renovation, there are always surprises.