What is wrong with the lawyer's statement?
The agreement in a personal conversation, at the end of which there is a termination/cancellation/whatever/..., is usually the best way. Of course, the lawyer earns more from a lengthy legal dispute, so hardly any lawyer will give this advice, but rather immediately shoot with cannons at sparrows and put the full program into motion. So that speaks more for the lawyer than against him.
That the clause is a double-edged sword, he is also right about that. And precisely because of this, an amicable agreement is the most desirable goal here. No emails, no threatening registered letters. Go there, seek the conversation on site, be honest and done. If it doesn’t help and it comes to a dispute, then what counts anyway is what is in the contract; that does not change now.
ps. If the agreement is not successful, I believe it comes down to the fact that the construction company pulls a broker out of their pocket: 5 alternative plots, one of which you have to take and build on, or you will be accused: you don’t actually want to build, which means the clause "subject to plot" would not be applicable, but rather termination with... 10% or whatever... penalty clause.