I admire your "composure." I personally find 1 year very long. In terms of even a possible neighborhood at all, last year was definitely a big peace offering, but since the colleague has already shown a lack of any respect and manners in the past, I would now make sure he bleeds as quickly as possible until he finally drops the subject, especially since the learning curve also seems very flat and you probably don’t feel much urge for him to become your neighbor.
Have your lawyer, who probably knows your case completely, create further facts. After all, that apparently was the only thing that showed any effect in the past. If he now starts with the permit and this probably won’t work in the foreseeable future, you can safely count on next summer as the earliest date for the start of your garden; or, if the omitted construction obligation forces a sale, I would not assume that he feels responsible for anything and the next person with a permit etc. might perhaps start next year, if the municipality(?) can even enforce a new sale by then.
Just read a few pages back, the topic did come up before... I can understand the time with the kids well ;) Still keeping my fingers crossed!