As usual, my advice, which has also come from others:
Look at the house carefully, think about what you are willing to pay. Make him a time-limited offer. And if not, then not. Then consider your limit for building and talk to the bank. Calculate your income. Look for a plot of land and build.
What are your thoughts? No matter what he thinks, wants, imagines, etc. You are interested in the house for price X. He tries, as mentioned, to emotionally manipulate you. He succeeded. But luckily you showed up here.
If someone else buys, then so be it. Stay firm if he says others want to pay amount y. Yes, good for the others.