In my opinion, the general contractor will not let the option of the assignment declaration be taken away. Put yourself in the position of the general contractor. You regularly try to circumvent contractually regulated matters. For the general contractor, it is rather a fortunate coincidence that you did not meet the bank's payment requirements at the time of payment to serve the installment from the payment plan. This is not the first mistake you have made. Therefore, the general contractor would be pretty stupid to give up this newly gained leverage.
By the way, what does the payment plan look like? But take comfort, he can only demand what "he" believes to have performed.