Since you are the buyer, did he leave you positive feedback already or is he playing the feedback hostage game?
My personal opinion is that the buyer should receive feedback first, and as soon as they pay and live up to their end of the agreement. Once payment has been received (in a timely manner, yada yada yada), the buyer no longer has any obligation to the deal and should be rated as how they met the payment obligation. However, many sellers feel that buyer feedback should only be given AFTER they receive positive feedback first. Since their obligations are significantly more and on-going (shipping the item timely, properly, etc.), their feedback should be given after.
On the rare occasions that I sell, I always give positive feedback as soon as payment clears. On the somewhat occasional times that I buy on eBay, I never give feedback until after it has been given to me. As a result, I quite often don't receive buyer feedback because I refuse to play those games with sellers.