Like others have said, you've learned a valuable lesson. Unless I'm working on something that doesn't pay until publication, which is unfortunately all too common in my industry, I don't touch anything until I have payment in hand. I do my damnedest to get 100% upfront, but on the occasions when I accept 50% in advance I make sure there is documentation that clearly lays out the expectations, pricing and details of the project. You would be wise to start doing the same.
In your current situation, I think the best course of action would be to contact your client (though I'm not sure if you can call them that since they haven't paid) and set a deadline. Something along of lines of them having 72 hours to pay in full or you will begin removing the work that you still have access to change. If you're not talking about a ton of money, it's a pain in the ass to try going to court over it. Since you said at least part of it was link building, those would be the first to go.
If he doesn't pay, remove what you can, tell anyone you know in the industry not to work with this person in the future and move on while taking a lesson away from the situation.
Also, sorry that happened to you.
Live. Learn. Fix.