I don't want to sound mean, but he is doing it because you are letting him.
It isn't a serious "behavioral" issue - this is evident in the fact that he doesn't do this to anyone but you.
There are MANY ways to handle this - but the bottom line is you have to stand up for yourself and teach him about respecting you.
Don't avoid him or the behavior. He is old enough for you to explain the situation to him, when he isn't angry find a way to bring up the topic and explain what the consequences will be if he chooses to disrespect you by biting, scratching, or hitting you.
Make him repeat the consequences back to you, so that you know that he knows that he understands. I don't know what your personal discipline style is, but you could take things away for a preset period of time, sit him in a corner, or on a step, cancel an outing - anything to let your kid know that you mean business.
It is most important to do this when he IS NOT ANGRY - you will get a better response out of him!

