I have a 6 year old son, and i just served his father with child support papers, he has not seen him in over 4 years no phone calls nothing at all, and now because i served his he tells me he is going to take me to court for visitation rights.

When you filed the CS, you should have had a parenting pan ready to file with the court.
It is not to late to offer your parenting plan; since the father is a stranger, the child will need visistion started in small increments
such as the first 6 weeks , a saturday supervised bisist of an hour
then the next 6 weeks 2 hours,
if he is still around by then, you could try 2 hours on saturday unsupervised
on like that ,
you need to confer with an aexperienced lawyer since you do not know the lving conditions , etc; you have a right to have a home visit done, to make sure the home is safe adn clean

somebody is gong to have to pay for all the extra care, the father since he dropped the ball and made himself a stranger
you need to reserach your states parenting plans and how the court prefers to do them

you cannot stop him from visiting his own child unles he is unfit or abusive/ neglectful
but you can make sure your child is safe and cared for in your absence