Moved out of state with children(ex signed court order to allow). Now he is wanting one child to stay with him.

I went to court and my ex agreed to allow me to move out of state with children. I moved during the summer and children stayed with their dad for vacation. Now it is coming time for them to come here and ex is telling one child that she can lived with him and it is her choice. My daughter just says she doesn't want to leave her friends. I know my son has to come as the papers are already signed; but there is something I am worried about. My fiance is on parole for possession drug charges. He is clean and has to take regular tests and go to meetings as part of his parole. He has a very good family that all live close and me and my children would want for nothing. If this would come out would I lose custody of my child. I have always put my children first and if there is a chance I would lose custody I would move out and not live with my fiance.