If you have no written agreement between you about who can claim the children, then the IRS will always side with the custodial parent.
You may find yourself faxing them pages of proof about where the children lived. If you get any letters from the IRS, respond to them as quickly as possible.

