Father of my children denying a visit. No court ordered custody. He is NOT listed on birth certs. Help?

The father of my children and I do not have a court ordered custody or visitation agreement. He lives in Arizona and I live in Alabama because I am married to a man stationed here in the Army. My kids were supposed to visit me for the summer. Their father has changed his mind and is saying he wants to file for custody before letting them visit. Two years ago I notarized a letter giving him temporary guardianship of the children so he could enroll them into school while with him because the school would not let him since he is not listed as the father on their birth certificates. Their father is threatening me with this letter saying it legally gives him custody. I am married to a man in the US Army, I have no criminal record and we have a big house with rooms for each child. Their father lives with a girlfriend in a two room apartment and five children living in the apartment. Both he and his girlfriend have criminal records, most of which are drug related. I have the money to buy plane tickets as we speak. What can I do to get them on a plane to visit me by the end of May 2011??? I just want my children and I haven't seen them in two years. He physically and emotionally abused me and I am still intimidated by him which is why I have allowed him to keep them for so long but I am ready to fight him now. I want to stand up to him and I know my children will live a better life with me having custody and him getting visitation. What can I do????