This took me too long to figure out.
I was unable to get my custom domain working with GitHub Pages; whenever I attempted to go to my site via the custom domain I would get a GitHub Pages 404.
Turns out I had saved my CNAME file as unicode and there was a byte order mark at the beginning. Re-saving it as as an ANSI file and waiting ~15 minutes for GitHub to pick up the change did the trick.
It also may be that:
- You must have a newline at the end of the domain name at the end of the CNAME file.
- You must change the CNAME as an actual commit vs. just force pushing an entire history like the DocPad ghpages plugin does.
I can't be sure about either one of those as I did them at the same time and it's possible that GitHub coincidentally updated itself at the same time, if you continue to have trouble it may be worth giving that a shot.
It probably would have saved me some time if I had actually paid attention to the emails from the GitHub Pages builder with the "Bad CNAME format" warning in them...