Current behavior of custom domain (connected via Cloudflare) is when you remove the custom domain it goes ahead and deletes the entire domain in Cloudflare portal.
Graphy is not the only subdomain for a particular domain, there are others like MX records, TXT records, etc. At least give a warning before going ahead and deleting the domain on Cloudflare.