I don't see the point of a cryptography site if it doesn't encompass all aspects of cryptography. The questions deemed on-topic look like they are currently accepted on Math, with perhaps a hint of CSTheory. If that's really the desired scope, the existing sites are fine.
I could see the point of a site that unites all aspects of cryptography, including the math and protocols (currently more Security or CSTheory or Computer Science Stack OverflowStack Overflow material) and implementation (currently Stack OverflowStack Overflow material) and usage (currently Security material).