Echoing Raphael's comments, this is highly subjective without knowing you personally. However, here are the things I'd consider:
Many schools are flexible for your undergrad program, meaning that what you choose likely won't affect your graduate career too too much, particularly if you take a good number of classes in each subject.
Do you like calculus/real analysis, or only discrete math? It varies from school to school, but if you're not into calculus at all, probably better to stick with CS.
Likewise, do you want to learn programming? If you have no interest in learning about caching, hardware, programming language concepts, or more "applied" aspects of computer science, then Math might suit you better. Many undergraduate CS programs serve the dual purpose of preparing coders for industry and preparing future grad students. You will likely have to take non-theoretical classes at some point.
Are you more interested in creating new algorithms, or analysing discrete structures? My guess is that CS will beter prepare you for creating new algorithms and solving problems, while Math will better prepare you for detailed proofs about discrete structure.
I would reccomend at least taking a few comp-sci classes. My experience is that people with a math backgorund, but not a computer science background, aren't equipped as well to communicate about things like algorithms and pseudocode. As an example, when talking about Breadth First Search in a math Graph Theory class, the professor had to go and define what a queue was. In a CS class, data structures will be something you cover once and then can keep using later on. Likewise, the pseudocode I've encountered in such classes was full of numbered steps and GOTOs, as apposed to the clean iterative or recursive definitions from CS classes.