Are there multiple professors you're interested in working with in the department (US based)? Are there multiple professors in the department working on topics you're interested in?
Is there a weekly seminar in your research area?
What are the written and oral qualifying requirements? Will more time spent on coursework at the outset benefit you (US based)? What about an accelerated path? Are the options for testing out if you already have mastered the relevant background material?
How many years of funding does the department guarantee (typically 5-6 in the US)?
Will you earn a masters degree along the way (US based)?
What is the overall environment like? Do graduate students collaborate with each other? Is the environment competitive? Do graduate students organize their own learning seminars?
What is the stipend to cost of living ratio? Are there good opportunities to participate in hobbies and socialize in the local town? What will your everyday life look like?
If you have multiple offers and fellowships, you can sometimes negotiate for a better offer/stipend.
Look for a (not necessarily mathematical) mentor within the department at the outset of your studies. Ideally this person is a faculty member or older graduate student who you can trust and turn to for advice if and when any interpersonal issues arise.
Figure out what works best for you. Experiment with the times of day that are best for you to do research and other departmental duties. Get a sense of how much time you can productively spend thinking about a challenging topic before taking a break becomes the optimal solution. Find out where you like to work - at the office, at home, at coffee shops, at the library etc. - and how long you like to work. Doing math well can look very different for different people.
Choose a mathematical mentor (advisor) with whom you get along well
Pay attention to the problems and questions you're enjoying the most. This can guide your research program