Large-scale robotics swarms are becoming increasingly crucial for robotics applications due to their potential to solve complex tasks by relying on properties such as robustness, scalability, efficiency, and resilience. However, controlling these systems relying on local information is very challenging. Centralized methods using global information are often faced with the issue of scalability, significant computational resources, and mostly impractical. This paper presents a novel multi-agent  reinforcement learning (MARL) approach with graph neural networks (GNNs) to design decentralized control policies for large-scale robot swarms. GNNs are distributed architectures that exhibit good transferability and scalability properties. They operate over a network in an entirely local and decentralized manner; hence, they are well-suited for this task. They learn both state and structure information in data. We parameterized the value and policy networks with Message Passing Graph Neural Networks. We trained our policies with heterogeneity constraints in the model using multi-agent Graph Neural Network Proximal Policy Optimization, an algorithm for training MARL policies that leverages a Graph Neural Network for differentiable inter-agent communication. We analyze segregative and aggregative navigation of heterogeneous robot swarms tasks on varying numbers and groups for three classes of 2D and 3D robots - holonomic point mass robots, non-holonomic ground robots, and quadrotors. Our policies, trained in simulation for holonomic point mass robots and zero-shot transferable to non-holonomic ground robots in physics-based Turtlebot3 robot swarms in Gazebo/ROS2 simulations, Crazyflie quadrotor swarms in Pybullet simulations and actual mobile robots in real environments simultaneously exhibit segregative or aggregative and navigation behaviors while avoiding collisions with each other. These behaviors have been widely studied in swarm robotics and have the potential for complex robotics tasks. Our experiments show that we can segregate or aggregate a swarm of heterogeneous robots with different dynamics using only local information. We showed the effect of Curriculum learning on the swarm performance and the scalability of our controller by exploring larger swarms and different groups of up to 30 robots. We demonstrate the successful deployment of the GNN policy learned in simulation to physical non-holonomic mobile robots - Turtlebot3 burger robots.