Released in 2001, the Game Boy Advance was the first true handheld successor to the GameBoy. It held power comparable to a portable SNES, and as a result, the game library was expanded to include many ports from Super Nintendo games, and even some rudimentary 3D titles. It objectively boasts the best game library of any GameBoy, with the staggering ability to play any game released for the original model, the Color, and the new Advance titles, which I consider to be the best overall. The design was shifted to be more horizontal, in other to prioritize increased comfort, and while the lack of a backlight was especially noticeable in the 21st century, the screen was improved to be easier to play in low-light conditions. Definitely an incredible successor.
Later, in 2003 and 2005, the Game Boy Advance SP and Micro systems were released, bringing in new form factors specific to each, and finally, thankfully, a backlight, making the SP arguably the Ultimate Gameboy model. It released with a clamshell design, the new backlight, and a charing port that eliminated the need to use batteries, instead using a cell-phone style charger. The only (arguable) downgrade might be the new form factor, which both protects the screen, and creates a much more constricted space for your fingers, and while I've never personally had an issue with this, many people with larger hands have definitely complained about this.
However, the Micro puts both other variants to shame, with it's incredibly small design. Unfortunately, it also loses out on the biggest draw of the GameBoy Advance; backwards compatibility, as it became literally so small that the hardware required to play original gameboy and gbcolor games wouldn't fit. Definitely more of a novelty than anything else, and it's by far the most portable variant, but for longer play sessions, a better option would be advised.
The original Advance and the GB Advance Sp are tied for the ultimate model, depending on whether or not you prefer the backlight (or lack thereof), or the ergonomics of either model. The Micro is by far the most expensive model due to it's novelty, and it's small size discourages the very thing it was made for; playing games portably, leaving it as a novelty.
Especially since these are beginning to break due to age, meaning ones in good condition are getting harder and harder to find nowadays, increasing the price. Unless you want to emulate all your games, which hey, more power to you, these are your best option by far. Good luck!