The best is a specialist kite shop - you can get good advise, good after sales care, etc. but not everyone has a kite shop near them, or would feel confident going into one. However in 2021 the chances of finding a specialist shop is increasingly hard - most (the ones that have survived) retailers are online.
I would strongly recommend having a go with the expensive and/or powerful kites before buying - from my experience most flyers are only to happy talk about their kites & let you have a go.
I got my first (& second) kite from the French sports shop chain Decathlon. If starting out & don't know much about this then go for a relatively cheap kite - a Delta costs around £15 including lines and handles. Once you have got to grips with the basics & know how to fly the kite & are sure that you want to continue then it is worth spending more money, especially if you have had a go on the different types.
Although people who are into kite flying in a big way end up spending quite a lot of money on large kites it doesn't mean you can't have great fun with a small cheap kite. At the end of the day it is about having fun, not how big your kite is.
If you are in the UK you will find certain kites (e.g. Revolution or Guildworks kites) are a lot cheaper in the USA or Canada. However think carefully before purchasing from abroad:
Its a very good choice these, but be especially careful of powerful traction kites selling cheaply - they might not have the care and attention to detail that "branded" kites have. You will be trusting this kite to pull you along, so safety is important.
As for stunt kites - there will be loads of choice -either new kites from a proper kite retailer, second hand, or cheaper Chinese made kites. As always do your homework, and if it seems to good to be true, it almost certainly is.
Also UK/EU buyers beware - the UK's exit from the European Union makes buying and selling more expensive and troublesome than before, so check carefully. That bargain might not end up being the bargain it seems if you have to add VAT and such to the price.