How to Exchange Ideas without Giving them Away? How to sound out potential coauthors? When does a discussion mean an acknowledgement and when does it mean you will write up a paper together?
Once you are alone with someone, who you might wish to coauthor with, then you might reveal some ideas in more detail being very explicit that these are your ideas and that you are wondering if he or she would like to examine the problem together with you. At this point you are offering them coauthorship if they work with you and you are conveying to them that you expect them to keep your ideas between the two of you (unless additional coauthors are brought on board). You should be doing this somewhere near the beginning of the project, not after you have already put in a year of effort on your own. Before telling them anything, you can ask if they are interested in possibly working with you on a project and when they might have time to work together (this semester, over summer break etc). There is no point in starting a collaboration if the other person will be too busy when you are available and visa versa.
Suppose you have already been working on a project for a long time and need help. You can be very precise asking someone a question which reveals nothing about the whole project. In fact, this is often a useful mathematical exercise. It is essential if you need a mathematician in a neighboring field. If the assistant comes up with a lemma or a reference in a day or so, they would get an acknowledgement and the lemma attributed to them within the work. Or they may write an appendix for you with their name on the appendix. Or sometimes, if they needed to prove a whole theorem, you might either decide to coauthor together or ask the person to write a paper and then cite that paper. Note if the theorem is really in a different field with a statement useful to your field, writing a separate paper will speed up the review process and allow you each to avoid learning a new field to follow each others' work.
Finally, there are times when people ask you questions about their project. If you spend just a day or two discussing it with them, you should expect no more than what has been described above. If the person has not asked you to coauthor, then presume you are getting only an acknowledgement and work under the assumption that you are building good will. If you feel tempted to give a more serious contribution, ask if they'd like you to join their project and make sure they are willing (and have checked with other coauthors) before you proceed to give more significant input. Whatever you do, do not go home, solve their problem and post a sole authored article on the web with your solution to their problem. You may feel you have discovered the solution and it is yours, but the other person has just told you the setting, what they had done so far and what had failed to work, which is a significant amount of information.
If you think you can solve someone else's problem with the information that person has given you, it is time to consider asking to coauthor with them, letting them know you think you have a solution. It is best to get them to agree with you in public that you are working on something together or to ensure in some other way that they don't just run off with your solution and not include you as a coauthor. You can quickly check mathscinet to see how often they've successfully collaborated in the past. You can check the arxiv to see if they have retracted articles or reposted them adding last minute additional acknowledgments and citations. You can contact other colleagues to ask about them (by phone) and see if the person has a reputation. Once you have decided to collaborate with them, make sure they have agreed to collaborate with you. They have the complete right to keep ownership of a problem for any number of reasons. They may have even been told they need sole authored articles for tenure. At this point, it is best not to be working on their problem and either give them a hint in exchange for an acknowledgement or do nothing at all.This is not to say you shouldn't protect yourself. If you have already been working on a problem and then someone starts talking to you about the same problem, it is important to stop them and inform them that you are working on the same question and establish immediately whether you will decide to work together combining ideas or not discuss anything together about it all. If its a famous problem, you might quickly establish that you are working on different aspects of the same problem: perhaps different cases. If not, both of you may well be racing to produce an arxiv preprint.
Keep in mind, that when people are combining research at a late stage, that both of you have the right to retain some related work for yourselves. You aren't necessarily coauthoring about everything you have ever thought about. You are working on a specific project together and are contributing ideas to that specific project. Your coauthor may very well be working with other people or doctoral students on a related project and then you should avoid pushing over to what they are doing. Once you have an established successful coauthorship with someone, that has lead to a coauthored paper, then you can start being more free to pass ideas back and forth with that person about a variety of topics, expecting that if you come to a new idea in this freeform way that you will naturally coauthor again. But even with an established coauthor, there are boundaries.