What to do when a collaborator makes mistakes?
We all know that a serious mistake can destroy one's mathematical career. So it can be very stressful when a coauthor makes claims which turn out to be false. The key issues are whether the collaborator is publishing and presenting false theorems or just suggesting them in the collaboration process and then revising the theorem as necessary when you find the error. As I am the kind of collaborator who regularly throws out "potential theorems" and lets my collaborators dismantle them until we end up with a true theorem, I thought I might address this question myself.
First it is essential to determine whether a potential coauthor has published mistakes in the past, before deciding to work with them. Again I use myself as an example. Looking me up on mathscinet we see I have one errata and that errata is for an example within a paper and does not affect any of the theorems. Looking me up on the arxiv one sees that none of my preprints were ever retracted. One sees that I do repost the preprints up to about four times, sometimes adding details to proofs or fixing a lemma here and there. Finally, one can look at my publications and see whether the proofs are done in the level of detail one might desire. Looking someone else up, an occasional errata and an occasional retracted preprint shouldn't really scare someone away as a coauthor, but you know the level of stress that you can handle. No errata at all might just indicate an inability to accept an error exists in a paper and that can be more difficult to deal with in a coauthor than having an errata. Asking a potential coauthor if they ever made a mistake and how they dealt with it can give you some insight into whether he or she handles things as you would, before starting a collaboration. Remember, some people don't submit errata because their advisors tell them not to or the journals refuse to publish errata. They may still repost with a correction on the arxiv.
Once one is coauthoring, it is important to establish a level of communication which indicates when one believes a theorem is proven or not. I find a running tex file with "theorem" statements and "proofs" that refer to specific lemmas needed with their "statements" and "proofs" keeps track of things quite well. I will often rapidly write out outlines of complete proofs for potential theorems where each component is marked with "Check this". My coauthors check the various lemmas or restate and replace them (and propose potential theorems themselves). We go back and forth this way until nothing is left that needs to be corrected or checked. I have had coauthors who were disturbed by the idea that I am texing things which are not yet proven. This is because I work directly in tex (in part due to habitual long distance communication and in part because I lose track of things written on paper). When both coauthors are comfortable with a running tex file that has potential theorems and mistakes that will be worked out and corrected before posting on the arxiv, this system is fine. When a coauthor prefers to have only correct statements and proofs texed, the solution is to have a second file which only contains carefully checked and perfected statements and proofs.
There is a much deeper problem when a coauthor proves something incorrectly and refuses to believe that it is incorrect. It can be nearly impossible to find a counter example in my field of research, so one needs to be willing to listen to the possibility that a theorem is incorrectly proven without asking one's coauthor to find a counter example. On some level, it doesn't matter if a proof is correct or not, it cannot be put in a paper unless both authors follow it and believe it. The coauthor who suspects a mistake can either say that a certain step is incorrect because it cites some result incorrectly or makes an assumption that isn't in the hypothesis or so on, or the coauthor can leave off the accusation that there is a mistake and just say he or she does not follow the particular step. In my case, a coauthor will often point out a suspected mistake and I will immediately agree and we work to handle it. This is because I often work from my head and later, when checking references, discover I use theorems more strongly than I should of. I consider this part of the refinement of the proof and am not defensive if I have made such an error. However if a coauthor is very upset by an accusation that he or she has made an error, then phrasing things as "not understanding a step" can avoid confrontations. I myself become annoyed if someone says I proved Lemma X is wrong without telling me a more precise location for the error. "I changed the proof of Lemma X because it was wrong" is less polite than "I changed the proof of Lemma X because I couldn't follow it" while ideally one can say "I was concerned that line Y of Lemma X wasn't justified so I proved the lemma using a new approach altogether." This last phrase is both polite to the coauthor and sounds a lot more intelligent than the second phrase which is perhaps too subservient. Presumably, with communication of this sort, huge differences should not arise over errors made during collaboration.
When the problem persists, there is the possibility of finding an expert mediator (see III.H.) and even the possibility of breaking up a collaboration (see III.I.). Ultimately, if a paper is posted on the arxiv as a joint paper and published as a joint paper, then all authors are responsible for any mistakes. If you are concerned that there is a part of the paper that you do not want to claim is correct (perhaps because you never followed the proof), you might ask the coauthor to put it in some separate sole-authored work or to put it in a sole-authored appendix. Asking a coauthor to do this does not mean you don't believe the proof just that you haven't had the time to check it on the level you would like. It also admits that this aspect of the paper is exclusively credited to the coauthor even if you believe it and have checked it. For example, one of my coauthors came up with a key example proving one of our hypothesis was necessary with an additional person who was an expert in the field the example came from. So we put this example in an appendix coauthored by the two of them without my name on it. This allowed us to immediately add the example to the paper without the delay of my reading it over and also gave them full credit for the example. Other papers I've seen have appendices with technical theorems needed as ingredients or interesting applications proven by only one of the coauthors or possibly even an additional collaborator not listed as a coauthor on the paper itself. This is perfectly natural and should lead to no arguments between coauthors.
If an error is found in a published paper, coauthors need to work together to handle the problem. The editor of the journal may also be involved and can act as a leader as to how quickly an errata must be posted or if there is time to post an errata with a corrected proof (hopefully). Sometimes an errata is both found and corrected by a third party and a paper is published with the corrected proof by that person alone or with one or both of the original authors (in which case the errata generally announces the upcoming paper correcting it). An errata should be very specific as to where the error occurred and how theorems may need to be restated or which theorems need to be retracted. There is no reason for the collaboration to fall apart because you have made a mistake together although perhaps by coauthors will need to be more cautious in the future.