Law: Don't dishonor your father and mother.
Notice Jesus earthly father is never mentioned again. "But they did not understand the statement which He spoke to them."
Death is a powerful thing and it is not mentioned of the father. I propose that Joseph had the last straw and left. Jesus is of age, He is always shining. Let Him take care of the family. Poor poor Mary.
Joseph died in another land. Forgotten.