Inverse constraint learning (ICL) recovers constraints from constrained demonstrations to be used in new scenarios. However, ICL suffers from an ill-posedness, leading to task-relevant reward present in the recovered constraints, a major factor of inaccurate inference of constraints from demonstrations. We introduce a transferable constraint learning (TCL) algorithm that jointly infers a task-relevant reward and task-agnostic constraint, enabling the generalization of inferred constraints. TCL additively decomposes the overall reward from the inverse reinforcement learning (IRL) algorithm into a task reward and a soft constraint, by maximizing the divergence between task-relevant policy and constraint policy.  Our statistical evaluation in three simulated two-dimensional manipulation environments shows TCL outperforms state-of-the-art IRL and ICL algorithms. Further, we demonstrate the effectiveness of TCL on two real-world object carrying tasks.