Cooperation is a fundamental characteristic of humans that enables complex social interactions and joint achievements beyond the capacity of individuals. Multi-agent reinforcement learning (MARL) is one prevailing approach employed to model such cooperative behavior. While MARL as a generic algorithm serves well for modeling agents’ different social intentions through adjusting the relations between their reward functions, theories from cognitive science suggest that MARL per se does not suffice for modeling human-unique cooperation. During cooperation, humans spontaneously establish a sophisticated framework of shared agency, in which a joint representation of an imagined central agent emerges automatically and owns normative power. We delve into the discrepancies between these two cooperation paradigms through two case studies. Inspired by the theory of shared intentionality in cognitive science, we further introduce the Imagined We framework, a novel approach that emulates human behavior across various tasks requiring joint efforts and communication.