How can a pointing gesture or simple utterance come to mean so much? Unlike a code or fixed mapping that can be used to uniquely identify a referent, humans flexibly use the same signal to mean many different things. As a result, to formally capture the impromptu, sparse nature of communication, it should be viewed as an inferential process. Both sending and understanding a signal in context requires reasoning about the underlying mind at the other end. This dissertation takes a cognitive approach to developing a computational framework for this type of uniquely human communication. Even young children who cannot yet speak in full sentences use simple gestures and utterances in uniquely flexible and intelligent ways. This highlights the promise of a reverse engineering approach: the underlying cognitive mechanisms and commonsense reasoning accumulated during pre-linguistic development become the foundation for modeling intelligent communication.