In this paper I will first describe the theoretical background of a certain kind of random process, namely, Markov Processes, and subsequently, the potential for their application in a game theoretic context. Consequently, these random processes can be used in a variety of game theoretic contexts to construct solutions to games whose state space can be realized as a discrete graph of nodes and the edges between them. Once the details of how Markov Decision Process based strategies may be implemented to develop convergent solutions are settled, this method will be tested by evaluating the performance of these solutions in Tic-tac-toe, Blackjack and Chess.