Simulation plays a crucial role in modern academic study, particularly in the field of artificial intelligence (AI). The simulation environment can mimic real-world scenarios, allowing the AI agent to learn, adapt, and make decisions in a controlled and safe setting. This thesis tackles two important problems in building the next generation of artificial general intelligence (AGI): how to efficiently train an AI agent with values and how to overcome the simulation to reality gap to bring the training results to real-world applications. The current studies of AI mainly consider learning about the potential or energy function (U), referring to understanding the impact of the outside environment. The U function helps the agent apprehend the physical world laws, natural potentials, and social norms. However, taking into account the value learning, usually representing modeling one's inside thinking, benefits the agent to derive its goals, intents, and social values.