Bayesian networks are a popular class of graphical models to encode conditional independence and causal relations among variables by directed acyclic graphs (DAGs). In this thesis, we focus on developing algorithms to estimate Bayesian network structures. We propose two structure learning methods, and both of them minimize regularized negative log-likelihood functions over the space of orderings.