Deep learning algorithms are increasingly popular for complex prediction and classification tasks, and hyperparameter configurations play an important role in algorithm performance. However, the best hyperparameter tuning strategy still remain unresolved. While grid search and random search can be used to detect better hyperparameters, they are costly for big deep learning algorithms and may not produce the optimal result. Bayesian Optimization balancing the exploration and exploitation trade-off shows significant improvement over grid search and random search in both efficiency and accuracy, but the algorithm makes computation on the entire domain, which can be still costly especially in higher dimension settings. In this paper, we propose a space adjustment algorithm selecting top percent points at each iteration that can be incorporated in additional to Bayesian Optimization framework to further reduce experimental cost and improve optimization efficiency. We show our algorithm’s adaptable nature to the response surface of hyperparameter configuration space. We demonstrate our algorithm’s outstanding performance compared with Efficient Global Optimization through a variety of test functions and an application to machine learning datasets.