Building energy use accounts for 40% of the primary energy consumed worldwide. Therefore, it is essential to explore the key influencing factors and the associated interactions among these factors in order to achieve energy-saving goals. In practical scenarios, the sample size is always limited; thus, conventional models are unable to analyze the interactions among factors, or their results are unstable. To resolve this issue, this paper proposes a method that combines the improved stochastic impacts by regression on population, affluence, and technology (EM-STIRPAT) model and a structural equation modeling (SEM) model using the entropy weight method. The results obtained by analyzing the building energy use in Beijing, which was considered as a case study, indicate that economic level, technical level, industry structure, and education structure have a significant positive impact on building energy consumption intensity (BECI). Based on this analysis, it was found that 94% of the technical level and 39% of the economic level indirectly affect BECI through the energy structure. In addition, through a scenario analysis, it was found that future changes in the age structure of the population will have a smaller impact on BECI than changes in the education structure. Finally, based on the results of this study, policymaking recommendations to reduce building energy consumption are also presented.