Objective This paper aims to explore a new method for constructing tree height-DBH model, and combine quantile regression with nonlinear mixed effect method to construct tree height-DBH model, so as to improve the fitting accuracy of the model.

Method Based on the measured tree height and DBH data of 1 306 Cunninghamia lanceolata trees in the 30 m × 30 m fixed sample plot of C. lanceolata in Jiangle State-Owned Forest Farm of Fujian Province, eastern China in 2018, the basic model with the best fitting effect was selected from four tree height-DBH models. Based on the basic model, the tree height-DBH model was constructed by nonlinear mixed effect, quantile regression and nonlinear quantile mixed effect. The evaluation indexes of RMSE, R_\mathrmadj^2 and MSE were used to evaluate and compare the fitting results of each model. Akaike information criterion(AIC), Bayesian information criterion(BIC) and log likelihood (Loglik) were used to compare the fitting accuracy and prediction accuracy of each optimal model.

Result According to the comparison of evaluation indicators, the Logistic model was the basic model. The fitting effect of nonlinear mixed effect model was the best (AIC = 3 953.986, BIC = 3 988.199, Loglik = −1 969.993), and the fitting effect of the nonlinear quantile mixed effect model (AIC = 3 979.418, BIC = 4 028.293, Loglik = −1 979.709) was only slightly lower than that of the nonlinear mixed effect model. The order of model fitting effect was nonlinear mixed effect model > nonlinear quantile mixed effect model > basic model > quantile regression model. By comparing the residual sample plots of each model, it can be seen that there was no heteroscedasticity. The order of prediction effect was nonlinear mixed effect model > nonlinear quantile mixed effect model > basic model > quantile regression model.

Conclusion This study combines quantile regression with nonlinear mixed effect method. This method explains the differences and associations between individuals at different quantiles in the grouped data structure, and improves the stability and fitting accuracy of the model. It is a feasible idea to apply this method to the study of tree height-diameter relationship, and provides a new method for constructing tree height-DBH model.