{"id":9819,"date":"2020-01-28T11:15:18","date_gmt":"2020-01-28T10:15:18","guid":{"rendered":"https:\/\/www.makingscience.com\/blog\/trees-and-forests-in-machine-learning\/"},"modified":"2020-01-28T11:15:18","modified_gmt":"2020-01-28T10:15:18","slug":"trees-and-forests-in-machine-learning","status":"publish","type":"post","link":"https:\/\/www.makingscience.com\/en\/blog\/trees-and-forests-in-machine-learning\/","title":{"rendered":"Trees and Forests in Machine Learning"},"content":{"rendered":"
Trees and Forests in Machine Learning<\/div>\n
Tree based learning algorithms are among the most popular and most frequently used supervised machine learning methods. They are versatile, can be applied on a large variety of problems, they\u2019re interpretable, and easy to use.\u00a0<\/span> First, let\u2019s see what those phrases in the previous sentence mean. <\/span>What are tree based learning algorithms?<\/b> The simplest of these is the decision tree, which exactly is as it sounds, as you can see in the image below. Starting from a root node, the algorithm makes decisions based on some predefined criteria about how to split the data, creating branches until it reaches a conclusion (leaf).<\/span> \"\" Source: By Stephen Milborrow – Own work, CC BY-SA 3.0, <\/span><\/i>https:\/\/commons.wikimedia.org \/w\/index.php?curid=14143467<\/span><\/i><\/a>   \u201cA tree showing survival of passengers on the<\/span><\/i> Titanic<\/span><\/i><\/a> (“sibsp” is the number of spouses or siblings aboard). The figures under the leaves show the probability of survival and the percentage of observations in the leaf. Summarizing: Your chances of survival were good if you were (i) a female or (ii) a male younger than 9.5 years with less than 2.5 siblings.\u201d<\/span><\/i>   Tree based algorithms are <\/span>supervised<\/b> methods, meaning that we need labeled data to train the model, to figure out what decisions to make and in which order. Only after training can we use the model to make predictions on unlabeled data.<\/span> Tree based algorithms can be used on both <\/span>classification<\/b> and <\/span>regression<\/b> problems. Classification problems are those where the variable we wish to predict has only a small number of discrete outcomes. For example in the image above, there are two outcomes: the person survived or not. Regression problems predict a continuous variable, for example, how much a customer will spend in an online shop.<\/span> An important aspect of tree based algorithms is that they are easily <\/span>interpretable<\/b>. As you can see in the image, it\u2019s quite simple to understand how the algorithm predicts what it predicts, what are the decisions it makes. In machine learning we often use algorithms that are practically \u201cblack boxes\u201d: they seem to work well, give accurate predictions, but we cannot tell how those conclusions were reached.\u00a0<\/span> In business settings \u201cblack box\u201d algorithms are often unacceptable no matter how good their scores are – which is understandable. If an algorithm refuses your mortgage application or predicts that you have a high chance to get a certain kind of disease, wouldn\u2019t you want to know why it reached that conclusion?<\/span> Of course, most problems are more complicated than the one in the above example and a simple decision tree is not sufficient to solve them. But ensemble methods provide enough flexibility to deal with very complex problems. Ensemble methods are constructed of several individually trained models, which are then combined. There are several such algorithms, let\u2019s take a quick look at the most popular ones.<\/span>   <\/p>\n

Random forest<\/b><\/h2>\n

Until a few years ago the Random Forest was considered as one of the most powerful machine learning algorithms. As you can guess from its name, a random forest model contains many decision trees. In order to train these trees, the algorithm samples the data randomly with replacement creating several subsamples and trains one tree on each of the subsamples. Then it combines the prediction of the trees, for example by using majority vote for a classification problem, and averaging the predictions for a regression problem.<\/span>   <\/p>\n

Boosting<\/b><\/h2>\n

Boosting algorithms are newer and even more powerful tools than Random Forest. While Random Forest builds trees in parallel,\u00a0 boosting algorithms build the trees one after the other, taking into account the weakest points of the previous models, and creating a model that strengthens those points in the next step, converting weak learners into a strong one as a result.<\/span> The two most popular methods both use gradient boosting. LightGBM\u00a0 was developed by Microsoft (<\/span>https:\/\/github.com\/microsoft\/LightGBM<\/span><\/a>), while XGBoost (<\/span>https:\/\/github.com\/dmlc\/xgboost<\/span><\/a>)\u00a0 was developed as an open source project. Both algorithms are fast, implement parallel processing, and can be used for a great variety of problems.<\/span>   <\/p>\n

Interpreting ensemble methods<\/b><\/h2>\n

As mentioned above, a huge advantage of tree-based methods is their interpretability, that it\u2019s possible to understand why the algorithm predicts what it predicts.\u00a0<\/span> All the algorithms we discussed in this post\u00a0 provide tools to examine the importance of each of the input features. There are also several external tools that help to interpret your model, such as xgboostExplainer for the R language (<\/span>https:\/\/github.com\/gameofdimension\/xgboost_explainer<\/span><\/a>) and LIME (<\/span>https:\/\/github.com\/marcotcr\/lime<\/span><\/a>) or SHAP (<\/span>https:\/\/github.com\/slundberg\/shap<\/span><\/a>) for Python. These tools can help you discover deeper connections between the features, and refine your\u00a0 models; you can even use them to create illustrations that help explain how your model works to non-experts.<\/span> The image below shows a figure created with SHAP using a model for predicting the survival of passengers of the Titanic (<\/span>https:\/\/meichenlu.com\/2018-11-10-SHAP-explainable-machine-learning\/<\/span><\/a>). It shows that females, first and second class passengers, and children had the best chance of survival.<\/span> \"\"<\/div>\n","protected":false},"excerpt":{"rendered":"

Trees and Forests in Machine Learning Tree based learning algorithms are among the most popular and most frequently used supervised machine learning methods. They are versatile, can be applied on a large variety of problems, they\u2019re interpretable, and easy to use.\u00a0 First, let\u2019s see what those phrases in the previous sentence mean. What are tree […]<\/p>\n","protected":false},"author":21,"featured_media":9822,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[913],"tags":[],"class_list":["post-9819","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology-ai-en"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.makingscience.com\/en\/wp-json\/wp\/v2\/posts\/9819","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.makingscience.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.makingscience.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.makingscience.com\/en\/wp-json\/wp\/v2\/users\/21"}],"replies":[{"embeddable":true,"href":"https:\/\/www.makingscience.com\/en\/wp-json\/wp\/v2\/comments?post=9819"}],"version-history":[{"count":0,"href":"https:\/\/www.makingscience.com\/en\/wp-json\/wp\/v2\/posts\/9819\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.makingscience.com\/en\/wp-json\/wp\/v2\/media\/9822"}],"wp:attachment":[{"href":"https:\/\/www.makingscience.com\/en\/wp-json\/wp\/v2\/media?parent=9819"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.makingscience.com\/en\/wp-json\/wp\/v2\/categories?post=9819"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.makingscience.com\/en\/wp-json\/wp\/v2\/tags?post=9819"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}