可信计算>> DecisionTree.jl>> 返回
项目作者： bensadeghi

项目描述 ：
Julia implementation of Decision Tree (CART) and Random Forest algorithms
高级语言： Julia
项目地址: git://github.com/bensadeghi/DecisionTree.jl.git
创建时间： 2013-01-14T01:29:53Z
项目社区：https://github.com/bensadeghi/DecisionTree.jl
开源协议：Other
下载


DecisionTree.jl

CI
Codecov
Docs Stable

Attention: this package is now maintained by JuliaAI. Its new home is here.

Julia implementation of Decision Tree (CART) and Random Forest algorithms

Available via:

  • AutoMLPipeline.jl - create complex ML pipeline structures using simple expressions
  • CombineML.jl - a heterogeneous ensemble learning package
  • MLJ.jl - a machine learning framework for Julia
  • ScikitLearn.jl - Julia implementation of the scikit-learn API

Classification

  • pre-pruning (max depth, min leaf size)
  • post-pruning (pessimistic pruning)
  • multi-threaded bagging (random forests)
  • adaptive boosting (decision stumps)
  • cross validation (n-fold)
  • support for ordered features (encoded as Reals or Strings)

Regression

  • pre-pruning (max depth, min leaf size)
  • multi-threaded bagging (random forests)
  • cross validation (n-fold)
  • support for numerical features

Note that regression is implied if labels/targets are of type Array{Float}

Installation

You can install DecisionTree.jl using Julia’s package manager

  1. Pkg.add("DecisionTree")

ScikitLearn.jl API

DecisionTree.jl supports the ScikitLearn.jl interface and algorithms (cross-validation, hyperparameter tuning, pipelines, etc.)

Available models: DecisionTreeClassifier, DecisionTreeRegressor, RandomForestClassifier, RandomForestRegressor, AdaBoostStumpClassifier.
See each model’s help (eg. ?DecisionTreeRegressor at the REPL) for more information

Classification Example

Load DecisionTree package

  1. using DecisionTree

Separate Fisher’s Iris dataset features and labels

  1. features, labels = load_data("iris")    # also see "adult" and "digits" datasets
  3. # the data loaded are of type Array{Any}
  4. # cast them to concrete types for better performance
  5. features = float.(features)
  6. labels   = string.(labels)

Pruned Tree Classifier

  1. # train depth-truncated classifier
  2. model = DecisionTreeClassifier(max_depth=2)
  3. fit!(model, features, labels)
  4. # pretty print of the tree, to a depth of 5 nodes (optional)
  5. print_tree(model, 5)
  6. # apply learned model
  7. predict(model, [5.9,3.0,5.1,1.9])
  8. # get the probability of each label
  9. predict_proba(model, [5.9,3.0,5.1,1.9])
  10. println(get_classes(model)) # returns the ordering of the columns in predict_proba's output
  11. # run n-fold cross validation over 3 CV folds
  12. # See ScikitLearn.jl for installation instructions
  13. using ScikitLearn.CrossValidation: cross_val_score
  14. accuracy = cross_val_score(model, features, labels, cv=3)

Also, have a look at these classification and regression notebooks.

Native API

Classification Example

Decision Tree Classifier

  1. # train full-tree classifier
  2. model = build_tree(labels, features)
  3. # prune tree: merge leaves having >= 90% combined purity (default: 100%)
  4. model = prune_tree(model, 0.9)
  5. # pretty print of the tree, to a depth of 5 nodes (optional)
  6. print_tree(model, 5)
  7. # apply learned model
  8. apply_tree(model, [5.9,3.0,5.1,1.9])
  9. # apply model to all the sames
  10. preds = apply_tree(model, features)
  11. # generate confusion matrix, along with accuracy and kappa scores
  12. confusion_matrix(labels, preds)
  13. # get the probability of each label
  14. apply_tree_proba(model, [5.9,3.0,5.1,1.9], ["Iris-setosa", "Iris-versicolor", "Iris-virginica"])
  15. # run 3-fold cross validation of pruned tree,
  16. n_folds=3
  17. accuracy = nfoldCV_tree(labels, features, n_folds)
  19. # set of classification parameters and respective default values
  20. # pruning_purity: purity threshold used for post-pruning (default: 1.0, no pruning)
  21. # max_depth: maximum depth of the decision tree (default: -1, no maximum)
  22. # min_samples_leaf: the minimum number of samples each leaf needs to have (default: 1)
  23. # min_samples_split: the minimum number of samples in needed for a split (default: 2)
  24. # min_purity_increase: minimum purity needed for a split (default: 0.0)
  25. # n_subfeatures: number of features to select at random (default: 0, keep all)
  26. # keyword rng: the random number generator or seed to use (default Random.GLOBAL_RNG)
  27. n_subfeatures=0; max_depth=-1; min_samples_leaf=1; min_samples_split=2
  28. min_purity_increase=0.0; pruning_purity = 1.0; seed=3
  30. model    =   build_tree(labels, features,
  31.                         n_subfeatures,
  32.                         max_depth,
  33.                         min_samples_leaf,
  34.                         min_samples_split,
  35.                         min_purity_increase;
  36.                         rng = seed)
  38. accuracy = nfoldCV_tree(labels, features,
  39.                         n_folds,
  40.                         pruning_purity,
  41.                         max_depth,
  42.                         min_samples_leaf,
  43.                         min_samples_split,
  44.                         min_purity_increase;
  45.                         verbose = true,
  46.                         rng = seed)

Random Forest Classifier

  1. # train random forest classifier
  2. # using 2 random features, 10 trees, 0.5 portion of samples per tree, and a maximum tree depth of 6
  3. model = build_forest(labels, features, 2, 10, 0.5, 6)
  4. # apply learned model
  5. apply_forest(model, [5.9,3.0,5.1,1.9])
  6. # get the probability of each label
  7. apply_forest_proba(model, [5.9,3.0,5.1,1.9], ["Iris-setosa", "Iris-versicolor", "Iris-virginica"])
  8. # run 3-fold cross validation for forests, using 2 random features per split
  9. n_folds=3; n_subfeatures=2
  10. accuracy = nfoldCV_forest(labels, features, n_folds, n_subfeatures)
  12. # set of classification parameters and respective default values
  13. # n_subfeatures: number of features to consider at random per split (default: -1, sqrt(# features))
  14. # n_trees: number of trees to train (default: 10)
  15. # partial_sampling: fraction of samples to train each tree on (default: 0.7)
  16. # max_depth: maximum depth of the decision trees (default: no maximum)
  17. # min_samples_leaf: the minimum number of samples each leaf needs to have (default: 5)
  18. # min_samples_split: the minimum number of samples in needed for a split (default: 2)
  19. # min_purity_increase: minimum purity needed for a split (default: 0.0)
  20. # keyword rng: the random number generator or seed to use (default Random.GLOBAL_RNG)
  21. #              multi-threaded forests must be seeded with an `Int`
  22. n_subfeatures=-1; n_trees=10; partial_sampling=0.7; max_depth=-1
  23. min_samples_leaf=5; min_samples_split=2; min_purity_increase=0.0; seed=3
  25. model    =   build_forest(labels, features,
  26.                           n_subfeatures,
  27.                           n_trees,
  28.                           partial_sampling,
  29.                           max_depth,
  30.                           min_samples_leaf,
  31.                           min_samples_split,
  32.                           min_purity_increase;
  33.                           rng = seed)
  35. accuracy = nfoldCV_forest(labels, features,
  36.                           n_folds,
  37.                           n_subfeatures,
  38.                           n_trees,
  39.                           partial_sampling,
  40.                           max_depth,
  41.                           min_samples_leaf,
  42.                           min_samples_split,
  43.                           min_purity_increase;
  44.                           verbose = true,
  45.                           rng = seed)

Adaptive-Boosted Decision Stumps Classifier

  1. # train adaptive-boosted stumps, using 7 iterations
  2. model, coeffs = build_adaboost_stumps(labels, features, 7);
  3. # apply learned model
  4. apply_adaboost_stumps(model, coeffs, [5.9,3.0,5.1,1.9])
  5. # get the probability of each label
  6. apply_adaboost_stumps_proba(model, coeffs, [5.9,3.0,5.1,1.9], ["Iris-setosa", "Iris-versicolor", "Iris-virginica"])
  7. # run 3-fold cross validation for boosted stumps, using 7 iterations
  8. n_iterations=7; n_folds=3
  9. accuracy = nfoldCV_stumps(labels, features,
  10.                           n_folds,
  11.                           n_iterations;
  12.                           verbose = true)

Regression Example

  1. n, m = 10^3, 5
  2. features = randn(n, m)
  3. weights = rand(-2:2, m)
  4. labels = features * weights

Regression Tree

  1. # train regression tree
  2. model = build_tree(labels, features)
  3. # apply learned model
  4. apply_tree(model, [-0.9,3.0,5.1,1.9,0.0])
  5. # run 3-fold cross validation, returns array of coefficients of determination (R^2)
  6. n_folds = 3
  7. r2 = nfoldCV_tree(labels, features, n_folds)
  9. # set of regression parameters and respective default values
  10. # pruning_purity: purity threshold used for post-pruning (default: 1.0, no pruning)
  11. # max_depth: maximum depth of the decision tree (default: -1, no maximum)
  12. # min_samples_leaf: the minimum number of samples each leaf needs to have (default: 5)
  13. # min_samples_split: the minimum number of samples in needed for a split (default: 2)
  14. # min_purity_increase: minimum purity needed for a split (default: 0.0)
  15. # n_subfeatures: number of features to select at random (default: 0, keep all)
  16. # keyword rng: the random number generator or seed to use (default Random.GLOBAL_RNG)
  17. n_subfeatures = 0; max_depth = -1; min_samples_leaf = 5
  18. min_samples_split = 2; min_purity_increase = 0.0; pruning_purity = 1.0 ; seed=3
  20. model = build_tree(labels, features,
  21.                    n_subfeatures,
  22.                    max_depth,
  23.                    min_samples_leaf,
  24.                    min_samples_split,
  25.                    min_purity_increase;
  26.                    rng = seed)
  28. r2 =  nfoldCV_tree(labels, features,
  29.                    n_folds,
  30.                    pruning_purity,
  31.                    max_depth,
  32.                    min_samples_leaf,
  33.                    min_samples_split,
  34.                    min_purity_increase;
  35.                    verbose = true,
  36.                    rng = seed)

Regression Random Forest

  1. # train regression forest, using 2 random features, 10 trees,
  2. # averaging of 5 samples per leaf, and 0.7 portion of samples per tree
  3. model = build_forest(labels, features, 2, 10, 0.7, 5)
  4. # apply learned model
  5. apply_forest(model, [-0.9,3.0,5.1,1.9,0.0])
  6. # run 3-fold cross validation on regression forest, using 2 random features per split
  7. n_subfeatures=2; n_folds=3
  8. r2 = nfoldCV_forest(labels, features, n_folds, n_subfeatures)
  10. # set of regression build_forest() parameters and respective default values
  11. # n_subfeatures: number of features to consider at random per split (default: -1, sqrt(# features))
  12. # n_trees: number of trees to train (default: 10)
  13. # partial_sampling: fraction of samples to train each tree on (default: 0.7)
  14. # max_depth: maximum depth of the decision trees (default: no maximum)
  15. # min_samples_leaf: the minimum number of samples each leaf needs to have (default: 5)
  16. # min_samples_split: the minimum number of samples in needed for a split (default: 2)
  17. # min_purity_increase: minimum purity needed for a split (default: 0.0)
  18. # keyword rng: the random number generator or seed to use (default Random.GLOBAL_RNG)
  19. #              multi-threaded forests must be seeded with an `Int`
  20. n_subfeatures=-1; n_trees=10; partial_sampling=0.7; max_depth=-1
  21. min_samples_leaf=5; min_samples_split=2; min_purity_increase=0.0; seed=3
  23. model = build_forest(labels, features,
  24.                      n_subfeatures,
  25.                      n_trees,
  26.                      partial_sampling,
  27.                      max_depth,
  28.                      min_samples_leaf,
  29.                      min_samples_split,
  30.                      min_purity_increase;
  31.                      rng = seed)
  33. r2 =  nfoldCV_forest(labels, features,
  34.                      n_folds,
  35.                      n_subfeatures,
  36.                      n_trees,
  37.                      partial_sampling,
  38.                      max_depth,
  39.                      min_samples_leaf,
  40.                      min_samples_split,
  41.                      min_purity_increase;
  42.                      verbose = true,
  43.                      rng = seed)

Saving Models

Models can be saved to disk and loaded back with the use of the JLD2.jl package.

  1. using JLD2
  2. @save "model_file.jld2" model

Note that even though features and labels of type Array{Any} are supported, it is highly recommended that data be cast to explicit types (ie with float.(), string.(), etc). This significantly improves model training and prediction execution times, and also drastically reduces the size of saved models.