JDBC SETUP

Driverless AI lets you explore Java Database Connectivity (JDBC) data
sources from within the Driverless AI application. This section provides
instructions for configuring Driverless AI to work with JDBC.

NOTE: Depending on your Docker install version, use either the

docker run --runtime=nvidia(>= Docker 19.03) ornvidia-docker(< Docker 19.03) command when starting the Driverless AI Docker image. Usedocker versionto check which version of Docker you are using. Tested Databases ----------------  The following databases have been tested for minimal functionality. Note that JDBC drivers that are not included in this list should work with Driverless AI. We recommend that you test out your JDBC driver even if you do not see it on list of tested databases. See the :ref:`untested-jdbc-driver` section at the end of this chapter for information on how to try out an untested JDBC driver. -  Oracle DB -  PostgreSQL -  Amazon Redshift -  Teradata  Description of Configuration Attributes ---------------------------------------  -jdbc_app_configs: Configuration for the JDBC connector. This is a    JSON/Dictionary String with multiple keys.

In addition, this should take the format:  ..     ::        """{"my_jdbc_database": {"url": "jdbc:my_jdbc_database://hostname:port/database",         "jarpath": "/path/to/my/jdbc/database.jar", "classpath": "com.my.jdbc.Driver"}}"""     For example:     ::        """{          "postgres": {          "url": "jdbc:postgresql://ip address:port/postgres",          "jarpath": "/path/to/postgres_driver.jar",          "classpath": "org.postgresql.Driver"          },          "mysql": {          "url":"mysql connection string",          "jarpath": "/path/to/mysql_driver.jar",          "classpath": "my.sql.classpath.Driver"          }       }"""     **Note**: The expected input ofjdbc_app_configsis a `JSON    string <https://docs.python.org/3/library/json.html>`__. Double    quotation marks ("...") must be used to denote keys and values    *within* the JSON dictionary, and *outer* quotations must be    formatted as either""",''', or'.

The following examples show two unique methods for    applying outer quotations.

For    example, "-Xmx4g". -jdbc_app_classpath: Optionally specify an alternative classpath    for the JDBC connector. -enabled_file_systems: The file systems you want to enable. This    must be configured in order for data connectors to function properly. Retrieve the JDBC Driver ------------------------  1. Download JDBC Driver JAR files:  ..     -  `Oracle       DB <https://www.oracle.com/technetwork/database/application-development/jdbc/downloads/index.html>`__    -  `PostgreSQL <https://jdbc.postgresql.org/download.html>`__    -  `Amazon       Redshift <https://docs.aws.amazon.com/redshift/latest/mgmt/configure-jdbc-connection.html#download-jdbc-driver>`__    -  `Teradata <https://downloads.teradata.com/download/connectivity/jdbc-driver>`__     **Note**: Remember to take note of the driver classpath, as it is    needed for the configuration steps (for example,    org.postgresql.Driver). 2. Copy the driver JAR to a location that can be mounted into the Docker    container.

Enable the JDBC Connector -------------------------  .. container:: tabs     .. group-tab:: Docker Image Installs     This example enables the JDBC connector for PostgresQL. Note that the    JDBC connection strings will vary depending on the database that is    used.

1. Configure the Driverless AI config.toml file. Set the following          configuration options:        ..           .. code:: bash              enabled_file_systems = "file, upload, jdbc"             jdbc_app_configs = """{"postgres": {"url": "jdbc:postgres://localhost:5432/my_database",                                  "jarpath": "/path/to/postgresql/jdbc/driver.jar",                                  "classpath": "org.postgresql.Driver"}}"""        2. Mount the config.toml file and requisite JAR files into the          Docker container.

**Notes**:           -  The JDBC connection strings will vary depending on the             database that is used. -  The configuration requires a JSON key (typically the name of             the database being configured) to be associated with a             nested JSON that contains theurl,jarpath, andclasspathfields. In addition, this should take the             format:           ::              """{"my_jdbc_database": {"url": "jdbc:my_jdbc_database://hostname:port/database",                 "jarpath": "/path/to/my/jdbc/database.jar", "classpath": "com.my.jdbc.Driver"}}"""        1. Export the Driverless AI config.toml file or add it to          ~/.bashrc. For example:        ..           ::              # DEB and RPM             export DRIVERLESS_AI_CONFIG_FILE="/etc/dai/config.toml"              # TAR SH             export DRIVERLESS_AI_CONFIG_FILE="/path/to/your/unpacked/dai/directory/config.toml"         2. Edit the following values in the config.toml file.

(jdbc_app_configs)             # hive: Hive Connector, remember to configure Hive below. (hive_app_configs)             # recipe_url: load custom recipe from URL             # recipe_file: load custom recipe from local file system             enabled_file_systems = "upload, file, hdfs, jdbc"              # Configuration for JDBC Connector. # JSON/Dictionary String with multiple keys. # Format as a single line without using carriage returns (the following example is formatted for readability). # Use triple quotations to ensure that the text is read as a single string.

Save the changes when you are done, then stop/restart          Driverless AI. Adding Datasets Using JDBC --------------------------  After the JDBC connector is enabled, you can add datasets by selecting **JDBC** from the **Add Dataset (or Drag and Drop)** drop-down menu. .. figure:: ../images/jdbc.png       :alt:   1. Click on the **Add Dataset** button on the Datasets page. 2. Select **JDBC** from the list that appears. 3. Click on the **Select JDBC Connection** button to select a JDBC    configuration. 4. The form will populate with the JDBC Database, URL, Driver, and Jar    information. Complete the following remaining fields:  ..     -  **JDBC Username**: Enter your JDBC username. -  **JDBC Password**: Enter your JDBC password. (See the *Notes*       section)    -  **Destination Name**: Enter a name for the new dataset. -  (Optional) **ID Column Name**: Enter a name for the ID column. Specify this field when making large data queries.

Instead,          enter the password in the **JDBC Password** field. The password          is entered separately for security purposes. -  Due to resource sharing within Driverless AI, the JDBC          Connector is only allocated a relatively small amount of          memory. -  When making large queries, the ID column is used to partition          the data into manageable portions. This ensures that the          maximum memory allocation is not exceeded. -  If a query that is larger than the maximum memory allocation is          made without specifying an ID column, the query will not          complete successfully. 5. Write a SQL Query in the format of the database that you want to    query. (See the `Query Examples <#queryexamples>`__ section below.) The format will vary depending on the database that is used. 6. Click the **Click to Make Query** button to execute the query.

On a successful query, you will be returned to the datasets page, and the queried data will be available as a new dataset. .. _queryexamples:  Query Examples --------------  The following are sample configurations and queries for Oracle DB and PostgreSQL:  .. container:: tabs     .. group-tab:: Oracle DB     1. Configuration:     ..        ::           jdbc_app_configs = """{"oracledb": {"url": "jdbc:oracle:thin:@localhost:1521/oracledatabase", "jarpath": "/home/ubuntu/jdbc-jars/ojdbc8.jar", "classpath": "oracle.jdbc.OracleDriver"}}"""     2. Sample Query:     ..        -  Select **oracledb** from the **Select JDBC Connection**          dropdown menu.

However, the **NEW_ID**       column can be used as the ID Column if the query is for larger       data. 3. Click the **Click to Make Query** button to execute the query. ..     .. container:: group-tab        PostgreSQL        1. Configuration:        ..           ::              jdbc_app_configs = """{"postgres": {"url": "jdbc:postgresql://localhost:5432/postgresdatabase", "jarpath": "/home/ubuntu/postgres-artifacts/postgres/Driver.jar", "classpath": "org.postgresql.Driver"}}"""        2. Sample Query:        ..           -  Select **postgres** from the **Select JDBC Connection**             dropdown menu. -  **JDBC Username**:postgres_user-  **JDBC Password**:pguserpassword-  **ID Column Name**:id``
    -   QUERY:
  3. Click the CLICK TO MAKE QUERY button to execute the query. ADDING AN UNTESTED JDBC DRIVER

We encourage you to try out JDBC drivers that are not tested in house. Docker Image Installs
1. Download the JDBC jar for your database. 2.

3. Start the Driverless AI Docker image using the JDBC-specific
    environment variables.

SCORERS

Classification or Regression
GINI (Gini Coefficient)
The Gini index is a well-established method to quantify the inequality
among values of a frequency distribution, and can be used to measure the
quality of a binary classifier. A Gini index of zero expresses perfect
equality (or a totally useless classifier), while a Gini index of one
expresses maximal inequality (or a perfect classifier). The Gini index is based on the Lorenz curve. The Lorenz curve plots the
true positive rate (y-axis) as a function of percentiles of the
population (x-axis). The Lorenz curve represents a collective of models represented by the
classifier. The location on the curve is given by the probability
threshold of a particular model. (i.e., Lower probability thresholds for
classification typically lead to more true positives, but also to more
false positives.)

[]

Regression
R2 (R Squared)
The R2 value represents the degree that the predicted value and the
actual value move in unison. The R2 value varies between 0 and 1 where 0
represents no correlation between the predicted and actual value and 1
represents complete correlation. Calculating the R2 value for linear models is mathematically equivalent
to 1 − SSE/SST (or 1 − residual sum of squares/total sum of squares). For all other models, this equivalence does not hold, so the 1 − SSE/SST
formula cannot be used. In some cases, this formula can produce negative
R2 values, which is mathematically impossible for a real number. Because
Driverless AI does not necessarily use linear models, the R2 value is
calculated using the squared Pearson correlation coefficient.

MSE takes the distances from the points to the regression
line (these distances are the “errors”) and squaring them to remove any
negative signs. MSE incorporates both the variance and the bias of the
predictor. MSE also gives more weight to larger differences. The bigger the error,
the more it is penalized. For example, if your correct answers are 2,3,4
and the algorithm guesses 1,4,3, then the absolute error on each one is
exactly 1, so squared error is also 1, and the MSE is 1. But if the
algorithm guesses 2,3,6, then the errors are 0,0,2, the squared errors
are 0,0,4, and the MSE is a higher 1.333. The smaller the MSE, the
better the model's performance. (TIP: MSE is sensitive to outliers. If
you want a more robust metric, try mean absolute error (MAE).) MSE equation:
$$MSE = \frac{1}{N} \sum_{i=1}^{N}(y_i -\hat{y}_i)^2$$
RMSE (Root Mean Squared Error)
The RMSE metric evaluates how well a model can predict a continuous
value.

The smaller the RMSE, the better the model's performance. (TIP: RMSE is
sensitive to outliers. If you want a more robust metric, try mean
absolute error (MAE).) RMSE equation:
$$RMSE = \sqrt{\frac{1}{N} \sum_{i=1}^{N}(y_i -\hat{y}_i)^2 }$$
Where:
-   _N_ is the total number of rows (observations) of your corresponding
    dataframe. -   _y_ is the actual target value. -   ŷ is the predicted target value. RMSLE (Root Mean Squared Logarithmic Error)
This metric measures the ratio between actual values and predicted
values and takes the log of the predictions and actual values. Use this
instead of RMSE if an under-prediction is worse than an over-prediction. You can also use this when you don't want to penalize large differences
when both of the values are large numbers.

-   _y_ is the actual target value. -   ŷ is the predicted target value. RMSPE (Root Mean Square Percentage Error)
This metric is the RMSE expressed as a percentage. The smaller the
RMSPE, the better the model performance. RMSPE equation:
$$RMSPE = \sqrt{\frac{1}{N} \sum_{i=1}^{N} \frac{(y_i -\hat{y}_i)^2 }{(y_i)^2}}$$
MAE (Mean Absolute Error)
The mean absolute error is an average of the absolute errors. The MAE
units are the same as the predicted target, which is useful for
understanding whether the size of the error is of concern or not. The
smaller the MAE the better the model's performance. (TIP: MAE is robust
to outliers. If you want a metric that is sensitive to outliers, try
root mean squared error (RMSE).) MAE equation:
$$MAE = \frac{1}{N} \sum_{i=1}^{N} | x_i - x |$$
Where:
-   _N_ is the total number of errors
-   |x_(i) − x| equals the absolute errors. MAPE (Mean Absolute Percentage Error)
MAPE measures the size of the error in percentage terms.

MAPE equation:
$$MAPE = \big(\frac{1}{N} \sum \frac {|Actual - Forecast |}{|Actual|} \big) * 100$$
Because the MAPE measure is in percentage terms, it gives an indication
of how large the error is across different scales. Consider the
following example:
  --------------------------------------------------------------------
  Actual     Predicted    Absolute Error   Absolute Percentage Error
  ---------- ------------ ---------------- ---------------------------
  5          1            4                80%
  15,000     15,004       4                0.03%
Both records have an absolute error of 4, but this error could be
considered "small" or "big" when you compare it to the actual value. SMAPE (Symmetric Mean Absolute Percentage Error)
Unlike the MAPE, which divides the absolute errors by the absolute
actual values, the SMAPE divides by the mean of the absolute actual and
the absolute predicted values.

Actual values near 0 cause the MAPE value to become
infinitely high. Because SMAPE includes both the actual and the
predicted values, the SMAPE value can never be greater than 200%. Consider the following example:
  -----------------------
  Actual     Predicted
  ---------- ------------
  0.01       0.05
  0.03       0.04
The MAPE for this data is 216.67% but the SMAPE is only 80.95%. Both records have an absolute error of 4, but this error could be
considered "small" or "big" when you compare it to the actual value. MER (Median Error Rate or Median Absolute Percentage Error)
MER measures the median size of the error in percentage terms. It is
calculated as the median of the unsigned percentage error.

Classification
MCC (Matthews Correlation Coefficient)
The goal of the MCC metric is to represent the confusion matrix of a
model as a single number. The MCC metric combines the true positives,
false positives, true negatives, and false negatives using the equation
described below. A Driverless AI model will return probabilities, not predicted classes. To convert probabilities to predicted classes, a threshold needs to be
defined. Driverless AI iterates over possible thresholds to calculate a
confusion matrix for each threshold. It does this to find the maximum
MCC value. Driverless AI's goal is to continue increasing this maximum
MCC. Unlike metrics like Accuracy, MCC is a good scorer to use when the
target variable is imbalanced. In the case of imbalanced data, high
Accuracy can be found by predicting the majority class.

MCC, on the other hand, takes the
proportion of each class into account. The MCC value ranges from -1 to 1
where -1 indicates a classifier that predicts the opposite class from
the actual value, 0 means the classifier does no better than random
guessing, and 1 indicates a perfect classifier. MCC equation:
$$MCC = \frac{TP \; x \; TN \; - FP \; x \; FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}$$
F05, F1, and F2
A Driverless AI model returns probabilities, not predicted classes. To
convert probabilities to predicted classes, a threshold needs to be
confusion matrix for each threshold. It does this to find the maximum F
metric value. Driverless AI's goal is to continue increasing this
maximum F metric. The F1 score provides a measure for how well a binary classifier can
classify positive cases (given a threshold value). The F1 score is
calculated from the harmonic mean of the precision and recall.

If either precision or recall are very low it
will be reflected with a F1 score closer to 0. F1 equation:
$$F1 = 2 \;\Big(\; \frac{(precision) \; (recall)}{precision + recall}\; \Big)$$
Where:
-   _precision_ is the positive observations (true positives) the model
    correctly identified from all the observations it labeled as
    positive (the true positives + the false positives). -   _recall_ is the positive observations (true positives) the model
    correctly identified from all the actual positive cases (the true
    positives + the false negatives). The F2 score is the weighted harmonic mean of the precision and recall
(given a threshold value). Unlike the F1 score, which gives equal weight
to precision and recall, the F2 score gives more weight to recall than
to precision. More weight should be given to recall for cases where
False Negatives are considered worse than False Positives.

In this case, you
want your predictions to capture all of the customers that will churn. Some of these customers may not be at risk for churning, but the extra
attention they receive is not harmful. More importantly, no customers
actually at risk of churning have been missed. F2 equation:
$$F2 = 5 \;\Big(\; \frac{(precision) \; (recall)}{((4)\;(precision)) + recall}\; \Big)$$
Where:
    positives + the false negatives). Accuracy
In binary classification, Accuracy is the number of correct predictions
made as a ratio of all predictions made.

A Driverless AI model will return probabilities, not predicted classes. To convert probabilities to predicted classes, a threshold needs to be
Accuracy value. Driverless AI's goal is to continue increasing this
maximum Accuracy. Accuracy equation:
$$Accuracy = \Big(\; \frac{\text{number correctly predicted}}{\text{number of observations}}\; \Big)$$
Logloss
The logarithmic loss metric can be used to evaluate the performance of a
binomial or multinomial classifier. Unlike AUC which looks at how well a
model can classify a binary target, logloss evaluates how close a
model's predicted values (uncalibrated probability estimates) are to the
actual target value.

Logloss can be any value greater than or equal to 0,
with 0 meaning that the model correctly assigns a probability of 0% or
100%. Binary classification equation:
$$Logloss = - \;\frac{1}{N} \sum_{i=1}^{N}w_i(\;y_i \ln(p_i)+(1-y_i)\ln(1-p_i)\;)$$
Multiclass classification equation:
$$Logloss = - \;\frac{1}{N} \sum_{i=1}^{N}\sum_{j=1}^{C}w_i(\;y_i,_j \; \ln(p_i,_j)\;)$$
Where:
    dataframe. -   _w_ is the per row user-defined weight (defaults is 1). -   _C_ is the total number of classes (C=2 for binary classification). -   _p_ is the predicted value (uncalibrated probability) assigned to a
    given row (observation). -   _y_ is the actual target value. AUC (Area Under the Receiver Operating Characteristic Curve)
This model metric is used to evaluate how well a binary classification
model is able to distinguish between true positives and false positives.

Use MACROAUC if you prefer the macro average. An AUC of 1 indicates a perfect classifier, while an AUC of .5 indicates
a poor classifier whose performance is no better than random guessing. AUCPR (Area Under the Precision-Recall Curve)
model is able to distinguish between precision recall pairs or points. These values are obtained using different thresholds on a probabilistic
or other continuous-output classifier. AUCPR is an average of the
precision-recall weighted by the probability of a given threshold. The main difference between AUC and AUCPR is that AUC calculates the
area under the ROC curve and AUCPR calculates the area under the
Precision Recall curve. The Precision Recall curve does not care about
True Negatives. For imbalanced data, a large quantity of True Negatives
usually overshadows the effects of changes in other metrics like False
Positives.

As such, AUCPR is
recommended over AUC for highly imbalanced data. MACROAUC (Macro Average of Areas Under the Receiver Operating Characteristic Curves)
For multiclass classification problems, this score is computed by
macro-averaging the ROC curves for each class (one per class). The area
under the curve is a constant. A MACROAUC of 1 indicates a perfect
classifier, while a MACROAUC of .5 indicates a poor classifier whose
performance is no better than random guessing. This option is not
available for binary classification problems. Scorer Best Practices - Regression
When deciding which scorer to use in a regression problem, consider the
following:
-   Do you want your scorer to be sensitive to outliers? -   What unit should the scorer be in? SENSITIVE TO OUTLIERS
Certain scorers are more sensitive to outliers. When a scorer is
sensitive to outliers, it means that it is important that the model
predictions are never exceedingly inaccurate.

RMSEdrops down significantly. **Performance Units**  Different scorers show the performance of the Driverless AI experiment in different units. This section continues with the previous example where the target is to predict the number of days until an event.

-  Is your data imbalanced? **Scorer Evaluates Probabilities or Classes**  The final output of a Driverless AI model is a predicted probability that a record is in a particular class. The scorer you choose either evaluates how accurate the probability is or how accurate the assigned class is from that probability. Choosing this depends on the use of the Driverless AI model. Do you want to use the probabilities, or do you want to convert those probabilities into classes? For example, if you are predicting whether a customer will churn, you may take the predicted probabilities and turn them into distinct classes—customers who will churn vs customers who will not churn. If you are predicting the expected loss of revenue, use the predicted probabilities instead (predicted probability of churn \* value of customer). If your use case requires a class assigned to each record, select a scorer that evaluates the model's performance based on how well it classifies the records.

**Robust to Imbalanced Data**  For certain use cases, positive classes may be very rare. In these instances, some scorers can be misleading. For example, if you have a use case where 99% of the records haveClass
= No, then a model that always predictsNo`` will have 99% accuracy. For these use cases, it is best to select a metric that does not include
True Negatives or considers relative size of the True Negatives like
AUCPR or MCC.

TIME_SERIES_RECIPE``

TIME-SERIES LAG-BASED RECIPE
This recipe specifies whether to include Time Series lag features when
training a model with a provided (or autodetected) time column. This is
enabled by default. Lag features are the primary automatically generated
time series features and represent a variable's past values. At a given
sample with time stamp t, features at some time difference T (lag) in
the past are considered. For example, if the sales today are 300, and
sales of yesterday are 250, then the lag of one day for sales is 250. Lags can be created on any feature as well as on the target. Lagging
variables are important in time series because knowing what happened in
different time periods in the past can greatly facilitate predictions
for the future. NOTE: Ensembling is disabled when the lag-based recipe
with time columns is activated because it only supports a single final
model.

time_series_leaderboard_mode--------------------------------  .. container:: dropdown     **Control the automatic time-series leaderboard mode**     Select from the following options:        -  'diverse': explore a diverse set of models built using various          expert settings. Note that it's possible to rerun another such          diverse leaderboard on top of the best-performing model(s),          which will effectively help you compose these expert settings.       -  'sliding_window': If the forecast horizon is N periods, create          a separate model for "each of the (gap, horizon) pairs of          (0,n), (n,n), (2*n,n), ..., (2*N-1, n) in units of time          periods. The number of periods to predict per model n is          controlled by the expert settingtime_series_leaderboard_periods_per_model``, which defaults

    to 1. This can help to improve short-term forecasting quality.

time_series_leaderboard_periods_per_model---------------------------------------------  .. container:: dropdown     **Number of periods per model if time_series_leaderboard_mode is    'sliding_window'**     Specify the number of periods per model iftime_series_leaderboard_modeis set tosliding_window``. Larger

    values lead to fewer models.

TIME_SERIES_MERGE_SPLITS``

LARGER VALIDATION SPLITS FOR LAG-BASED RECIPE

Specify whether to create larger validation splits that are not bound to
the length of the forecast horizon. This can help to prevent overfitting
on small data or short forecast horizons. This is enabled by default.

MERGE_SPLITS_MAX_VALID_RATIO``

MAXIMUM RATIO OF TRAINING DATA SAMPLES USED FOR VALIDATION

Specify the maximum ratio of training data samples used for validation
across splits when larger validation splits are created (see
time_series_merge_splits setting). The default value (-1) will set the
ratio automatically depending on the total amount of validation splits.

FIXED_SIZE_SPLITS``

FIXED-SIZE TRAIN TIMESPAN ACROSS SPLITS

Specify whether to keep a fixed-size train timespan across time-based
splits during internal validation. That leads to roughly the same amount
of train samples in every split. This is disabled by default.

TIME_SERIES_VALIDATION_FOLD_SPLIT_DATETIME_BOUNDARIES``

CUSTOM VALIDATION SPLITS FOR TIME-SERIES EXPERIMENTS

Specify date or datetime timestamps (in the same format as the time
column) to use for custom training and validation splits.

TIMESERIES_SPLIT_SUGGESTION_TIMEOUT``

TIMEOUT IN SECONDS FOR TIME-SERIES PROPERTIES DETECTION IN UI

Specify the timeout in seconds for time-series properties detection in
Driverless AI's user interface. This value defaults to 30.

HOLIDAY_FEATURES``

GENERATE HOLIDAY FEATURES

For time-series experiments, specify whether to generate holiday
features for the experiment. This is enabled by default.

HOLIDAY_COUNTRIES``

COUNTRY CODE(S) FOR HOLIDAY FEATURES

Specify country codes in the form of a list that is used to look up
holidays.

NOTE: This setting is for migration purposes only.

OVERRIDE_LAG_SIZES``

TIME-SERIES LAGS OVERRIDE

Specify the override lags to be used. The lag values provided here are
the only set of lags to be explored in the experiment. The following
examples show the variety of different methods that can be used to
specify override lags:

-   "[0]" disable lags
-   "[7, 14, 21]" specifies this exact list
-   "21" specifies every value from 1 to 21
-   "21:3" specifies every value from 1 to 21 in steps of 3
-   "5-21" specifies every value from 5 to 21
-   "5-21:3" specifies every value from 5 to 21 in steps of 3

OVERRIDE_UFAPT_LAG_SIZES``

LAGS OVERRIDE FOR FEATURES THAT ARE NOT KNOWN AHEAD OF TIME

Specify lags override for non-target features that are not known ahead
of time.

OVERRIDE_NON_UFAPT_LAG_SIZES``

LAGS OVERRIDE FOR FEATURES THAT ARE KNOWN AHEAD OF TIME

Specify lags override for non-target features that are known ahead of
time.

MIN_LAG_SIZE``

SMALLEST CONSIDERED LAG SIZE

Specify a minimum considered lag size. This value defaults to -1.

ALLOW_TIME_COLUMN_AS_FEATURE``

ENABLE FEATURE ENGINEERING FROM TIME COLUMN

Specify whether to enable feature engineering based on the selected time
column, e.g. Date~weekday. This is enabled by default.

ALLOW_TIME_COLUMN_AS_NUMERIC_FEATURE``

ALLOW INTEGER TIME COLUMN AS NUMERIC FEATURE

Specify whether to enable feature engineering from an integer time
column. Note that if you are using a time series recipe, using a time
column (numeric time stamps) as an input feature can lead to a model
that memorizes the actual timestamps instead of features that generalize
to the future. This is disabled by default.

datetime_funcs------------------  .. container:: dropdown     **Allowed Date and Date-Time Transformations**     Specify the date or date-time transformations to allow Driverless AI    to use. Choose from the following transformers:     -  year    -  quarter    -  month    -  week    -  weekday    -  day    -  dayofyear    -  num (direct numeric value representing the floating point value of       time, disabled by default)    -  hour    -  minute    -  second     Features in Driverless AI will appear asgetfollowed by the    name of the transformation. Note thatget_num`` can lead to

    overfitting if used on IID problems and is disabled by default.

FILTER_DATETIME_FUNCS``

AUTO FILTERING OF DATE AND DATE-TIME TRANSFORMATIONS

Whether to automatically filter out date and date-time transformations
that would lead to unseen values in the future. This is enabled by
default.

ALLOW_TGC_AS_FEATURES``

CONSIDER TIME GROUPS COLUMNS AS STANDALONE FEATURES

Specify whether to consider time groups columns as standalone features.
This is disabled by default.

ALLOWED_COLTYPES_FOR_TGC_AS_FEATURES``

WHICH TGC FEATURE TYPES TO CONSIDER AS STANDALONE FEATURES

Specify whether to consider time groups columns (TGC) as standalone
features. If "Consider time groups columns as standalone features" is
enabled, then specify which TGC feature types to consider as standalone
features. Available types are NUMERIC, CATEGORICAL, OHE_CATEGORICAL,
DATETIME, DATE, and TEXT. All types are selected by default. Note that
"TIME_COLUMN" is treated separately via the "Enable Feature Engineering
from Time Column" option. Also note that if "Time Series Lag-Based
Recipe" is disabled, then all time group columns are allowed features.

ENABLE_TIME_UNAWARE_TRANSFORMERS``

ENABLE TIME UNAWARE TRANSFORMERS

Specify whether various transformers (clustering, truncated SVD) are
enabled, which otherwise would be disabled for time series experiments
due to the potential to overfit by leaking across time within the fit of
each fold. This is set to AUTO by default.

TGC_ONLY_USE_ALL_GROUPS``

ALWAYS GROUP BY ALL TIME GROUPS COLUMNS FOR CREATING LAG FEATURES

Specify whether to group by all time groups columns for creating lag
features, instead of sampling from them. This is enabled by default.

tgc_allow_target_encoding-----------------------------  .. container:: dropdown     **Allow Target Encoding of Time Groups Columns**     Specify whether it is allowed to target encode the time groups    columns. This is disabled by default.     **Notes**:     -  This setting is not affected byallow_tgc_as_features.    -  Subgroups can be encoded by disablingtgc_only_use_all_groups``.

TIME_SERIES_HOLDOUT_PREDS``

GENERATE TIME-SERIES HOLDOUT PREDICTIONS

Specify whether to create diagnostic holdout predictions on training
data using moving windows. This is enabled by default. This can be
useful for MLI, but it will slow down the experiment considerably when
enabled. Note that the model itself remains unchanged when this setting
is enabled.

TIME_SERIES_VALIDATION_SPLITS``

NUMBER OF TIME-BASED SPLITS FOR INTERNAL MODEL VALIDATION

Specify a fixed number of time-based splits for internal model
validation. Note that the actual number of allowed splits can be less
than the specified value, and that the number of allowed splits is
determined at the time an experiment is run. This value defaults to -1
(auto).

TIME_SERIES_SPLITS_MAX_OVERLAP``

MAXIMUM OVERLAP BETWEEN TWO TIME-BASED SPLITS

Specify the maximum overlap between two time-based splits. The amount of
possible splits increases with higher values. This value defaults to
0.5.

time_series_max_holdout_splits----------------------------------  .. container:: dropdown     **Maximum Number of Splits Used for Creating Final Time-Series    Model's Holdout Predictions**     Specify the maximum number of splits used for creating the final    time-series Model's holdout predictions. The default value (-1) will    use the same number of splits that are used during model validation.    Usetime_series_validation_splits`` to control amount of time-based

    splits used for model validation.

MLI_TS_FAST_APPROX``

WHETHER TO SPEED UP CALCULATION OF TIME-SERIES HOLDOUT PREDICTIONS

Specify whether to speed up time-series holdout predictions for
back-testing on training data. This setting is used for MLI and
calculating metrics. Note that predictions can be slightly less accurate
when this setting is enabled. This is disabled by default.

MLI_TS_FAST_APPROX_CONTRIBS``

WHETHER TO SPEED UP CALCULATION OF SHAPLEY VALUES FOR TIME-SERIES
HOLDOUT PREDICTIONS

Specify whether to speed up Shapley values for time-series holdout
predictions for back-testing on training data. This setting is used for
MLI. Note that predictions can be slightly less accurate when this
setting is enabled. This is enabled by default.

MLI_TS_HOLDOUT_CONTRIBS``

GENERATE SHAPLEY VALUES FOR TIME-SERIES HOLDOUT PREDICTIONS AT THE TIME
OF EXPERIMENT

Specify whether to enable the creation of Shapley values for holdout
predictions on training data using moving windows at the time of the
experiment. This can be useful for MLI, but it can slow down the
experiment when enabled. If this setting is disabled, MLI will generate
Shapley values on demand. This is enabled by default.

TIME_SERIES_MIN_INTERPRETABILITY``

LOWER LIMIT ON INTERPRETABILITY SETTING FOR TIME-SERIES EXPERIMENTS
(IMPLICITLY ENFORCED)

Specify the lower limit on interpretability setting for time-series
experiments. Values of 5 (default) or more can improve generalization by
more aggressively dropping the least important features. To disable this
setting, set this value to 1.

LAGS_DROPOUT``

DROPOUT MODE FOR LAG FEATURES

Specify the dropout mode for lag features in order to achieve an equal
n.a. ratio between train and validation/tests. INDEPENDENT mode performs
a simple feature-wise dropout. DEPENDENT mode takes the lag-size
dependencies per sample/row into account. DEPENDENT is enabled by
default.

PROB_LAG_NON_TARGETS``

PROBABILITY TO CREATE NON-TARGET LAG FEATURES

Lags can be created on any feature as well as on the target. Specify a
probability value for creating non-target lag features. This value
defaults to 0.1.

ROLLING_TEST_METHOD``

METHOD TO CREATE ROLLING TEST SET PREDICTIONS

Specify the method used to create rolling test set predictions. Choose
between test time augmentation (TTA) and a successive refitting of the
final pipeline (REFIT). TTA is enabled by default.

NOTES:

-   This setting only applies to the test set that is provided by the
    user during an experiment.
-   This setting only has an effect if the provided test set spans more
    periods than the forecast horizon and if the target values of the
    test set are known.

FAST_TTA_INTERNAL``

FAST TTA FOR INTERNAL VALIDATION

Specify whether the genetic algorithm applies Test Time Augmentation
(TTA) in one pass instead of using rolling windows for validation splits
longer than the forecast horizon. This is enabled by default.

PROB_DEFAULT_LAGS``

PROBABILITY FOR NEW TIME-SERIES TRANSFORMERS TO USE DEFAULT LAGS

Specify the probability for new lags or the EWMA gene to use default
lags. This is determined independently of the data by frequency, gap,
and horizon. This value defaults to 0.2.

PROB_LAGSINTERACTION``

PROBABILITY OF EXPLORING INTERACTION-BASED LAG TRANSFORMERS

Specify the unnormalized probability of choosing other lag time-series
transformers based on interactions. This value defaults to 0.2.

PROB_LAGSAGGREGATES``

PROBABILITY OF EXPLORING AGGREGATION-BASED LAG TRANSFORMERS

transformers based on aggregations. This value defaults to 0.2.

TS_TARGET_TRAFO``

TIME SERIES CENTERING OR DETRENDING TRANSFORMATION
Specify whether to use centering or detrending transformation for time
series experiments. Select from the following:
-   None (Default)
-   Centering (Fast)
-   Centering (Robust)
-   Linear (Fast)
-   Linear (Robust)
-   Logistic
-   Epidemic (Uses the SEIRD model)
The fitted signal is removed from the target signal per individual time
series once the free parameters of the selected model are fitted. LINEAR
or LOGISTIC will remove the fitted linear or logistic trend, CENTERING
will only remove the mean of the target signal and EPIDEMIC will remove
the signal specified by a Susceptible-Infected-Exposed-Recovered-Dead
(SEIRD) epidemic model. Predictions are made by adding the previously
removed signal once the pipeline is fitted on the residuals. NOTES:
-   MOJO support is currently disabled when this setting is enabled. -   The FAST centering and linear detrending options use least squares
    fitting.

ts_target_trafo_epidemic_params_dict----------------------------------------  .. container:: dropdown     **Custom Bounds for SEIRD Epidemic Model Parameters**     Specify the custom bounds for controlling    `Susceptible-Infected-Exposed-Recovered-Dead <https://arxiv.org/abs/1411.3435>`__    (SEIRD) epidemic model parameters for detrending of the target for    each time series group. The target column must correspond to *I(t)*,    which represents infection cases as a function of time. For each training split and time series group, the SEIRD model is fit    to the target signal by optimizing a set of free parameters for each    time series group. The model's value is then subtracted from the    training response, and the residuals are passed to the feature    engineering and modeling pipeline. For predictions, the SEIRD model's    value is added to the residual predictions from the pipeline for each    time series group.

TS_TARGET_TRAFO_EPIDEMIC_TARGET``

WHICH SEIRD MODEL COMPONENT THE TARGET COLUMN CORRESPONDS TO

Specify a SEIRD model component for the target column to correspond to.
Select from the following:

-   I (Default): Infected
-   R: Recovered
-   D: Deceased

ts_lag_target_trafo-----------------------  .. container:: dropdown     **Time Series Lag-Based Target Transformation**     Specify whether to use either the difference between or ratio of the    current target and a lagged target. Select from **None** (default),    **Difference**, and **Ratio**.     **Notes**:     -  MOJO support is currently disabled when this setting is enabled.    -  The corresponding lag size is specified with thets_target_trafo_lag_size``
expert setting.

ts_target_trafo_lag_size----------------------------  .. container:: dropdown     **Lag Size Used for Time Series Target Transformation**     Specify the lag size used for time series target transformation.    Specify this setting when using thets_lag_target_trafo`` setting.

    This value defaults to -1.

    NOTE: The lag size should not be smaller than the sum of forecast
    horizon and gap.

H2O AI FEATURE STORE SETUP

You can use the H2O AI Feature Store to store, update, and share the
features data scientists, developers, and engineers need to build AI
models. This page describes how to configure Driverless AI to work with
the H2O AI Feature Store. NOTE: For more information on the H2O AI Feature Store, refer to the
official documentation. Description of relevant configuration attributes
The following are descriptions of the relevant configuration attributes
when enabling the H2O AI Feature Store data connector:
-   enabled_file_systems: A list of file systems you want to enable. To
    enable the Feature Store data connector, feature_store must be added
    to this list of data sources. -   feature_store_endpoint_url: A URL that points to the Feature Store
    server. -   feature_store_enable_tls: To enable TLS communication between DAI
    and the Feature Store server, set this to true.

included_transformers-------------------------  .. container:: dropdown     **Include Specific Transformers**     Select the :ref:`transformer(s) <Transformations>` that you want to    use in the experiment. Use the **Check All**/**Uncheck All** button    to quickly add or remove all transfomers at once. **Note**: If you    uncheck all transformers so that none is selected, Driverless AI will    ignore this and will use the default list of transformers for that    experiment. This list of transformers will vary for each experiment.     The equivalent config.toml parameter isincluded_transformers``.

INCLUDED_MODELS``

INCLUDE SPECIFIC MODELS
Specify the types of models that you want Driverless AI to build in the
experiment. This list includes natively supported algorithms and models
added with custom recipes. NOTE: The ImbalancedLightGBM and ImbalancedXGBoostGBM models are closely
tied with the sampling_method_for_imbalanced option. Specifically:
  -   If the ImbalancedLightGBM and/or ImbalancedXGBoostGBM models are
      ENABLED and the sampling_method_for_imbalanced is ENABLED (set to
      a value other than OFF), then Driverless AI will check your target
      imbalance fraction. If the target fraction proves to be above the
      allowed imbalance threshold, then sampling will be triggered. -   If the ImbalancedLightGBM and/or ImbalancedXGBoostGBM models are
      DISABLED and the sampling_method_for_imbalanced option is ENABLED,
      then no special sampling technique will be performed.

INCLUDED_SCORERS``

INCLUDE SPECIFIC SCORERS

Specify the scorer(s) that you want Driverless AI to include when
running the experiment.

included_pretransformers----------------------------  .. container:: dropdown     **Include Specific Preprocessing Transformers**     Specify which :ref:`transformers <Transformations>` to use for    preprocessing before other transformers are activated. Preprocessing    transformers can take any original features and output arbitrary    features that are used by the normal layer of transformers. **Notes**:     -  Preprocessing transformers and all other layers of transformers       are part of the Python and (if applicable) MOJO scoring packages. -  Any :ref:`custom transformer recipe <custom-recipes>` or native       DAI transformer can be used as a preprocessing transformer. For       example, a preprocessing transformer can perform interactions,       string concatenations, or date extractions as a preprocessing step       before the next layer of Date and DateTime transformations are       performed.

num_pipeline_layers-----------------------  .. container:: dropdown     **Number of Pipeline Layers**     Specify the number of pipeline layers. This value defaults to 1. The    equivalent config.toml parameter isnum_pipeline_layers``.

  NOTE: This does not include the preprocessing layer specified by the
  included_pretransformers expert setting.

included_datas------------------  .. container:: dropdown     **Include Specific Data Recipes During Experiment**     Specify whether to include specific data recipes during the    experiment. Avoids need for separate data preparation step, builds    data preparation within experiment and within python scoring package.    But Mojo will require data preparation applied before making    predictions.     The equivalent config.toml parameter isincluded_datas``.

included_individuals------------------------  .. container:: dropdown     **Include Specific Individuals**     In Driverless AI, every completed experiment automatically generates    Python code for the experiment that corresponds to the individual(s)    used to build the final model. You can edit this auto-generated    Python code offline and upload it as a recipe, or edit and save it    using the built-in    :ref:`custom recipe management editor <custom-recipes>`. This feature    gives you code-first access to a significant portion of DAI's    internal transformer and model generation process. This expert setting lets you do one of the following:     -  Leave this field empty to have all individuals be freshly       generated and treated by DAI's AutoML as a container of model and       transformer choices. -  Select recipe display names of custom individuals through the UI.

THRESHOLD_SCORER``

SCORER TO OPTIMIZE THRESHOLD TO BE USED IN OTHER CONFUSION-MATRIX BASED
SCORERS (FOR BINARY CLASSIFICATION)

Specify the scorer used to optimize the binary probability threshold
that is being used in related Confusion Matrix based scorers such as
Precision, Recall, FalsePositiveRate, FalseDiscoveryRate,
FalseOmissionRate, TrueNegativeRate, FalseNegativeRate, and
NegativePredictiveValue. Select from the following:

-   AUTO (Default): Use this option to sync the threshold scorer with
    the scorer used for the experiment. If this is not possible, F1 is
    used.
-   F05 More weight on precision, less weight on recall.
-   F1: Equal weight on precision and recall.
-   F2: Less weight on precision, more weight on recall.
-   MCC: Use this option when all classes are equally important.

PROB_ADD_GENES``

PROBABILITY TO ADD TRANSFORMERS

Specify the unnormalized probability to add genes or instances of
transformers with specific attributes. If no genes can be added, other
mutations are attempted. This value defaults to 0.5.

PROB_ADDBEST_GENES``

PROBABILITY TO ADD BEST SHARED TRANSFORMERS

transformers with specific attributes that have shown to be beneficial
to other individuals within the population. This value defaults to 0.5.

PROB_PRUNE_GENES``

PROBABILITY TO PRUNE TRANSFORMERS

Specify the unnormalized probability to prune genes or instances of
transformers with specific attributes. This value defaults to 0.5.

PROB_PERTURB_XGB``

PROBABILITY TO MUTATE MODEL PARAMETERS

Specify the unnormalized probability to change model hyper parameters.
This value defaults to 0.25.

PROB_PRUNE_BY_FEATURES``

PROBABILITY TO PRUNE WEAK FEATURES

Specify the unnormalized probability to prune features that have low
variable importance instead of pruning entire instances of
genes/transformers. This value defaults to 0.25.

SKIP_TRANSFORMER_FAILURES``

WHETHER TO SKIP FAILURES OF TRANSFORMERS

Specify whether to avoid failed transformers. This is enabled by
default.

SKIP_MODEL_FAILURES``

WHETHER TO SKIP FAILURES OF MODELS

Specify whether to avoid failed models. Failures are logged according to
the specified level for logging skipped failures. This is enabled by
default.

DETAILED_SKIP_FAILURE_MESSAGES_LEVEL``

LEVEL TO LOG FOR SKIPPED FAILURES

Specify one of the following levels for the verbosity of log failure
messages for skipped transformers or models:

-   0 = Log simple message
-   1 = Log code line plus message (Default)
-   2 = Log detailed stack traces

notify_failures-------------------  .. container:: dropdown     **Whether to Notify About Failures of Transformers or Models or Other    Recipe Failures**     Specify whether to display notifications in the GUI about recipe    failures. This is enabled by default.     The equivalent config.toml parameter isnotify_failures``.

ACCEPTANCE_TEST_TIMEOUT``

TIMEOUT IN MINUTES FOR TESTING ACCEPTANCE OF EACH RECIPE

Specify the number of minutes to wait until a recipe's acceptance
testing is aborted. A recipe is rejected if acceptance testing is
enabled and it times out. This value defaults to 20.0.

CUSTOM INDIVIDUAL RECIPE

The following sections describe Driverless AI's Individual Recipe
feature. -   individual-recipe-understanding
-   individual-recipe-getting
-   individual-recipe-using
-   individual-recipe-including
-   individual-recipe-example

Understanding the Individual Recipe
In Driverless AI, every completed experiment automatically generates
Python code for the experiment that corresponds to the individual(s)
used to build the final model. You can edit this auto-generated Python
code offline and upload it as a recipe, or edit and save it using the
built-in custom recipe management editor <custom-recipes>. This feature
gives you code-first access to a significant portion of DAI's internal
transformer and model generation process. The Individual Recipe contains information about model type, model
hyperparameters, data science types for input features, transformers
used, and transformer parameters.

The
Individual Recipe can be used in DAI's expert-settings with the
Include specific individuals <included_individuals> parameter. This feature is supported for experiments made using DAI 1.7.2 and
later. Using custom individuals
A custom individual can be run as is, evolved alongside other models or
individuals, or frozen to be included as is during the final evolution
stage alongside other models from the experiment. -   AS IS: To ensemble the custom individuals as they are, set
    enable_genetic_algorithm <enable_genetic_algorithm> to off. Note
    that to get reproducible results, set reproducibility to on and make
    sure that the same accuracy knob settings are selected (as accuracy
    settings affects the internal cross validation fold data
    assignment).

-   FROZEN INDIVIDUALS: By default, a custom individual behaves like a
    standard internal DAI individual, which has its features and model
    hyperparameters mutated during evolution. To disable the mutation of
    features and model hyperparameters and "freeze" a custom individual
    during the DAI genetic evolution process, set self.params values (in
    the code):
        self.params = {'prob_perturb_xgb': 0,
                      'prob_add_genes': 0,
                      'prob_prune_genes': 0,
                      'prob_prune_by_features': 0,
                      'prob_addbest_genes': 0,
                      'prob_prune_by_features': 0}
    If all individuals in an experiment are FROZEN, then _no tuning or
    evolution is performed_ for them. You can specify the number of such
    individuals to be included in an ensemble along with any other, by
    modifying the
    Ensemble Level for Final Modeling Pipeline <fixed_ensemble_level>
    expert setting.

The following sections
describe how to get the Individual Recipe code for a completed
experiment. -   FROM A COMPLETED EXPERIMENT: From a completed experiment page, click
    TUNE EXPERIMENT > CREATE INDIVIDUAL RECIPE, then select UPLOAD AS
    CUSTOM RECIPE. When this option is selected, the Individual Recipe
    becomes available on the RECIPES page and in the Expert Settings
    under the INCLUDE SPECIFIC INDIVIDUALS setting. You can also select
    DOWNLOAD to download the Individual Recipe Python file directly to
    your local file system. You can then add the downloaded Individual
    Recipe to DAI by clicking RECIPES in the main navigation, then
    clicking ADD CUSTOM RECIPES > FROM COMPUTER. -   FROM THE EXPERIMENTS LISTING PAGE: Click the drop-down button next
    to the experiment you want to create the Individual Recipe for, and
    then select CREATE INDIVIDUAL RECIPE.

To download the summary file, click the DOWNLOAD SUMMARY & LOGS
    button of any completed experiment. The individual recipe filename
    is final_indiv0.py. Using the Individual Recipe
This section describes how you can use the Individual Recipe to view
detailed information about how the final model was built and make
fine-tuned adjustments to the model by editing the auto-generated Python
code and using the edited Individual Recipe in a new experiment. -   individual-recipe-transparency
-   individual-recipe-model-control
-   individual-recipe-feature-control
Model Transparency
The following functions in the Individual Recipe provide significant
transparency for the final model:
-   The set_model function lets you view various details about the final
    model such as model type and the model's parameters. -   The set_genes function lets you view each feature that is in the
    model and information about how each feature was transformed.

You can make minor modifications to these
parameters by editing the self.model_params dictionary. This can be
helpful if you want to see whether minor changes to the parameters
result in more robust or accurate models or if you are required to
change the model parameters for business or regulatory purposes. Feature Control
Each feature used in the model is listed in the set_genes function,
beginning with features that were not engineered and followed by
engineered features. The following examples show original and
transformed features as they appear in the auto-generated Python code. Original features
The following example provides details on an original feature called
Humidity3pm. Note
Original features are labeled with the value OriginalTransformer in the
add_transformer() field.

The following sections describe how to perform these actions
using the Individual Recipe. Adding features
During the experiment, Driverless AI uses a Genetic Algorithm <ga> to
determine which features to drop from the model. However, your use case
may require you to force a column to be used by the model. The following
steps describe how to force in a numeric column that was dropped by
Driverless AI:
1. Copy an OriginalTransformer feature that is already in the code and
    paste it below. 2. Specify the column you want to force in with the num_cols field. In
    the example below, Driverless AI dropped YearsSinceLastPromotion, so
    an OriginalTransformer example that was already present was copied
    and the value for num_cols was edited. 3. To ensure that the model uses the feature, set forced=True. 4. Change the gene_index to a value that is not used .

However, you may still want to use a column when Driverless AI
engineers the column in a manner that is not ideal for your particular
use case. In this scenario, you can delete the unwanted feature from the
Individual Recipe code. Modifying features
Driverless AI automatically creates engineered features that have a list
of editable parameters that are specific to the transformer. Because
these are internal parameters, contacting support@h2o.ai is recommended
when modifying these parameters. The following are two common use cases for modifying specific features
in the Individual Recipe code:
-   FORCING FEATURES INTO A MODEL: To force in a specific feature and
    ensure that it is not pruned, set forced=True. -   ENFORCING MONOTONICITY: To enforce monotonicity for a specific
    feature, set mono=True.

1. On the CUSTOM RECIPES page, click the Individual Recipe you want to
    edit. 2. Use the built-in recipe editor to make changes to the Individual
    Recipe. 3. Click SAVE AS NEW RECIPE AND ACTIVATE. 4. Click MORE ACTIONS > USE IN NEW EXPERIMENT. Including specific individuals in an experiment
The downloaded individual recipe (zip or Python file) can be directly
uploaded from the computer via the expert settings when creating a new
experiment. You can also perform the following steps to include an Individual Recipe
that has already been uploaded by using the
Include specific individuals <included_individuals> expert setting. 1. On the Experiment Setup page, click EXPERT SETTINGS. The Expert
    Settings window is displayed. 2. Click the RECIPES tab, then click SELECT VALUES for the INCLUDE
    SPECIFIC INDIVIDUALS expert setting. 3. Select the custom individuals you want to include in the experiment,
    then click DONE. 4. In the Expert Settings window, click SAVE.

Individual Recipe Example
This section contains a list of minimum required parameters for a custom
Individual Recipe, as well as an example of a custom Individual Recipe
using the Credit Card dataset. Note
For more Individual Recipe examples, see the Driverless AI recipes
GitHub repository. Minimum required parameters
The following is a list of the minimum required parameters for a custom
Individual Recipe:
-   MODEL TYPE: Specify the model type. For example:
-   MODEL PARAMETERS: Specify the parameters of the model. For example:
-   GENOME: Specify all valid parameters for genes. For example:
Sample Individual Recipe
The following is an example of a custom Individual Recipe using the
Credit Card dataset. Note
The following example does not contain all available parameters for
custom Individual Recipes. For an example Individual Recipe that
features all available parameters, see creditcard.py from the official
Driverless AI recipes GitHub repository.

DATASET OPTIONS

The following is a list of options that are available for every dataset
on the DATASETS page. To view these options, click CLICK FOR ACTIONS
next to any dataset listed on the Datasets page. -   DETAILS: View detailed information about the dataset. For more
    information, see view_dataset. -   VISUALIZE: View a variety of visualizations generated by Driverless
    AI using the dataset. For more information, see visualize_dataset. -   DATA PREP:
      -   SPLIT: Split the dataset into two subsets. For more
          information, see split_dataset. -   JOIN WIZARD: Opens the Driverless AI dataset join wizard. For
          more information, see join_dataset_wizard. -   TRANSFORM DATASET: Opens a page that lets you edit specific
          values in a dataset. To confirm your changes, click the SAVE
          button. To undo your most recent change, click the UNDO
          button. You can also reset all your changes by clicking the
          RESET button.

The F0.5 score is the weighted harmonic mean of the precision and recall
to precision and recall, the F0.5 score gives more weight to precision
than to recall. More weight should be given to precision for cases where
False Positives are considered worse than False Negatives. For example,
if your use case is to predict which products you will run out of, you
may consider False Positives worse than False Negatives. In this case,
you want your predictions to be very precise and only capture the
products that will definitely run out. If you predict a product will
need to be restocked when it actually doesn't, you incur cost by having
purchased more inventory than you actually need.

INSTALL AND RUN IN A DOCKER CONTAINER ON GOOGLE COMPUTE ENGINE

This section describes how to install and start Driverless AI FROM
SCRATCH using a Docker container in a Google Compute environment. This installation assumes that you already have a Google Cloud Platform
account. If you don't have an account, go to
https://console.cloud.google.com/getting-started to create one. In
addition, refer to Google's Machine Types documentation for information
on Google Compute machine types. WATCH THE INSTALLATION VIDEO here. Note that some of the images in this
video may change between releases, but the installation steps remain the
same. Before You Begin
If you are trying GCP for the first time and have just created an
account, check your Google Compute Engine (GCE) resource quota limits. By default, GCP allocates a maximum of 8 CPUs and no GPUs. You can
change these settings to match your quota limit, or you can request more
resources from GCP.

Installation Procedure
1. In your browser, log in to the Google Compute Engine Console at
    https://console.cloud.google.com/. 2. In the left navigation panel, select COMPUTE ENGINE > VM INSTANCES. 3. Click CREATE INSTANCE. 4. Specify the following at a minimum:
5. Create a Firewall rule for Driverless AI. On the Google Cloud
    Platform left navigation panel, select VPC NETWORK > FIREWALL RULES. Specify the following settings:
6. On the VM Instances page, SSH to the new VM Instance by selecting
    OPEN IN BROWSER WINDOW from the SSH dropdown. 7. H2O provides a script for you to run in your VM instance. Open an
    editor in the VM instance (for example, vi). Copy one of the scripts
    below (depending on whether you are running GPUs or CPUs). Save the
    script as INSTALL.SH. 8. Type the following commands to run the install script. 9. In your user folder, create the following directories as your user. 10. Add your Google Compute user name to the Docker container. 11.

12. Retrieve the Driverless AI Docker image from
    https://www.h2o.ai/download/. 13. Load the Driverless AI Docker image. The following example shows how
    to load Driverless AI. Replace VERSION with your image. 14. If you are running CPUs, you can skip this step. Otherwise, you must
    enable persistence of the GPU. Note that this needs to be run once
    every reboot. Refer to the following for more information:
    http://docs.nvidia.com/deploy/driver-persistence/index.html. 15. Start the Driverless AI Docker image and replace TAG below with the
    image tag. Depending on your install version, use the
    docker run --runtime=nvidia (>= Docker 19.03) or nvidia-docker (<
    Docker 19.03) command. Refer to Data Connectors for information on
    how to add the GCS and GBQ data connectors to your Driverless AI
    instance. 16.

You can stop the instance using one of the following
methods:
STOPPING IN THE BROWSER
1. On the VM Instances page, click on the VM instance that you want to
    stop. 2. Click STOP at the top of the page. 3. A confirmation page will display. Click STOP to stop the instance. STOPPING IN TERMINAL
SSH into the machine that is running Driverless AI, and then run the
    h2oai stop

Upgrading Driverless AI
This section provides instructions for upgrading Driverless AI versions
that were installed in a Docker container. These steps ensure that
existing experiments are saved. WARNING: Experiments, MLIs, and MOJOs reside in the Driverless AI tmp
directory and are not automatically upgraded when Driverless AI is
upgraded. -   Build MLI models before upgrading. -   Build MOJO pipelines before upgrading. -   Stop Driverless AI and make a backup of your Driverless AI tmp
      directory before upgrading.

Before upgrading, be sure to run MLI jobs on models that you want to
  continue to interpret in future releases. If that MLI job appears in
  the list of Interpreted Models in your current version, then it will
  be retained after upgrading. If you did not build a MOJO pipeline on a model before upgrading
  Driverless AI, then you will not be able to build a MOJO pipeline on
  that model after upgrading. Before upgrading, be sure to build MOJO
  pipelines on all desired models and then back up your Driverless AI
  tmp directory. NOTE: Stop Driverless AI if it is still running. Requirements
We recommend to have NVIDIA driver >= installed (GPU only) in your host
environment for a seamless experience on all architectures, including
Ampere. Driverless AI ships with CUDA 11.2.2 for GPUs, but the driver
must exist in the host environment. Go to NVIDIA download driver to get the latest NVIDIA Tesla A/T/V/P/K
series drivers.

DRIVERLESS AI K-LIME MOJO REASON CODE PIPELINE - JAVA RUNTIME

For completed MLI experiments, users can download the k-LIME MOJO. The
k-LIME MOJO Reason Code Pipeline is a reason code engine that can be
deployed in any Java environment to generate reason codes in real time. To obtain Java runtime MOJO for K-LIME reason codes, DOWNLOAD K-LIME
MOJO REASON CODE PIPELINE and for Python scoring pipeline for K-LIME
reason codes and Shapley, download the SCORING PIPELINE. Note
The k-LIME MOJO Reason Code pipeline does not support multinomial,
natural language processing (NLP), and time series models. []

Prerequisites
The following are required in order to run the k-LIME MOJO reason code
pipeline. -   Java 7 runtime (JDK 1.7) or newer. NOTE: Using Java 11+ is
    recommended due to a bug in Java. For more information, see
    https://bugs.openjdk.java.net/browse/JDK-8186464. -   Valid Driverless AI license.

Copy the license file into the downloaded mojo-pipeline
    folder. -   mojo2-runtime.jar file. This is available from the top navigation
    menu in the Driverless AI UI and in the downloaded mojo-pipeline.zip
    file for an experiment. License Specification
Driverless AI requires a license to be specified in order to run any
DAI/MLI MOJO. The license can be specified with one of the following:
-   An environment variable:
      -   DRIVERLESS_AI_LICENSE_FILE: Path to the Driverless AI license
          file, or
      -   DRIVERLESS_AI_LICENSE_KEY: The Driverless AI license key
          (Base64 encoded string)
-   A system property of JVM (-D option):
      -   ai.h2o.mojos.runtime.license.file: Path to the Driverless AI
          license file, or
      -   ai.h2o.mojos.runtime.license.key: The Driverless AI license
          key (Base64 encoded string)
-   An application classpath:
      -   The license is loaded from a resource called /license.sig.

For example:
    # Specify the license with a temporary environment variable
    export DRIVERLESS_AI_LICENSE_FILE="path/to/license.sig"

k-LIME Reason Code Pipeline Files
The KLIME_MOJO folder includes the following files/folders:
-   DOMAINS
-   EXPERIMENTAL
-   MODEL.INI
-   MODELS

Quickstart
1. On the completed MLI page, click on the DOWNLOAD K-LIME MOJO REASON
    CODE PIPELINE button. 2. To run the Java application for reason code generation directly, use
    the following command:
    java -Dai.h2o.mojos.runtime.license.file=license.sig -cp mojo2-runtime.jar ai.h2o.mojos.ExecuteMojo klime_mojo.zip example.csv

k-LIME MOJO Command Line Options
Executing the Java Runtime
The following are two general examples of how the Java runtime can be
executed from the command-line.

JVM Options
-   sys.ai.h2o.mojos.parser.csv.keepCarriageReturn (boolean) -Specify
    whether to keep the carriage return after parsing. This value
    defaults to True. -   sys.ai.h2o.mojos.parser.csv.stripCrFromLastColumn (boolean)
    -Workaround for issues relating to the OpenCSV parser. This value
    defaults to True. -   sys.ai.h2o.mojos.parser.csv.quotedHeaders (boolean) - Specify
    whether to quote header names in the output CSV file. This value
    defaults to False. -   sys.ai.h2o.mojos.parser.csv.separator (char) - Specify the separator
    used between CSV fields. The special value `TAB` can be used for
    tab-separated values. This value defaults to `,`. -   sys.ai.h2o.mojos.parser.csv.escapeChar (char) - Specify the escape
    character for parsing CSV fields. If this value is not specified,
    then no escaping is attempted. This value defaults to an empty
    string.

This value defaults to 1000. -   sys.ai.h2o.mojos.pipelineFormats (string) - When multiple formats
    are recognized, this option specifies the order in which they are
    tried. This value defaults to `pbuf,toml,klime,h2o3`. -   sys.ai.h2o.mojos.parser.csv.date.formats (string) - Specify a format
    for dates. This value defaults to an empty string. -   sys.ai.h2o.mojos.exposedInputs (string) - Specify a comma separated
    list of input cols that are needed on output. The special value
    `ALL` takes all inputs. This defaults to a null value. -   sys.ai.h2o.mojos.useWeakHash (boolean) - Specify whether to use
    WeakHashMap. This is set to False by default. Enabling this setting
    may improve MOJO loading times. JVM OPTIONS FOR ACCESS CONTROL
-   ai.h2o.mojos.runtime.license.key - Specify a license key. -   ai.h2o.mojos.runtime.license.file - Specify the location of a
    license key. -   ai.h2o.mojos.runtime.license.filename - Override the default license
    file name.

-   ai.h2o.mojos.runtime.watermark.filename - Override the default
    watermark file name. JVM OPTIONS FOR ACCESS CONTROL
    file name. -   ai.h2o.mojos.runtime.signature.filename - Override the default
    signature file name. -   ai.h2o.mojos.runtime.watermark.filename - Override the default
    watermark file name. Execute the MOJO from Java
1. Open a new terminal window. Create an experiment folder and change
    directories to that folder:
2. Create your main program in the EXPERIMENT folder by creating a new
    file called DOCSEXAMPLE.JAVA (for example, with
    vim DocsExample.java). Include the following content:
3. Compile the source code with the files of the MOJO runtime
    (mojo2-runtime.jar) copied into the experiment:
4.

DRIVERLESS AI USER SETTINGS

You can configure several user-specific settings from the UI by clicking
USER -> USER SETTINGS. A window is displayed that lets you set user
settings for various connectors. You can also use the search box to
locate specific user settings. Click the SAVE button to confirm your
changes.

AWS

Specify the following AWS-related user settings:

-   AWS Access Key ID
-   AWS Secret Access Key
-   AWS S3 Bucket name for artifact export

USING MULTIPLE AUTHENTICATORS

Driverless AI lets you enable multiple authentication methods at the
same time. The following are some examples of when this can be useful:

-   When you want to use single sign-on (SSO) options for the front-end
    and also give users direct access with credentials for headless
    setups like the Driverless AI Python client.
-   When you want to allow access to users that are not managed by the
    provider of the primary authentication option.

To enable additional authentications methods, use the

additional_authentication_methods:ref:`config.toml <sample-configtoml>` setting. **Note**: In order to let users access their data when using multiple authenticators, usernames for all of the enabled authentication methods need to match one another. Multiple Authentication Methods Example ---------------------------------------  In this example, a user wants to use OpenID Connect authentication on the front-end and also let users use LDAP credentials to gain access with the Driverless AI Python client. To enable both authentication methods, use the :ref:`config.toml file <sample-configtoml>` to set the following parameters:  ::     authentication_method = "openid"    additional_authentication_methods = "['ldap']"     # Configure OpenID Connect    auth_openid_provider_base_uri = ...     # Configure LDAP    ldap_server = ... The primary authentication method's login page is available on the standard/loginpath.

EXPORTING ARTIFACTS

In some cases, you might find that you do not want your users to
download artifacts directly to their machines. Driverless AI provides
several configuration options/environment variables that enable
exporting of artifacts instead of downloading. Artifacts can be exported
to a file system directory, an Amazon S3 bucket, a Bitbucket repository,
or Azure Blob storage. NOTE: The option to download artifacts is automatically disabled when
exporting is enabled. Enabling Artifact Exports
The config.toml file exposes the following variables:
-   enable_artifacts_upload: Replace all the downloads on the experiment
    page to exports, and lets users push to the artifact store with
    artifacts_store. This is disabled by default. -   artifacts_store: Specify one of the following storage methods:
      -   file_system: Store artifacts in the file system directory
          specified by the artifacts_file_system_directory setting.

-   bitbucket: Store artifacts in a Bitbucket repository. -   azure: Store artifacts in Azure Blob storage. Specify the following for the storage method you selected:
FILE SYSTEM DIRECTORY
-   artifacts_file_system_directory: The file system location where
    artifacts will be copied. This is expected to be a directory on your
    server. AWS S3
-   artifacts_s3_bucket: The AWS S3 bucket where artifacts will be
    stored. BITBUCKET
-   bitbucket_skip_cert_verification: Specify whether to skip
    certificate verification for Bitbucket when using a repository with
    HTTPS. This is disabled by default. -   bitbucket_tmp_relative_dir: Specify a local temporary directory to
    clone artifacts to (relative to data_directory). AZURE BLOB STORAGE
-   artifacts_azure_blob_account_name: Specify your Azure Blob Storage
    account name. -   artifacts_azure_blob_account_key: Specify your Azure Blob Storage
    account key.

enable_dataset_downloading`` configuration option, which is set to

trueby default. Set this tofalse`` if you do not want users to download
datasets to their local machine. There is currently no configuration
option that enables exporting datasets to a file system. Docker Image Installs
The following example shows how to enable artifact exporting to a file
system when starting the Driverless AI Docker image.

PROJECT WORKSPACE

Driverless AI provides a Project Workspace for managing datasets and
experiments related to a specific business problem or use case. Whether
you are trying to detect fraud or predict user retention, datasets and
experiments can be stored and saved in the individual projects. A
Leaderboard on the Projects page lets you easily compare performance and
results and identify the best solution for your problem. The following sections describe how to create and manage projects. -   create-project
-   link-datasets
-   link-experiments
-   experiments-list
NOTE:
-   For information on remote storage and importing datasets and
    experiments through the Projects page, see remote_storage. -   For information on how to export Driverless AI experiments to H2O
    MLOps from the Projects page, see the official H2O MLOps
    documentation. Projects listing page options
The following is a list of options that are available from the PROJECTS
listing page.

-   RENAME: Rename the project. -   EDIT DESCRIPTION: Edit the description for the project. -   SHARE: Share the project with other users. For more information, see
    share_with_other_users. -   GO TO MLOPS: View the project in the MLOps Wave app. -   DELETE: Delete the project. []

Creating a Project Workspace
To create a Project Workspace:
1. Click the PROJECTS option on the top menu. The Projects listing page
    is displayed. 2. Click NEW PROJECT. 3. Specify a name for the project and provide a description. 4. Click CREATE PROJECT. This creates an empty Project page. From the Project page, you can link datasets and/or experiments, run new
experiments, and score experiments on a scoring dataset. When you link
an existing experiment to a Project, the datasets used for the
experiment are automatically linked to the project (if not already
linked). Linking Datasets
Any dataset that has been added to Driverless AI can be linked to a
project.

Note that each
linked dataset is tagged with either LOCAL, REMOTE, or LOCAL & REMOTE to
indicate whether the dataset exists in local storage, remote storage, or
both. The following steps describe how to link a dataset to a project. 1. Click the SHOW DATASETS button. []
2. Click the LINK DATASET button, then select the type of dataset you
    want to upload. Choose from TRAINING, TESTING, and VALIDATION. []
3. Select the dataset(s) that you want to link. 4. (Optional) If there are any completed experiments that are based on
    the selected dataset(s), you can choose to link them as well. 5. (Optional) To filter the list of linked datasets by type, click
    FILTER DATASET TYPE and select the type of dataset you want to view. Choose from Training, Testing, and Validation. When datasets are linked, the same menu options are available here as on
the Datasets page. For more information, refer to Datasets.

The Experiments section shows experiments in the
Project that use the selected dataset. Linking Experiments
Existing experiments can be selected and linked to a Project. Additionally, you can run new experiments or checkpoint existing
experiments from this page. Experiments started from the Project page
are automatically linked to the Project. To link an existing experiment to the project, click LINK EXPERIMENTS
and select one of the following options:
-   BY SELECTING EXPERIMENTS: Select one or more experiments to link to
    the Project. -   BY SELECTING DATASET USED IN EXPERIMENTS: Upload all experiments
    that used the selected dataset as a TRAINING, TESTING, or VALIDATION
    dataset. For example, if you select BY SELECTING DATASET USED IN
    EXPERIMENTS > TRAINING and then select the dataset
    example-dataset.csv, all the experiments that used the
    example-dataset.csv as a training dataset are linked.

1. Click the NEW EXPERIMENT link to begin a new experiment. 2. Select your training data and optionally your validation and/or
    testing data. 3. Specify your desired experiment settings (refer to
    experiment_settings and expert-settings), and then click LAUNCH
    EXPERIMENT. As the experiment is running, it will be listed at the top of the
Experiments Leaderboard until it is completed. It will also be available
on the EXPERIMENTS page. Checkpointing Experiments
When experiments are linked to a Project, the same checkpointing options
for experiments are available here as on the Experiments page. Refer to
checkpointing for more information. []

Experiments List
When attempting to solve a business problem, a normal workflow will
include running multiple experiments, either with different/new data or
with a variety of settings, and the optimal solution can vary for
different users and/or business problems.

Other users might be willing to make an acceptable
compromise on the accuracy of the model for a model with greater
performance (faster prediction). For some, it could also mean how
quickly the model could be trained with acceptable levels of accuracy. The Experiments list allows you to find the best solution for your
business problem. The list is organized based on experiment name. You can change the
sorting of experiments by selecting the up/down arrows beside a column
heading in the experiment menu. Hover over the right menu of an experiment to view additional
information about the experiment, including the problem type, datasets
used, and the target column. Experiment Scoring
Finished experiments linked to the project show their validation and
test scores. You can also score experiments on other datasets. To do
this, you first need to add a dataset by clicking the LINK DATASET
button and choosing TESTING from the drop-down menu.

This triggers a diagnostics job, the results of
which are located on the diagnostics page. (Refer to Diagnosing a Model
for more information.) After the scoring process has completed, the
result appears in the SCORE and SCORING TIME columns. The SCORE column
shows results for the scorer specified by the SHOW RESULTS FOR SCORER
picker. NOTES:
-   If an experiment has already been scored on a dataset, Driverless AI
    cannot score it again. The scoring step is deterministic, so for a
    particular test dataset and experiment combination, the score will
    be same regardless of how many times you repeat it. -   The test dataset must have all the columns that are expected by the
    various experiments you are scoring it on. However, the columns of
    the test dataset need not be exactly the same as input features
    expected by the experiment. There can be additional columns in the
    test dataset. If these columns were not used for training, they will
    be ignored.

-   A TEST TIME column is available in the Experiments Leaderboard. This
    value shows the total time (in seconds) that it took for calculating
    the experiment scores for all applicable scorers for the experiment
    type. This is valuable to users who need to estimate the runtime
    performance of an experiment. Comparing Experiments
You can compare two or three experiments and view side-by-side detailed
information about each. 1. Select either two or three experiments that you want to compare. You
    cannot compare more than three experiments. 2. Click the COMPARE _n_ ITEMS button. This opens the COMPARE EXPERIMENTS page. This page includes the
experiment summary and metric plots for each experiment. The metric
plots vary depending on whether this is a classification or regression
experiment.

Predicted Graph
[]
Unlinking Data on a Projects Page
Unlinking datasets and/or experiments does not delete that data from
Driverless AI. The datasets and experiments will still be available on
the DATASETS and EXPERIMENTS pages. -   Unlink a dataset by clicking on the dataset and selecting UNLINK
    from the menu. NOTE: You cannot unlink datasets that are tied to
    experiments in the same project. -   Unlink an experiment by selecting the experiment and clicking the
    UNLINK ITEM button. Note that this will not automatically unlink
    datasets that were tied to the experiment. Deleting Projects
To delete a project, click the PROJECTS option on the top menu to open
the main Projects page. Click the dotted menu the right-most column, and
then select Delete. You will be prompted to confirm the deletion. Note that deleting projects does not delete datasets and experiments
from Driverless AI.

EXPERIMENT QUEUING IN DRIVERLESS AI

Driverless AI supports automatic queuing of experiments to avoid system
overload. You can launch multiple experiments simultaneously that are
automatically queued and run when the necessary resources become
available. The worker queue indicates the number of experiments that are waiting
for their turn on a CPU or GPU + CPU system. Significant jobs like
running experiments and making predictions are distinguished from minor
tasks. In the following image, 'GPU queue' indicates that there are two
experiments waiting in the worker queue on a GPU-enabled system, and not
that two workers are waiting for a GPU:
[]
NOTES:
-   By default, each node runs two experiments at a time. This is
    controlled by the worker_remote_processors option in the
    config.toml file <sample-configtoml>.

MAC OS X

This section describes how to install, start, stop, and upgrade the
Driverless AI Docker image on Mac OS X. Note that this uses regular
Docker and not NVIDIA Docker. NOTE: Support for GPUs and MOJOs is not available on Mac OS X. The installation steps assume that you have a license key for Driverless
AI. For information on how to obtain a license key for Driverless AI,
visit https://h2o.ai/o/try-driverless-ai/. Once obtained, you will be
prompted to paste the license key into the Driverless AI UI when you
first log in, or you can save it as a .sig file and place it in the
license folder that you will create during the installation process. CAUTION:
-   This is an extremely memory-constrained environment for experimental
    purposes only. Stick to small datasets! For serious use, please use
    Linux. -   Be aware that there are known performance issues with Docker for
    Mac.

Environment
  -------------------------------------------------------------------
  Operating System        GPU Support? Min Mem   Suitable for
  ----------------------- --------------- --------- -----------------
  Mac OS X                No              16 GB     Experimentation

Installing Driverless AI
1. Retrieve the Driverless AI Docker image from
    https://www.h2o.ai/download/. 2. Download and run Docker for Mac from
    https://docs.docker.com/docker-for-mac/install. 3. Adjust the amount of memory given to Docker to be at least 10 GB. Driverless AI won't run at all with less than 10 GB of memory. You
    can optionally adjust the number of CPUs given to Docker. You will
    find the controls by clicking on (Docker
    Whale)->Preferences->Advanced as shown in the following screenshots. (Don't forget to Apply the changes after setting the desired memory
    value.) [image]
[image]
4.

More information is available here:
    https://docs.docker.com/docker-for-mac/osxfs/#namespaces. [image]
5. Set up a directory for the version of Driverless AI within the
    Terminal:
6. With Docker running, open a Terminal and move the downloaded
    Driverless AI image to your new directory. 7. Change directories to the new directory, then load the image using
8. Set up the data, log, license, and tmp directories (within the new
    Driverless AI directory):
9. Optionally copy data into the DATA directory on the host. The data
    will be visible inside the Docker container at /DATA. You can also
    upload data after starting Driverless AI. 10. Run docker images to find the image tag. 11. Start the Driverless AI Docker image (still within the new
    Driverless AI directory). REPLACE TAG BELOW WITH THE IMAGE TAG. Note
    that GPU support will not be available.

12. Connect to Driverless AI with your browser at
    http://localhost:12345. Stopping the Docker Image
To stop the Driverless AI Docker image, type CTRL + C in the Terminal
(Mac OS X) or PowerShell (Windows 10) window that is running the
Driverless AI Docker image. Upgrading the Docker Image
      directory before upgrading. If you did not build MLI on a model before upgrading Driverless AI,
  then you will not be able to view MLI on that model after upgrading.

If that MLI job appears in
  tmp directory. NOTE: Stop Driverless AI if it is still running. Upgrade Steps
1. SSH into the IP address of the machine that is running Driverless
    AI. 2. Set up a directory for the version of Driverless AI on the host
    machine:
3. Retrieve the Driverless AI package from https://www.h2o.ai/download/
    and add it to the new directory. 4. Load the Driverless AI Docker image inside the new directory:
5. Copy the data, log, license, and tmp directories from the previous
    Driverless AI directory to the new Driverless AI directory:
6.

UPGRADING THE DRIVERLESS AI COMMUNITY IMAGE

WARNINGS:
-   This release deprecates experiments and MLI models from 1.7.0 and
    earlier. -   Experiments, MLIs, and MOJOs reside in the Driverless AI tmp
    directory and are not automatically upgraded when Driverless AI is
    upgraded. We recommend you take the following steps before
    upgrading. -   Build MLI models before upgrading. -   Build MOJO pipelines before upgrading. -   Stop Driverless AI and make a backup of your Driverless AI tmp
        directory before upgrading. The upgrade process inherits the service user and group from
/etc/dai/User.conf and /etc/dai/Group.conf. You do not need to manually
specify the DAI_USER or DAI_GROUP environment variables during an
upgrade. Upgrading from Version 1.2.2 or Earlier
The following example shows how to upgrade from 1.2.2 or earlier to the
current version. Upgrading from these earlier versions requires an edit
to the start and h2oai scripts. 1.

EXPERIMENT PERFORMANCE

This page describes the factors that contribute to the performance of
Driverless AI experiments. Each completed experiment iteration in Driverless AI experiments is a
fitted model, but you can control the number of iterations with the time
dial and the parameter_tuning_num_models TOML config mentioned in the
following section. Additionally, each model takes some number of model
iterations. XGBoost builds trees with a default up to about 3000 trees,
but this can be modified with the max_nestimators TOML config mentioned
in the following section. List of TOML configs that can affect performance
The following list describes a variety of controls over the experiment
and model runtimes:
-   Set max_runtime_minutes to a smaller number of minutes, e.g. 60 for
    1 hour allowed. By default, DAI uses minimum of its estimate of an
    experiment runtime and max_runtime_minutes, or greater than 1 hour
    as chosen by min_auto_runtime_minutes.

Using GPUs is highly recommended for XGBoost when one
    has several classes, when many trees are built, or data is 100k to
    1M rows and 100-300 columns, or whatever would fit onto the GPU
    memory for larger data. -   Set the time dial to a lower value, which will do fewer models in
    tuning and evolution phases. -   Set the interpretability dial to a larger value, which will more
    aggressively prune weak features, prune weak base models in
    ensemble, and avoid high-order feature interactions (interaction
    depth). You can also set fixed_feature_interaction_depth to control
    interaction depth directly. -   Set parameter_tuning_num_models to a fixed non-zero but small value,
    to directly control number of tuning models instead of set
    automatically by dials. -   Set the max_nestimators TOML config to a lower value (for example,
    500, 1000, 1500, or 2000) instead of the default value of
    3000.

Sometimes the data and model
    are such that many trees continue to learn, but the gains are
    minimal for the metric chosen. You can also set
    fixed_max_nestimators to directly control max independent of dials. For RF and Dart, change n_estimators_list_no_early_stopping instead. -   If the system is used by single user, set exclusive_mode to
    moderate. -   Set enable_early_stopping_threshold to 0.01-0.1, which for (only)
    LightGBM will avoid using too many trees when evaluation metric for
    tree building has relative change less than this value. -   Set max_abs_score_delta_train_valid and
    max_rel_score_delta_train_valid to a non-zero value to limit the
    number of trees by difference between train and valid scores on
    metric chosen to optimize. -   Set reduce_mojo_size=True. In cases where the MOJO is too large or
    slow, you can also set the nfeatures_max TOML config to a value that
    is lower than the number of features you have.

-   Set the min_learning_rate_final to a higher value (for example,
    0.03). You can set max_learning_rate_final equal to
    min_learning_rate_final to force a fixed learning rate in final
    model. Use min_learning_rate and max_learning_rate to control the
    learning rate for tuning and evolution models. -   Set nfeatures_max to limit the number of features. This is useful in
    conjuction with ngenes_max to control the maximum number of
    transformations (each could make 1 or more features). -   Set ensemble_level and fixed_ensemble_level to smaller values, e.g. 0 or 1, to limit the number of base models in final model. -   Set fixed_fold_reps to a smaller value, e.g. 1, to limit the number
    of repeats. -   Set max_max_depth to a smaller value, e.g. 8, to avoid trying larger
    depths for tree models. -   Set max_max_bin to a smaller value, e.g. 128, to avoid larger
    max_bin values for tree models.

This
    loses reproducibility for the TensorFlow model, but the rest of DAI
    will be reproducible. Note that the runtime estimate doesn't take into account the number of
trees needed for your data. The more trees needed by your data, the
greater the amount of time needed to complete an experiment. To reduce
the upper limit on the number of trees, set the max_nestimators TOML
config to a value lower than 3000. Also note that the dai_wizard displays comparisons of various runtime
estimations and lets you specify runtime limits. Additional information
It's possible that your experiment has gone from using GPUs to using
CPUs due to a change of the host system outside of Driverless AI's
control. You can verify this using any of the following methods:
-   Check GPU usage by going to your Driverless AI experiment page and
    clicking on the GPU USAGE tab in the lower-right quadrant of the
    experiment.

GENERAL CONSIDERATIONS

Machine Learning and Approximate Explanations
For years, common sense has deemed the complex, intricate formulas
created by training machine learning algorithms to be uninterpretable. While great advances have been made in recent years to make these often
nonlinear, non-monotonic, and non-continuous machine-learned response
functions more understandable (Hall et al, 2017), it is likely that such
functions will never be as directly or universally interpretable as more
traditional linear models. Why consider machine learning approaches for inferential purposes? In
general, linear models focus on understanding and predicting average
behavior, whereas machine-learned response functions can often make
accurate, but more difficult to explain, predictions for subtler aspects
of modeled phenomenon. In a sense, linear models create very exact
interpretations for approximate models. The approach here seeks to make
approximate explanations for very exact models.

Moreover, the use of machine learning techniques for inferential
or predictive purposes does not preclude using linear models for
interpretation (Ribeiro et al, 2016). The Multiplicity of Good Models in Machine Learning
It is well understood that for the same set of input variables and
prediction targets, complex machine learning algorithms can produce
multiple accurate models with very similar, but not exactly the same,
internal architectures (Breiman, 2001). This alone is an obstacle to
interpretation, but when using these types of algorithms as
interpretation tools or with interpretation tools it is important to
remember that details of explanations will change across multiple
accurate models. Expectations for Consistency Between Explanatory Techniques
-   The decision tree surrogate is a global, nonlinear description of
    the Driverless AI model behavior.

For certain, more linear Driverless
    AI models, variables that appear in the decision tree surrogate
    model may also have large coefficients in the global K-LIME model. -   K-LIME explanations are linear, do not consider interactions, and
    represent offsets from the local linear model intercept. LOCO
    importance values are nonlinear, do consider interactions, and do
    not explicitly consider a linear intercept or offset. LIME
    explanations and LOCO importance values are not expected to have a
    direct relationship but can align roughly as both are measures of a
    variable's local impact on a model's predictions, especially in more
    linear regions of the Driverless AI model's learned response
    function. -   ICE is a type of nonlinear sensitivity analysis which has a complex
    relationship to LOCO feature importance values.

When
    comparing ICE to LOCO the total value of the prediction for the row,
    the value of the variable in the selected row, and the distance of
    the ICE value from the average prediction for the selected variable
    at the value in the selected row must all be considered. -   ICE curves that are outside the standard deviation of partial
    dependence would be expected to fall into less populated decision
    paths of the decision tree surrogate; ICE curves that lie within the
    standard deviation of partial dependence would be expected to belong
    to more common decision paths. -   Partial dependence takes into consideration nonlinear, but average,
    behavior of the complex Driverless AI model without considering
    interactions. Variables with consistently high partial dependence or
    partial dependence that swings widely across an input variable's
    domain will likely also have high global importance values.

NLP IN DRIVERLESS AI

This section describes NLP (text) processing capabilities of Driverless
AI. The Driverless AI platform has the ability to support both
standalone text and text with other column types as predictive features. TENSORFLOW based and PYTORCH Transformer Architectures (for example,
BERT) are used for Feature Engineering and Model Building. For details, see:
  -   NLP Feature Engineering and Modeling <nlp_fe>
  -   NLP Expert Settings <nlp_expert>
  -   NLP Feature Naming Convention <nlp_name>
  -   nlp-explainers
  -   An NLP example in Driverless AI <nlp_exp>
  -   NLP Models to Production <nlp_prod>
Note
- NLP and image use cases in Driverless benefit significantly from
GPU usage <gpu_in_dai>. - To download pretrained NLP models, visit
http://s3.amazonaws.com/artifacts.h2o.ai/releases/ai/h2o/pretrained/bert_models.zip. You can use the pytorch_nlp_pretrained_models_dir configuration option
to specify a path to pretrained PyTorch NLP models.

For example, you can set this option as
follows:
pytorch_nlp_pretrained_models_dir=/path/on/server/to/bert_models_folder. - You can use the Driverless AI Experiment Setup Wizard to guide you
through the process of setting up NLP experiments. For more information,
see dai_wizard. NLP Feature Engineering and Modeling
[]
Pretrained PyTorch Models in Driverless AI
[]
The following NLP recipes are available for a text column. A full list
of NLP TRANSFORMERS is available here <text_transformers>.

N-GRAM FREQUENCY
Frequency-based features represent the count of each word from a given
text in the form of vectors. These are created for different n-gram
values. For example, a one-gram is equivalent to a single word, a
two-gram is equivalent to two consecutive words paired together, and so
on. Words and n-grams that occur more often will receive a higher
weightage. The ones that are rare will receive a lower weightage. TF-IDF OF N-GRAMS
Frequency-based features can be multiplied with the inverse document
frequency to get term frequency–inverse document frequency (TF-IDF)
vectors. Doing so also gives importance to the rare terms that occur in
the corpus, which may be helpful in certain classification tasks. []
TRUNCATED SVD FEATURES
TF-IDF and the frequency of n-grams both result in higher dimensions of
the representational vectors. To counteract this, TRUNCATED SVD is
commonly used to decompose the vectorized arrays into lower dimensions.

These
capture linear dependencies that are crucial to the process of achieving
high accuracy rates and are used as features in the base DAI model. WORD EMBEDDINGS
Word embeddings is the term for a collective set of feature engineering
techniques for text where words or phrases from the vocabulary are
mapped to vectors of real numbers. Representations are made so that
words with similar meanings are placed close to or equidistant from one
another. For example, the word "king" is closely associated with the
word "queen" in this kind of vector representation. []
TF-IDF and frequency-based models represent counts and significant word
information, but they lack the semantic context for these words. Word
embedding techniques are used to make up for this lack of semantic
information.

In Driverless AI, we pass word embeddings as input to CNN
models, which return cross validated predictions that can be used as a
new set of features. []
BI-DIRECTIONAL GRU MODELS FOR WORD EMBEDDING
Recurrent neural networks, like long short-term memory units (LSTM) and
gated recurrent units (GRU), are state-of-the-art algorithms for NLP
problems. In Driverless AI, we implement bi-directional GRU features for
previous word steps and for later steps to predict the current state. For example, in the sentence "John is walking on the golf course," a
unidirectional model would represent states that represent "golf" based
on "John is walking on," but would not represent "course." Using a
bi-directional model, the representation would also account the later
representations, giving the model more predictive power. In simple terms, a bi-directional GRU model combines two independent RNN
models into a single model.

As with CNN models, we
pass word embeddings as input to these models, which return cross
validated predictions that can be used as a new set of features. []
CNN MODELS FOR CHARACTER EMBEDDING
For languages like Japanese and Mandarin Chinese, where characters play
a major role, character level embedding is available as an NLP recipe. In character embedding, each character is represented in the form of
vectors rather than words. Driverless AI uses character level embedding
as the input to CNN models and later extracts class probabilities to
feed as features for downstream models. The image below represents the overall set of features created by this
NLP recipe:
[]
BERT/DISTILBERT MODELS FOR FEATURE ENGINEERING
Transformer based language models like BERT are state-of-the-art NLP
models that can be used for a wide variety of NLP tasks. These models
capture the contextual relation between words by using an attention
mechanism.

The embeddings obtained by these models show improved
results in comparison to earlier embedding approaches. []
BERT and DistilBERT models can be used for generating embeddings for any
text columns. These pretrained models are used to get embeddings for the
text followed by Linear/Logistic Regression to generate features that
can then be used for any downstream models in Driverless AI. Refer to
nlp-settings in the Expert Settings topic for more information on how to
enable these models for feature engineering. We recommend using GPU(s)
to leverage the power of these models and accelerate the feature
engineering process. PYTORCH TRANSFORMER ARCHITECTURE MODELS (EG. BERT) AS MODELING
ALGORITHMS
Starting with Driverless AI 1.9 release, the Transformer-based
architectures shown in the diagram below is supported as models in
Driverless AI. []
The BERT model support multiple languages.

The DistilBERT model can be useful when training time and
model size is important. Refer to nlp-settings in the Expert Settings
topic for more information on how to enable these models as modeling
algorithms. We recommend using GPU(s) to leverage the power of these
models and accelerate the model training time. In addition to these techniques, Driverless AI supports
custom NLP recipes <custom-recipes> using, for example, PyTorch or
Flair. NLP Feature Naming Convention
The naming conventions of the NLP features help to understand the type
of feature that has been created. The syntax for the feature names is as follows:
[FEAT TYPE]:[COL]. [TARGET_CLASS]
-   [FEAT TYPE] represents one of the following:
-   [COL] represents the name of the text column. -   [TARGET_CLASS] represents the target class for which the model
    predictions are made.

[]

NLP Explainers
The following is a list of available NLP explainers. For more
information, refer to mli_default_recipes and mli-nlp-plots. DEFAULT EXPLAINERS
The following NLP explainers are run by default for NLP experiments. -   NLP LOCO EXPLAINER: The NLP LOCO plot applies a
    leave-one-covariate-out (LOCO) styled approach to NLP models by
    removing a specific token from all text features in a record and
    predicting local importance without that token. The difference
    between the resulting score and the original score (token included)
    is useful when trying to determine how specific changes to text
    features alter the predictions made by the model. -   NLP PARTIAL DEPENDENCE PLOT EXPLAINER: NLP partial dependence
    (yellow) portrays the average prediction behavior of the Driverless
    AI model when an input text token is left in its respective text and
    not included in its respective text along with +/- 1 standard
    deviation bands.

The text
    tokens are generated from TF-IDF. LEGACY EXPLAINERS
The following legacy NLP explainers are not run by default for NLP
experiments. You can run legacy NLP explainers by using config
variables. -   NLP TOKENIZER EXPLAINER: NLP tokenizer plot shows both the global
    and local importance values of each token in a corpus (a large and
    structured set of texts). The corpus is automatically generated from
    text features used by Driverless AI models prior to the process of
    tokenization. Local importance values are calculated by using the
    term frequency-inverse document frequency (TF-IDF) as a weighting
    factor for each token in each row. The TF-IDF increases
    proportionally to the number of times a token appears in a given
    document and is offset by the number of documents in the corpus that
    contain the token.

The coefficients of the linear model give the
    importance of the words. Note that by default, this explainer uses
    the first text column based on alphabetical order. NLP Expert Settings
A number of configurable settings are available for NLP in Driverless
AI. For more information, refer to nlp-settings in the Expert Settings
topic. Also see NLP MODEL and NLP TRANSFORMER in
pipeline building recipes <pipeline-building-recipe> under experiment
settings. []

An NLP Example: Sentiment Analysis
The following section provides an NLP example. This information is based
on the Automatic Feature Engineering for Text Analytics blog post. A
similar example using the Python Client is available in python_client. This example uses a classic example of sentiment analysis on tweets
using the US Airline Sentiment dataset. Note that the sentiment of each
tweet has been labeled in advance and that our model will be used to
label new tweets.

We will use the tweets
in the ‘text’ column and the sentiment (positive, negative or neutral)
in the ‘airline_sentiment’ column for this demo. Here are some samples
from the dataset:
[]
Once we have our dataset ready in the tabular format, we are all set to
use the Driverless AI. Similar to other problems in the Driverless AI
setup, we need to choose the dataset, and then specify the target column
(‘airline_sentiment’). []
Because we don't want to use any other columns in the dataset, we need
to click on DROPPED COLS, and then exclude everything but TEXT as shown
below:
[]
Next, we will turn on our TensorFlow NLP recipes. We can go to the
EXPERT SETTINGS window, NLP <nlp-settings> and turn on the following:
CNN TENSORFLOW MODELS, BIGRU TENSORFLOW MODELS, CHARACTER-BASED
TENSORFLOW MODELS or PRETRAINED PYTORCH NLP MODELS. []
At this point, we are ready to launch an experiment. Text features will
be automatically generated and evaluated during the feature engineering
process.

LINUX DOCKER IMAGES

To simplify local installation, Driverless AI is provided as a Docker
image for the following system combinations:
  ---------------------------------------------------------------------
  Host OS                     Docker Version Host Architecture Min Mem
  --------------------------- -------------- ----------------- --------
  Ubuntu 16.04 or later       Docker CE      x86_64            64 GB
  RHEL or CentOS 7.4 or later Docker CE      x86_64            64 GB
  NVIDIA DGX Registry                        x86_64            
NOTE: CUDA 11.2.2 or later with NVIDIA drivers >= is recommended (GPU
only). Note that if you are using K80 GPUs, the minimum required NVIDIA
driver version is 450.80.02. For the best performance, including GPU support, use nvidia-docker. For
a lower-performance experience without GPUs, use regular docker (with
the same docker image).

--shm-size=2g --cap-add=SYS_NICE --ulimit nofile=131071:131071 --ulimit
nproc=16384:16384` in docker launch command. But if user plans to build
:ref:`image auto model <image-model> extensively, then

--shm-size=4g`` is recommended for Driverless AI docker command.

ubuntu rhel nvidia-dgx

ADDING DATASETS

You can add datasets using one of the following methods:
Drag and drop files from your local machine directly onto this page. Note that this method currently works for files that are less than 10
GB. or
Click the ADD DATASET (OR DRAG & DROP) button to upload or add a
dataset. NOTES:
-   Upload File, File System, HDFS, S3, Data Recipe URL, and Upload Data
    Recipe are enabled by default. These can be disabled by removing
    them from the enabled_file_systems setting in the config.toml file. (Refer to Using the config.toml file section for more information.) -   If File System is disabled, Driverless AI will open a local
    filebrowser by default. -   If Driverless AI was started with data connectors enabled for Azure
    Blob Store, BlueData Datatap, Google Big Query, Google Cloud
    Storage, KDB+, Minio, Snowflake, or JDBC, then these options will
    appear in the ADD DATASET (OR DRAG & DROP) dropdown menu.

-   When specifying to add a dataset using DATA RECIPE URL, the URL must
    point to either an HTML or raw version of the file, a GitHub
    repository or tree, or a local file. When adding or uploading
    datasets via recipes, the dataset will be saved as a .jay file. -   Datasets must be in delimited text format. -   Driverless AI can detect the following separators: ,|;t
-   When importing a folder, the entire folder and all of its contents
    are read into Driverless AI as a single file. -   When importing a folder, all of the files in the folder must have
    the same columns. -   If you try to import a folder via a data connector on Windows, the
    import will fail if the folder contains files that do not have file
    extensions (the resulting error is usually related to the above
    note). Upon completion, the datasets will appear in the Datasets Overview page. Click on a dataset to open a submenu.

DRIVERLESS AI HEALTH API

The following sections describe the Driverless AI Health API. -   health-api-overview
-   retrieve-health-status
-   health-api-json-attributes

Overview
The Driverless AI Health API is a publicly available API that exposes
basic system metrics and statistics. Its primary purpose is to provide
information for resource monitoring and auto-scaling of
Driverless AI multinode <multinode-training> clusters. The API outputs a
set of metrics in a JSON format so that they can be used by tools like
KEDA or K8S Autoscaler. NOTES:
-   The Health API is only available in multinode or singlenode mode. For more information, refer to the worker_mode
    config.toml <sample-configtoml> option. -   For security purposes, the Health API endpoint can be disabled by
    setting the enable_health_api config.toml <sample-configtoml> option
    to false. This setting is enabled by default.

It can also be used in
    tandem with services like Enterprise Puddle to skip the
    authentication step and instead retrieve the needed information
    directly.

-   API_VERSION (string): API version
-   SERVER_VERSION (string): Driverless AI server version
-   APPLICATION_ID (STRING): Driverless AI instance ID that is randomly
    created at startup or overridden with config.application_id
-   TIMESTAMP (string): Current server time in ISO8601 format
-   LAST_SYSTEM_INTERACTION (string): ISO8601 format timestamp of last
    interaction with the Driverless AI server. The following are
    considered as system interactions:
-   IS_IDLE (boolean): System is considered idle when there is no task
    running or scheduled, and no upload or download going on from the
    user browser session
-   ACTIVE_USERS (int): Number of active users in the system.

MODEL RETRAINING USING RECIPES

This page describes how to retrain a model using recipes.

To avoid any extra auto-generated individuals for refit or retrained
experiments with a custom individual, set fixed_ensemble_level and

fixed_num_individuals`` equal to the desired number of custom
individuals in the final model. Also, ensure that

enable_genetic_algorithmis set tooff.  Additionally, you can set the following parameters to prevent the mutation of the genome for this individual (frozen case), in self.params dictionary in Individual Recipe'sset_params``
method:

    prob_add_genes = 0.0
    prob_prune_genes = 0.0
    prob_prune_by_features = 0.0
    prob_addbest_genes = 0.0
    prob_prune_by_top_features = 0.0

Also, uncomment the following code in the Individual Recipe's

set_params`` method:

    self.config_dict.update(self.config_dict_individual)
    self.config_dict.update(self.config_dict_experiment)

Note that the validation scores can have some randomness, but when the
parent experiment has a reproducible level set with the relevant
reproducible levels, the test scores should match.

USING THE CONFIG.TOML FILE

The config.toml file is a configuration file that uses the TOML v0.5.0
file format. Administrators can customize various aspects of a
DRIVERLESS AI (DAI) environment by editing the config.toml file before
starting DAI. Note
For information on configuration security, see configuration-security. Configuration Override Chain
The configuration engine reads and overrides variables in the following
order:
1. DRIVERLESS AI DEFAULTS: These are stored in a Python config module. 2.  config.toml - Place this file in a folder or mount it in a Docker
    container and specify the path in the "DRIVERLESS_AI_CONFIG_FILE"
    environment variable. 3. KEYSTORE FILE - Set the keystore_file parameter in the config.toml
    file or the environment variable "DRIVERLESS_AI_KEYSTORE_FILE" to
    point to a valid DAI keystore file generated using the
    h2oai.keystore tool.

4. ENVIRONMENT VARIABLE - Configuration variables can also be provided
    as environment variables. They must have the prefix DRIVERLESS_AI_
    followed by the variable name in all caps. For example,
    "authentication_method" can be provided as
    "DRIVERLESS_AI_AUTHENTICATION_METHOD". Setting environment variables
    overrides values from the keystore file. Docker Image Users
1. Copy the config.toml file from inside the Docker image to your local
    filesystem. 2. Edit the desired variables in the config.toml file. Save your
    changes when you are done. 3. Start DAI with the DRIVERLESS_AI_CONFIG_FILE environment variable. Ensure that this environment variable points to the location of the
    edited config.toml file so that the software can locate the
    configuration file. Sample config.toml File
The following is a copy of the standard config.toml file included with
this version of DAI.

USING A CUSTOM SCORER

Driverless AI supports a number of scorers, including:
-   Regression: GINI, MAE, MAPE, MER, MSE, R2, RMSE (default), RMSLE,
    RMSPE, SMAPE, TOPDECILE
-   Classification: ACCURACY, AUC (default), AUCPR, F05, F1, F2, GINI,
    LOGLOSS, MACROAUC, MCC
This example shows how you can include a custom scorer in your
experiment. This example will use the Explained Variance scorer, which
is used for regression experiments. 1. Start an experiment in Driverless AI by selecting your training
    dataset along with (optionally) validation and testing datasets and
    then specifying a (regression) Target Column. 2. The scorer defaults to RMSE. Click on EXPERT SETTINGS. 3. Specify the custom scorer recipe using one of the following methods:
4. In the Experiment Summary page, select the new Explained Variance
    (EXPVAR) scorer.

DRIVERLESS AI STANDALONE PYTHON SCORING PIPELINE

A standalone Python scoring pipeline is available after successfully
completing an experiment. This package contains an exported model and
Python 3.8 source code examples for productionizing models built using
H2O Driverless AI. The files in this package let you transform and score on new data in
several different ways:
-   From Python 3.8, you can import a scoring module and use it to
    transform and score on new data. -   From other languages and platforms, you can use the TCP/HTTP scoring
    service bundled with this package to call into the scoring pipeline
    module through remote procedure calls (RPC).

Note
If you use the virtualenv or pip run methods of Python scoring, CUDA,
OpenCL, and cuDNN must be manually installed. For more information, see
cuda-opencl-cudnn. Note
The downloaded scorer zip file contains a shell script called
run_example.sh, which is used to set up a virtual environment and run an
example Python script.

If this pipeline is then deployed
in a container, the H2O recipe server causes the size of the pipeline to
be much larger. In addition, Java has to be installed in the container,
which further increases the runtime storage and memory requirements. A
workaround is to set the following environment variable before running
the Python Scoring Pipeline:
    export dai_enable_custom_recipes=0
CUDA, OpenCL, and cuDNN Install Instructions
Refer to the following sections for instructions on installing CUDA,
OpenCL, and cuDNN when using the virtualenv or pip run methods of Python
scoring. Installing CUDA with NVIDIA Drivers
Before installing CUDA, make sure you have already installed wget, gcc,
make, and elfutils-libelf-devel:
    sudo yum -y install wget
    sudo yum -y install gcc
    sudo yum -y install make
    sudo yum -y install elfutils-libelf-devel
Next, visit
https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html
for instructions on installing CUDA.

If prompted to select what tools you would like to install, select
Drivers only. Installing OpenCL
Run the following to install OpenCL for Centos7 or RH7 based systems
using yum and x86. sudo yum -y clean all
    sudo yum -y makecache
    sudo yum -y update
    wget http://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/c/clinfo-2.1.17.02.09-1.el7.x86_64.rpm
    wget http://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/o/ocl-icd-2.2.12-1.el7.x86_64.rpm
    sudo rpm -if ocl-icd-2.2.12-1.el7.x86_64.rpm
    sudo rpm -if clinfo-2.1.17.02.09-1.el7.x86_64.rpm
    clinfo
    mkdir -p /etc/OpenCL/vendors && \
        echo "libnvidia-opencl.so.1" > /etc/OpenCL/vendors/nvidia.icd
Installing cuDNN
For information on installing cuDNN on Linux, refer to
https://docs.nvidia.com/deeplearning/cudnn/install-guide/index.html. Note
cuDNN 8 or later is required.

-   run_example.sh: Runs example.py (also sets up a virtualenv with
    prerequisite libraries). For more information, refer to the second
    note in the python-scoring-before section. -   tcp_server.py: A standalone TCP server for hosting scoring services. -   http_server.py: A standalone HTTP server for hosting scoring
    services. -   run_tcp_server.sh: Runs TCP scoring service (runs tcp_server.py). -   run_http_server.sh: Runs HTTP scoring service (runs http_server.py). -   example_client.py: An example Python script demonstrating how to
    communicate with the scoring server. -   run_tcp_client.sh: Demonstrates how to communicate with the scoring
    service via TCP (runs example_client.py). -   run_http_client.sh: Demonstrates how to communicate with the scoring
    service via HTTP (using curl). Quick Start
There are two methods for starting the Python Scoring Pipeline.

Use this method if:
-   You have an air gapped environment with no access to the Internet. -   You want to use a quick start approach. Prerequisites
-   A valid Driverless AI license key. -   A completed Driverless AI experiment. -   Downloaded Python Scoring Pipeline. Running the Python Scoring Pipeline - Recommended
NOTE: Scorers that use custom recipes must be run from the working
directory of the scoring-pipeline tmp folder. 1. Download the TAR SH version of Driverless AI from
    https://www.h2o.ai/download/. 2. Use bash to execute the download. This creates a new
    DAI-<DAI_VERSION> folder, where <DAI_VERSION> represents your
    version of Driverless AI, for example, 1.7.1-LINUX-X86_64.) 3. Change directories into the new Driverless AI folder. (Replace
    <dai_version> below with your the version that was created in Step
    2.) 4. Run the following to change permissions:
5.

Run the following command from the scoring-pipeline directory:
Quick Start - Alternative Method
This section describes an alternative method for running the Python
Scoring Pipeline. This version requires Internet access. Note
If you use a scorer from a version prior to 1.10.4.1, you need to add
export SKLEARN_ALLOW_DEPRECATED_SKLEARN_PACKAGE_INSTALL=True prior to
creating the new scorer python environment, either in run_example.sh or
in the same terminal where the shell scripts are executed. Note that
this change will not be necessary when installing the scorer on top of
an already built tar.sh, which is the recommended way. Prerequisites
-   The scoring module and scoring service are supported only on Linux
    with Python 3.8 and OpenBLAS. -   The scoring module and scoring service download additional packages
    at install time and require Internet access. Depending on your
    network environment, you might need to set up internet access via a
    proxy.

Driverless AI requires a license to be
    specified in order to run the Python Scoring Pipeline. -   Apache Thrift (to run the scoring service in TCP mode)
-   Linux environment
-   Python 3.8
-   libopenblas-dev (required for H2O4GPU)
-   OpenCL
For info on how to install these prerequisites, refer to the following
examples. INSTALLING PYTHON 3.8 AND OPENBLAS ON UBUNTU 16.10 OR LATER:
    sudo apt install python3.8 python3.8-dev python3-pip python3-dev \
      python-virtualenv python3-virtualenv libopenblas-dev
INSTALLING PYTHON 3.8 AND OPENBLAS ON UBUNTU 16.04:
    sudo add-apt-repository ppa:deadsnakes/ppa
    sudo apt-get update
    sudo apt-get install python3.8 python3.8-dev python3-pip python3-dev \
INSTALLING CONDA 3.6:
  You can install Conda using either Anaconda or Miniconda.

DRIVERLESS_AI_LICENSE_KEYwould be similar. **Installing the Thrift Compiler**  Thrift is required to run the scoring service in TCP mode, but it is not required to run the scoring module. The following steps are available on the Thrift documentation site at: https://thrift.apache.org/docs/BuildingFromSource. ::     sudo apt-get install automake bison flex g++ git libevent-dev \      libssl-dev libtool make pkg-config libboost-all-dev ant    wget https://github.com/apache/thrift/archive/0.10.0.tar.gz    tar -xvf 0.10.0.tar.gz    cd thrift-0.10.0    ./bootstrap.sh    ./configure    make    sudo make install  Run the following to refresh the runtime shared after installing Thrift:  ::     sudo ldconfig /usr/local/lib  Running the Python Scoring Pipeline - Alternative Method ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  1.

..     .. figure:: images/download_scoring_pipeline.png       :alt:        :width: 314px       :height: 341px  2. Extract the scoring pipeline. You can run the scoring module and the scoring service after downloading and extracting the pipeline. **Score from a Python Program**  If you intend to score from a Python program, run the scoring module example. (Requires Linux and Python 3.8.) ::     export DRIVERLESS_AI_LICENSE_FILE="/path/to/license.sig"    bash run_example.sh  **Score Using a Web Service**  If you intend to score using a web service, run the HTTP scoring server example. (Requires Linux x86_64 and Python 3.8.) ::     export DRIVERLESS_AI_LICENSE_FILE="/path/to/license.sig"    bash run_http_server.sh    bash run_http_client.sh  **Score Using a Thrift Service**  If you intend to score using a Thrift service, run the TCP scoring server example. (Requires Linux x86_64, Python 3.8 and Thrift.)

The scripts can also leverage Conda (Anaconda/Mininconda) to create Conda virtual environment and install required package dependencies. The package manager to use is provided as an argument to the script. ::        # to use conda package manager       export DRIVERLESS_AI_LICENSE_FILE="/path/to/license.sig"       bash run_example.sh --pm conda        # to use pip package manager       export DRIVERLESS_AI_LICENSE_FILE="/path/to/license.sig"       bash run_example.sh --pm pip  If you experience errors while running any of the above scripts, check to make sure your system has a properly installed and configured Python 3.8 installation. Refer to the `Troubleshooting Python Environment Issues <#troubleshooting-python-environment-issues>`__ section that follows to see how to set up and test the scoring module using a cleanroom Ubuntu 16.04 virtual machine.

All the prerequisites for the scoring module to work correctly are listed in the requirements.txt file. To use the scoring module, all you have to do is create a Python virtualenv, install the prerequisites, and then import and use the scoring module as follows:  ::     # See 'example.py' for complete example. from scoring_487931_20170921174120_b4066 import Scorer    scorer = Scorer()       # Create instance. score = scorer.score([  # Call score()        7.416,              # sepal_len        3.562,              # sepal_wid        1.049,              # petal_len        2.388,              # petal_wid    ])  The scorer instance provides the following methods (and more):  -  score(list): Score one row (list of values). -  score_batch(df): Score a Pandas dataframe. -  fit_transform_batch(df): Transform a Pandas dataframe. -  get_target_labels(): Get target column labels (for classification    problems).

virtualenv -p python3.8 env    source env/bin/activate    pip install --use-deprecated=legacy-resolver -r requirements.txt    export DRIVERLESS_AI_LICENSE_FILE="/path/to/license.sig"    python example.py  .. _python-scoring-service:  The Scoring Service -------------------  The scoring service hosts the scoring module as an HTTP or TCP service. Doing this exposes all the functions of the scoring module through remote procedure calls (RPC). In effect, this mechanism lets you invoke scoring functions from languages other than Python on the same computer or from another computer on a shared network or on the Internet. The scoring service can be started in two ways:  -  In TCP mode, the scoring service provides high-performance RPC calls    via Apache Thrift (https://thrift.apache.org/) using a binary wire    protocol. -  In HTTP mode, the scoring service provides JSON-RPC 2.0 calls served    by Tornado (http://www.tornadoweb.org).

Scoring Service - TCP Mode (Thrift) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  The TCP mode lets you use the scoring service from any language supported by Thrift, including C, C++, C#, Cocoa, D, Dart, Delphi, Go, Haxe, Java, Node.js, Lua, perl, PHP, Python, Ruby and Smalltalk. To start the scoring service in TCP mode, you will need to generate the Thrift bindings once, then run the server:  ::     # See 'run_tcp_server.sh' for complete example. thrift --gen py scoring.thrift    python tcp_server.py --port=9090  Note that the Thrift compiler is only required at build-time. It is not a run time dependency, i.e. once the scoring services are built and tested, you do not need to repeat this installation process on the machines where the scoring services are intended to be deployed. To call the scoring service, generate the Thrift bindings for your language of choice, then make RPC calls via TCP sockets using Thrift's buffered transport in conjunction with its binary protocol.

thrift --gen py scoring.thrift     # See 'example_client.py' for complete example. socket = TSocket.TSocket('localhost', 9090)    transport = TTransport.TBufferedTransport(socket)    protocol = TBinaryProtocol.TBinaryProtocol(transport)    client = ScoringService.Client(protocol)    transport.open()    row = Row()    row.sepalLen = 7.416  # sepal_len    row.sepalWid = 3.562  # sepal_wid    row.petalLen = 1.049  # petal_len    row.petalWid = 2.388  # petal_wid    scores = client.score(row)    transport.close()  You can reproduce the exact same result from other languages, e.g.

This is usually less performant compared to Thrift, but has the advantage of being usable from any HTTP client library in your language of choice, without any dependency on Thrift. For JSON-RPC documentation, see http://www.jsonrpc.org/specification. To start the scoring service in HTTP mode:  ::     # See 'run_http_server.sh' for complete example. export DRIVERLESS_AI_LICENSE_FILE="/path/to/license.sig"    python http_server.py --port=9090  To invoke scoring methods, compose a JSON-RPC message and make a HTTP POST request to `http://host:port/rpc <http://host:port/rpc>`__ as follows:  ::     # See 'run_http_client.sh' for complete example. curl http://localhost:9090/rpc \      --header "Content-Type: application/json" \      --data @- <<EOF     {      "id": 1,      "method": "score",      "params": {        "row": [ 7.486, 3.277, 4.755, 2.354 ]      }     }    EOF  Similarly, you can use any HTTP client library to reproduce the above result.

The following example demonstrates how to retrieve Shapley contributions for transformed and original features when making predictions:  .. code:: python     from scoring_487931_20170921174120_b4066 import Scorer     # Transformed Features Shapley Values    scorer = Scorer()       # Create instance. score = scorer.score([  # Call score()        7.416,              # sepal_len        3.562,              # sepal_wid        1.049,              # petal_len        2.388,              # petal_wid    ], pred_contribs=True, pred_contribs_original=False)     # Original Features Shapley Values    scorer = Scorer()       # Create instance. score = scorer.score([  # Call score()        7.416,              # sepal_len        3.562,              # sepal_wid        1.049,              # petal_len        2.388,              # petal_wid    ], pred_contribs=True, pred_contribs_original=True)  .. note::     - Settingpred_contribs_original=Truerequires thatpred_contribsis also set toTrue.

For ensemble with ExtraTrees meta learner       (ensemble_meta_learner='extra_trees') models we suggest to use the       Python scoring packages. -  Shapley values for **original features** are approximated from the       accompanying Shapley values for transformed features with the       Naive Shapley (:ref:`even split <dai-shapley>`) method. -  The :ref:`Shapley fast approximation <completed_experiment>` uses       only one model (from the first fold) with no more than the first       50 trees. For details seefast_approx_num_treesandfast_approx_do_one_fold_one_model:ref:`config.toml settings <sample-configtoml>`. .. _python-scoring-faq:  Frequently asked questions --------------------------  **I'm getting GCC compile errors on Red Hat / CentOS when not using tar and**SCORING_PIPELINE_INSTALL_DEPENDENCIES
=
0. **How do I fix this?

**     If you ran an experiment when TensorFlow was enabled and then attempt    to run the Python Scoring Pipeline, you may receive a message similar    to the following:     ::        TensorFlow is disabled. To enable, export DRIVERLESS_AI_ENABLE_TENSORFLOW=1 or set enable_tensorflow=true in config.toml. To successfully run the Python Scoring Pipeline, you must enable theDRIVERLESS_AI_ENABLE_TENSORFLOW``
flag. For example:
      export DRIVERLESS_AI_LICENSE_FILE="/path/to/license.sig"
      DRIVERLESS_AI_ENABLE_TENSORFLOW=1 bash run_example.sh

TROUBLESHOOTING PYTHON ENVIRONMENT ISSUES

The following instructions describe how to set up a cleanroom Ubuntu
16.04 virtual machine to test that this scoring pipeline works
correctly. PREREQUISITES:
-   Install Virtualbox: sudo apt-get install virtualbox
-   Install Vagrant: https://www.vagrantup.com/downloads.html
1. Create configuration files for Vagrant.

UPDATING LICENSES

If your current Driverless AI license has expired, you will be required
to update it in order to continue running Driverless AI, in order to run
the scoring pipeline, in order to access deployed pipelines to AWS
Lambdas, etc. Updating the License for Driverless AI
Similar to adding a license for the first time, you can update your
license for running Driverless AI either by replacing your current
license.sig file or via the Web UI. Updating the license.sig File
Update the license key in your
/OPT/H2OAI/DAI/HOME/.DRIVERLESSAI/LICENSE.SIG file by replacing the
existing license with your new one. Updating the License in the Web UI
If your license is expired, the Web UI will prompt you to enter a new
one. The steps are the same as adding a license for the first time via
the Driverless AI Web UI. Updating the License for Scoring Pipelines
For the Python Scoring Pipeline, include the updated license file when
setting the environment variable in Python.

For the MOJO Scoring Pipeline, the updated license file can be specifed
using an environment variable, using a system property of JVM, or via an
application classpath. This is the same as adding a license for the
first time. Refer to the above mojo_scoring_license section for adding
licenses. Updating Driverless AI Licenses on AWS Lambda
Users can manually update each of their Driverless AI licenses deployed
in production on AWS Lambda. For users with many MOJOs in production,
though, H2O provides a script that will update Driverless AI licenses
for all of your MOJOs currently deployed on AWS Lambda. Manual Update
The Driverless AI deployment pipeline to AWS Lambdas explicitly sets the
license key as an environment variable. Replace the expired license key
with your updated one. []
Automatic Update
H2O provides a script that can be used to update Driverless AI licenses
for all of your MOJOs deployed on a specific AWS Lambda region. This
script can be run for any machine.

ADD CUSTOM RECIPES

Custom recipes are Python code snippets that can be uploaded into
Driverless AI at runtime like plugins. Restarting Driverless AI is not
required. If you do not have a custom recipe, you can select from a
number of recipes available in the Recipes for H2O Driverless AI
repository. For more information and examples, refer to custom-recipes. To add a custom recipe to Driverless AI, click ADD CUSTOM RECIPE and
select one of the following options:
-   FROM COMPUTER: Add a custom recipe as a Python or ZIP file from your
    local file system. -   FROM URL: Add a custom recipe from a URL. -   FROM BITBUCKET: Add a custom recipe from a Bitbucket repository. To
    use this option, your Bitbucket username and password must be
    provided along with the custom recipe Bitbucket URL. OFFICIAL RECIPES (OPEN SOURCE)

To access H2O's official recipes repository, click OFFICIAL RECIPES
(OPEN SOURCE).

make_mojo_scoring_pipeline = "off"is displayed in the TOML editor.  The TOML configuration editor lets you manually add, remove, or edit expert setting parameters. To confirm your changes, click **Save**. The experiment preview updates to reflect your specified configuration changes. For a full list of available settings, see :ref:`expert-settings`.  .. note::     Do not edit the section below the[recipe_activation]`` line. This

    section provides Driverless AI with information about which custom
    recipes can be used by the experiment. This is important for keeping
    experiments comparable when performing retrain / refit operations.

Note

For information on TOML, see https://toml.io/en/v0.4.0.

EXPERIMENT GRAPHS

This section describes the dashboard graphs that display for running and
completed experiments. These graphs are interactive. Hover over a point
on the graph for more details about the point. Binary Classification Experiments
For Binary Classification experiments, Driverless AI shows a ROC Curve,
a Precision-Recall graph, a Lift chart, a Kolmogorov-Smirnov chart, and
a Gains chart. []
-   ROC: This shows Receiver-Operator Characteristics curve stats on
    validation data along with the best Accuracy, MCC, and F1 values. An
    ROC curve is a useful tool because it only focuses on how well the
    model was able to distinguish between classes. Keep in mind, though,
    that for models where one of the classes happens rarely, a high AUC
    could provide a false sense that the model is correctly predicting
    the results. This is where the notion of precision and recall become
    important.

The area under this curve is called AUCPR. Prec-Recall is a
    complementary tool to ROC curves, especially when the dataset has a
    significant skew. The Prec-Recall curve plots the precision or
    positive predictive value (y-axis) versus sensitivity or true
    positive rate (x-axis) for every possible classification threshold. At a high level, you can think of precision as a measure of
    exactness or quality of the results and recall as a measure of
    completeness or quantity of the results obtained by the model. Prec-Recall measures the relevance of the results obtained by the
    model. -   LIFT: This chart shows lift stats on validation data. For example,
    "How many times more observations of the positive target class are
    in the top predicted 1%, 2%, 10%, etc. (cumulative) compared to
    selecting observations randomly?" By definition, the Lift at 100% is
    1.0.

Lift is a measure of the effectiveness of a
    predictive model calculated as the ratio between the results
    obtained with a model and with a random model(or no model). In other
    words, the ratio of gain % to the random expectation % at a given
    quantile. The random expectation of the xth quantile is x%. -   KOLMOGOROV-SMIRNOV: This chart measures the degree of separation
    between positives and negatives for validation or test data. -   GAINS: This shows Gains stats on validation data. For example, "What
    fraction of all observations of the positive target class are in the
    top predicted 1%, 2%, 10%, etc. (cumulative)?" By definition, the
    Gains at 100% are 1.0. Multiclass Classification Experiments
For multiclass classification experiments, a Confusion Matrix is
available in addition to the ROC Curve, Precision-Recall graph, Lift
chart, Kolmogorov-Smirnov chart, and Gains chart.

These graphs and charts (Confusion Matrix excepted)
are based on a method known as micro-averaging (reference:
http://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html#multiclass-settings). For example, you may want to predict the species in the iris data.

Driverless AI concatenates these 3 vectors together to compute
the charts. predicted = [0.9628, 0.0182, 0.0191, 0.021, 0.3172, 0.9534, 0.0158, 0.6646, 0.0276]
    actual = [1, 0, 0, 0, 1, 1, 0, 0, 0]
Multiclass Confusion Matrix
A confusion matrix shows experiment performance in terms of false
positives, false negatives, true positives, and true negatives. For each
threshold, the confusion matrix represents the balance between TPR and
FPR (ROC) or Precision and Recall (Prec-Recall). In general, most useful
operating points are in the top left corner. In this graph, the actual results display in the columns and the
predictions display in the rows; correct predictions are highlighted. In
the example below, Iris-setosa was predicted correctly 30 times, while
Iris-virginica was predicted correctly 32 times, and Iris-versicolor was
predicted as Iris-virginica 2 times (against the validation set).

A CM for all rows can't be
displayed since, in general, DAI isn't performing k-fold CV but could be
performing 2 repeats of 1/3 validation splits with overlaps. The "full"
k-fold CV is only performed for the final model, and this results in a
CM entry for each row, which is displayed when the experiment completes. If a test set was provided for the experiment, then click on the
VALIDATION METRICS button below the graph to view these stats on test
data. []

Regression Experiments
-   RESIDUALS: Residuals are the differences between observed responses
    and those predicted by a model. Any pattern in the residuals is
    evidence of an inadequate model or of irregularities in the data,
    such as outliers, and suggests how the model may be improved. This
    chart shows Residuals (Actual-Predicted) vs Predicted values on
    validation or test data. Note that this plot preserves all outliers. For a perfect model, residuals are zero. -   ACTUAL VS.

MACHINE LEARNING INTERPRETABILITY

interpreting interpret-the-mli-page.rst interpret-non-ts interpret-ts
interpret-recipes

VISUALIZING THE SCORING PIPELINE

A visualization of the scoring pipeline is available for each completed

NOTES:

-   This pipeline is best viewed in the latest version of Chrome.
-   A .png image of this pipeline is available in the AutoDoc <autodoc>
    and in the mojo.zip file ONLY with the Driverless AI Docker image.
    For tar, deb, and rpm installs, you must install Graphviz manually
    in order for the visualization pipeline to be included in the
    AutoDoc and mojo.zip.

Click the VISUALIZE SCORING PIPELINE (EXPERIMENTAL) button on the
completed experiment page to view the visualization.

[]

To view a visual representation of a specific model, click on the oval
that corresponds with that model.

[]

[]

To change the orientation of the visualization, click the TRANSPOSE
button in the bottom right corner of the screen.

[]

MLI FOR TIME-SERIES EXPERIMENTS

This section describes how to run MLI for time-series experiments. Refer
to interpret-regular for MLI information with regular experiments. There are two methods you can use for interpreting time-series models:
-   Using the MLI link in the top main menu on the upper right corner of
    the UI to interpret either a Driverless AI model or an external
    model. This process is described in the
    Interpreting a Driverless AI Model <interpret-dai-model> and
    Interpreting Predictions from an External Model <interpret-external-model>
    sections. -   Using the INTERPRET THIS MODEL button on a completed experiment page
    to interpret a Driverless AI model on original and transformed
    features. Run Interpretation from Completed Experiment page<from-exp-page>
    (See below.)

MLI
    migration is not supported for experiments from versions <= 1.8.9. -   MLI is not available for multiclass Time Series. -   When the test set contains actuals, you will see the time series
    metric plot and the group metrics table. If there are no actuals,
    MLI will run, but you will see only the prediction value time series
    and a Shapley table. -   MLI does not require an Internet connection to run on current
    models. Multi-Group Time Series MLI
This section describes how to run MLI on time series data for multiple
groups. 1. Click the INTERPRET THIS MODEL button on a completed time series
    experiment to launch Model Interpretation for that experiment. This
    page includes the following:
2. Scroll to the bottom of the panel and select a grouping in the GROUP
    SEARCH field to view a graph of Actual vs. Predicted values for the
    group. The outputted graph can be downloaded to your local machine. 3.

The Shapley values plot can also
    be downloaded to your local machine. 4. Click ADD PANEL to add a new MLI Time Series panel. This lets you
    compare different groups in the same model and also provides the
    flexibility to do a "side-by-side" comparison between different
    models. Single Time Series MLI
Time Series MLI can also be run when only one group is available. 1. Click the INTERPRET THIS MODEL button on a completed time series
2. Scroll to the bottom of the panel and select an option in the GROUP
    group. (Note that for Single Time Series MLI, there will only be one
    option in this field.) The outputted graph can be downloaded to your
    local machine. 3. Click on a prediction point in the plot (white line) to view Shapley
    values for that prediction point.

TIME SERIES IN DRIVERLESS AI

Time series forecasting is one of the most common and important tasks in
business analytics. There are many real-world applications like sales,
weather, stock market, and energy demand, just to name a few. At H2O, we
believe that automation can help our users deliver business value in a
timely manner. Therefore, we combined advanced time series analysis and
our Kaggle Grand Masters’ time series recipes into Driverless AI. The key features/recipes that make automation possible are:
-   Automatic handling of time groups (e.g., different stores and
    departments)
-   Robust time series validation
    -   Accounts for gaps and forecast horizon
    -   Uses past information only (i.e., no data leakage)
-   Time series-specific feature engineering recipes
    -   Date features like day of week, day of month, etc.

Note
Locale-dependent datetime formats may cause issues in Driverless AI. Converting datetime to a locale-independent format prior to running
experiments is recommended. For information on how to convert datetime
formats so that they are accepted in DAI, refer to the final note in the
modify_by_recipe section. Understanding Time Series
The following is an in depth description of time series in Driverless
AI. For an overview of best practices when running time series
experiments, see ts_bestpractices. Modeling Approach
Driverless AI uses GBMs, GLMs and neural networks with a focus on time
series-specific feature engineering.

The selection is still based on validation accuracy. In other
words, the same transformations/genes apply; plus there are new
transformations that come from time series. Some transformations (like
target encoding) are deactivated. When running a time series experiment, Driverless AI builds multiple
models by rolling the validation window back in time (and potentially
using less and less training data). User-Configurable Options
Gap
The guiding principle for properly modeling a time series forecasting
problem is to use the historical data in the model training dataset such
that it mimics the data/information environment at scoring time (i.e. deployed predictions). Specifically, you want to partition the training
set to account for: 1) the information available to the model when
making predictions and 2) the number of units out that the model should
be optimized to predict.

GAP is the amount of missing time bins between the end of a training set
and the start of test set (with regards to time). For example:
-   Assume there are daily data with days 1/1/2020, 2/1/2020, 3/1/2020,
    4/1/2020 in train. There are 4 days in total for training. -   In addition, the test data will start from 6/1/2020. There is only 1
    day in the test data. -   The previous day (5/1/2020) does not belong to the train data. It is
    a day that cannot be used for training (i.e because information from
    that day may not be available at scoring time). This day cannot be
    used to derive information (such as historical lags) for the test
    data either. -   Here the time bin (or time unit) is 1 day. This is the time interval
    that separates the different samples/rows in the data. -   In summary, there are 4 time bins/units for the train data and 1
    time bin/unit for the test data plus the Gap.

-   Gap = min(time bin test) - max(time bin train) - 1. -   In this case min(time bin test) is 6 (or 6/1/2020). This is the
    earliest (and only) day in the test data. -   max(time bin train) is 4 (or 4/1/2020). This is the latest (or the
    most recent) day in the train data. -   Therefore the GAP is 1 time bin (or 1 day in this case), because Gap
    = 6 - 4 - 1 or Gap = 1
[]
Forecast Horizon
It's often not possible to have the most recent data available when
applying a model (or it's costly to update the data table too often);
therefore some models need to be built accounting for a “future gap”. For example, if it takes a week to update a specific data table, you
ideally want to predict 7 days ahead with the data as it is “today”;
therefore a gap of 6 days is recommended. Not specifying a gap and
predicting 7 days ahead with the data as it is is unrealistic (and
cannot happen, as the data is updated on a weekly basis in this
example).

For example, if you want to know what will happen 7 days in the
future, then set the gap to 6 days. FORECAST HORIZON (or prediction length) is the period that the test data
spans for (for example, one day, one week, etc.). In other words it is
the future period that the model can make predictions for (or the number
of units out that the model should be optimized to predict). Forecast
horizon is used in feature selection and engineering and in model
selection. Note that forecast horizon might not equal the number of
predictions. The actual predictions are determined by the test dataset. []
The periodicity of updating the data may require model predictions to
account for significant time in the future. In an ideal world where data
can be updated very quickly, predictions can always be made having the
most recent data available.

However
this is not always the case, and a model needs to be able to make
predictions that span deep into the future because it may be too costly
to make predictions every single day after the data gets updated. In addition, each future data point is not the same. For example,
predicting tomorrow with today’s data is easier than predicting 2 days
ahead with today’s data. Hence specifying the forecast horizon can
facilitate building models that optimize prediction accuracy for these
future time intervals. Prediction Intervals
For regression problems, enable the compute-intervals expert setting to
have Driverless AI provide two additional columns _y.lower_ and
_y.upper_ in the prediction frame. The true target value _y_ for a
predicted sample is expected to lie within [_y.lower_, _y.upper_] with a
certain probability. The default value for this confidence level can be
specified with the confidence-level expert setting, which has a default
value of 0.9.

num_prediction_periods``) needs to be in periods, and the size is
unknown. To overcome this, you can use the optional

start_experiment_sync(in Python) ortrain(in R). This is used to specify the forecast horizon in real time units (as well as for gap.) If this parameter is not specified, then Driverless AI will automatically detect the period size in the experiment, and the forecast horizon value will respect this period. I.e., if you are sure that your data has a 1 week period, you can saynum_prediction_periods=14``;
otherwise it is possible that the model will not work correctly. GROUPS

Groups are categorical columns in the data that can significantly help
predict the target variable in time series problems. For example, one
may need to predict sales given information about stores and products. Being able to identify that the combination of store and products can
lead to very different sales is key for predicting the target variable,
as a big store or a popular product will have higher sales than a small
store and/or with unpopular products.

[]

LAG

The primary generated time series features are lag features, which are a
variable's past values. At a given sample with time stamp t, features at
some time difference T (lag) in the past are considered. For example, if
the sales today are 300, and sales of yesterday are 250, then the lag of
one day for sales is 250. Lags can be created on any feature as well as
on the target. []
As previously noted, the training dataset is appropriately split such
that the amount of validation data samples equals that of the testing
dataset samples. If we want to determine valid lags, we must consider
what happens when we will evaluate our model on the testing dataset. Essentially, the minimum lag size must be greater than the gap size. Aside from the minimum useable lag, Driverless AI attempts to discover
predictive lag sizes based on auto-correlation.

config.tomlfile. For more information, see :ref:`config_usage`. **Note:** Driverless AI does not attempt time series target transformations automatically; they must be set manually. :ref:`ts-target-transformation` (ts_lag_target_trafo): With this target transformation, you can select between the difference and ratio of the current and a lagged target. You can specify the corresponding lag size with the **Lag size used for time series target transformation** (ts_target_trafo_lag_size) setting. **Note:** This target transformation can be used together with the **Time series centering or detrending transformation** (ts_target_trafo) target transformation, but it is mutually exclusive with regular target transformations. :ref:`centering-detrending` (ts_target_trafo): With this target transformation, the free parameters of the trend model are fitted. The trend is removed from the target signal, and the pipeline is fitted on the residuals.

Select one of the following options:  -none-centering
(fast)-centering (robust)-linear (fast)-linear
(robust)-logistic-epidemic**Notes:**  -  This target transformation can be used together with the **Time    series lag-based target transformation** (ts_lag_target_trafo),    but it is mutually exclusive with regular target transformations. -  Thecentering
(robust)andlinear
(robust)detrending    variants use scikit-learn's implementation of random sample consensus    (RANSAC) to achieve a higher tolerance with regard to outliers. As    stated on scikit-learn's `page on robust linear model estimation    using    RANSAC <https://scikit-learn.org/stable/auto_examples/linear_model/plot_ransac.html>`__,    "The ordinary linear regressor is sensitive to outliers, and the    fitted line can easily be skewed away from the true underlying    relationship of data.

It may include various lags values, for example lags [1-30] or lags [20-40] or lags [7-70 by 7]. Exponential Weighting ^^^^^^^^^^^^^^^^^^^^^  Exponential weighting is a form of weighted moving average where more recent values have higher weight than less recent values. That weight is exponentially decreased over time based on an **alpha** (a) (hyper) parameter (0,1), which is normally within the range of [0.9 - 0.99]. For example:  -  Exponential Weight = a**(time) -  If sales 1 day ago = 3.0 and 2 days ago =4.5 and a=0.95: -  Exp. smooth = 3.0*(0.95\*\ *1) + 4.5*\ (0.95\**2) / ((0.951) +    (0.95*\ \*2)) =3.73 approx. Rolling-Window-Based Predictions --------------------------------  Driverless AI supports rolling-window-based predictions for time series experiments with two options: `Test Time Augmentation <https://github.com/h2oai/driverlessai-tutorials/tree/master/driverlessai_experiments/timeseries/ts-full-pipeline>`__ (TTA) or re-fit.

TTA simulates the process where the model stays the same but the features are refreshed using newly available data. Re-fit simulates the process of re-fitting the entire pipeline (including the model) once new data is available. This process is automated when the test set spans for a longer period than the forecast horizon and if the target values of the test set are known. If the user scores a test set that meets these conditions after the experiment is finished, rolling predictions with TTA will be applied. Re-fit, on the other hand, is only applicable for test sets provided during an experiment. TTA is the default option and can be changed with the `Method to Create Rolling Test Set Predictions <expert-settings.html#method-to-create-rolling-test-set-predictions>`__ expert setting.

:math:`N_{TEST} = H`). You want to have enough training data time periods :math:`N_{TRAIN}` to score well on the testing dataset. At a minimum, the training dataset should contain at least three times as many time periods as the testing dataset (i.e. :math:`N_{TRAIN} >= 3 × N_{TEST}`). This allows for the training dataset to be split into a validation set with the same amount of time periods as the testing dataset while maintaining enough historical data for feature engineering. .. _time-series-use-case:  Time Series Use Case: Sales Forecasting ---------------------------------------  Below is a typical example of sales forecasting based on the `Walmart competition on Kaggle <https://www.kaggle.com/c/walmart-recruiting-store-sales-forecasting>`__.

In this example, the goal is to forecast the next week of sales, so all of the attributes included in the data must be known at least one week in advance. In this case, you can assume that you will know whether or not a Store and Department will be running a promotional markdown. Features like the temperature of the Week are not used because that information is not available at the time of scoring. Once you have your data prepared in tabular format (see raw data above), Driverless AI can formulate it for machine learning and sort out the rest. If this is your very first session, the Driverless AI assistant will guide you through the journey. .. figure:: images/first_time_user.png    :alt:   Similar to previous Driverless AI examples, you need to select the dataset for training/test and define the target. For time series, you need to define the time column (by choosing AUTO or selecting the date column manually).

.. figure:: images/time_series_experiment_settings.png    :alt:   If you prefer to use automatic handling of time groups, you can leave the setting for time groups columns as AUTO, or you can define specific time groups. You can also specify the columns that will be unavailable at prediction time (see :ref:`ucapt` below for more information), the forecast horizon (in weeks), and the gap (in weeks) between the train and test periods. Once the experiment is finished, you can make new predictions and download the scoring pipeline just like any other Driverless AI experiments. .. _ucapt:  More About Unavailable Columns at Time of Prediction ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  The **Unavailable Columns at Prediction Time** (UCAPT) option is a way to mark features that will not be available in the test dataset or at the time of prediction but might still be predictive when looking at historical values.

For example, if we were predicting the sales amount each day, we might have the number of customers each day as a feature in our training dataset. In the future, we won't know how many customers will be coming into the store, so this would be a leaky feature to use. However, the average number of customers last week might be predictive and is something that we could calculate ahead of time. So in this case, looking at the historical values would be better than just dropping the feature. The default value for this setting is often--, meaning that all features can be used as they are. If you include a test dataset before selecting a time column, and that test dataset is missing any columns, then you will see a number as the default for **Unavailable Columns at Prediction Time**, which will be the number of columns that are in the training dataset but not the testing dataset.

Using a Driverless AI Time Series Model to Forecast ---------------------------------------------------  When you set the experiment's forecast horizon, you are telling the Driverless AI experiment the dates this model will be asked to forecast for. In the Walmart Sales example, we set the Driverless AI forecast horizon to 1 (1 week in the future). This means that Driverless AI expects this model to be used to forecast 1 week after training ends. Because the training data ends on 2020-10-26, this model should be used to score for the week of 2020-11-02. What should the user do once the 2020-11-02 week has passed? There are two options:  -  Option 1: Trigger a Driverless AI experiment to be trained once the    forecast horizon ends. A Driverless AI experiment will need to be    re-trained every week. -  Option 2: Use **Test Time Augmentation** (TTA) to update historical    features so that we can use the same model to forecast outside of the    forecast horizon.

In our Walmart Sales Forecasting example, a feature that may be very important is the Weekly Sales from the previous week. Once we move outside of the forecast horizon, our model no longer knows the Weekly Sales from the previous week. By performing TTA, Driverless AI will automatically generate these historical features if new data is provided. In Option 1, we would launch a new Driverless AI experiment every week with the latest data and use the resulting model to forecast the next week. In Option 2, we would continue using the same Driverless AI experiment outside of the forecast horizon by using TTA. Both options have their advantages and disadvantages. By retraining an experiment with the latest data, Driverless AI has the ability to possibly improve the model by changing the features used, choosing a different algorithm, and/or selecting different parameters.

There may be clear advantages for retraining an experiment after each forecast horizon or for using TTA. Refer to `this example <https://github.com/h2oai/driverlessai-tutorials/tree/master/driverlessai_experiments/timeseries/ts-full-pipeline>`__ to see how to use the scoring pipeline to predict future data instead of using the prediction endpoint on the Driverless AI server. Using TTA to continue using the same experiment over a longer period of time means there is no longer any need to continually repeat a model review process. However, it is possible for the model to become out of date.

The following notebook shows how to perform both methods and compare their performance: `Time Series Model Rolling Window <https://github.com/h2oai/driverlessai-tutorials/blob/master/driverlessai_experiments/timeseries/walmart_timeseries_experiment/timeseries_model_rollingwindow.ipynb>`__. **Notes**:  -  Scorers cannot refit or retrain a model. -  To specify a method for creating rolling test set predictions, use    :ref:`this expert setting <rolling-test-set-method>`. Note that    refitting performed with this expert setting is only applied to the    test set that is provided by the user during an experiment. The final    scoring pipeline always uses TTA. Triggering Test Time Augmentation ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  To perform Test Time Augmentation, create your forecast data to include any data that occurred after the training data ended up to the dates you want a forecast for.

-  If the date being forecasted contains any non-missing value in the    target column, then TTA is not triggered for that row. Forecasting Future Dates ~~~~~~~~~~~~~~~~~~~~~~~~  To forecast or predict future dates, upload a dataset that contains the future dates of interest and provide additional information such as group IDs or features known in the future. The dataset can then be used to run and score your predictions.

This panel is available from within the experiment page right above the Scorer knob. Refer to :ref:`time-series-settings` for more information on these options. .. figure:: images/time_series_expert_settings.png    :alt:   Time-series leaderboard mode ----------------------------  You can control the time-series :ref:`leaderboard <leaderboard>` with thetime_series_leaderboard_modeTOML config. Specify one of the following options:  -diverse: Explore a diverse set of models built using various    expert settings. Note that you can rerun another diverse leaderboard    on top of the best-performing model(s). -sliding_window: If the forecast horizon is *N* periods, create a    separate model for each of the (gap, horizon) pairs of (0,n), (n,n),    (2*n,n), ..., (2*N-1, n) in units of time periods. The number of    periods to predict per model *n* is controlled by the expert settingtime_series_leaderboard_periods_per_model``,
which defaults to 1.

USING THE STANDALONE PYTHON SCORING PIPELINE IN A DIFFERENT DOCKER CONTAINER

The Standalone Python Scoring Pipeline runs inside of the Driverless AI
Docker container. This is the recommended method for running the Python
Scoring Pipeline. If necessary, though, this pipeline can also be run
inside of a different Docker container. The following steps describe how
to do this. This setup assumes that you have a valid Driverless AI
license key, which will be required during setup. It also assumes that
you have completed a Driverless AI experiment and downloaded the Scoring
Pipeline. 1. On the machine where you want to run the Python Scoring Pipeline,
    create a new directory for Driverless AI (for example, DAI-NNN.) 2. Download the TAR SH version of Driverless AI from
    https://www.h2o.ai/download/ (for either Linux or IBM Power). 3. Use bash to execute the download and unpack it into the new
    Driverless AI folder. 4. Change directories into the new Driverless AI folder. 5.

DATASETS IN DRIVERLESS AI

The DATASETS OVERVIEW page is the Driverless AI home page. It displays
the datasets that have been imported into Driverless AI. Data Connectors
can be used to connect to various data sources.

datasets-import datasets-options datasets-download datasets-modify
datasets-join-wizard datasets-split

[]

UNDERSTANDING THE MODEL INTERPRETATION PAGE

This page describes the various interpretations available from the
Machine Learning Interpretability (MLI) explanations page for
non-time-series experiments. The explanations page is organized into four tabs:
  -   Summary Tab <summary-tab>
  -   Interpretations Using Driverless AI Model - DAI Model Tab <dai-tab>
  -   Interpretations Using Surrogate Model - Surrogate Model Tab <surrogate-tab>
  -   Interpretations Using NLP Dataset - NLP Tab <nlp-tab> (Only
      visible for NLP problems)
The mli-dashboard button reveals a dashboard with an overview of the
interpretations built using surrogate models. The
Actions button <mli-action> on the MLI page can be used to download
REASON CODES, SCORING PIPELINES for productionization, and MLI LOGS. The task bar <mli-task-bar> lists the status and logs of MLI
explainers <mli_default_recipes>.

Note that some of these actions may
not be available for certain explainers. -   DISPLAY LOGS: View and download logs for the explainer. -   DETAILS: View and download details about the explainer. -   DOWNLOAD: Download an image of the current view on the explainer
    page. -   VIDEO TUTORIAL: You can access video tutorials through the UI for
    the following explainers by clicking the play arrow icon. -   Partial Dependence (PDP)
      -   Shapley for transformed features
      -   Shapley for original features
      -   Shapley summary plot
      -   Disparate Impact Analysis (DIA)
      -   Sensitivity Analysis
      -   Surrogate decision tree
-   HELP: View a description of the explainer. -   CLOSE: Exit to the interpretation page. MLI NOTIFICATION CENTER
You can view a list of recent notifications from the interpretation page
and explainer detail pages by clicking the NOTIFICATIONS button.

The notification center displays general notifications
as well as notifications raised by specific explainers. Summary Tab
The Summary tab provides an overview of the interpretation, including
the dataset and Driverless AI experiment name (if available) that were
used for the interpretation along with the feature space (original or
transformed), target column, problem type, and k-Lime information. If
the interpretation was created from a Driverless AI model, then a table
with the Driverless AI model summary is also included along with the top
variables for the model. []

Interpretations Using Driverless AI Model (DAI Model Tab)
The DAI Model tab is organized into tiles for each interpretation
method. To view a specific plot, click the tile for the plot that you
want to view.

For multiclass classification
experiments, this tab includes Feature Importance and Shapley plots for
original and transformed features. The following is a list of the interpretation plots available from the
Driverless AI Model tab:
  -   Feature Importance (Original and Transformed Features) <dai-feature-imp>
  -   Shapley (Original and Transformed Features) <dai-shapley>
  -   Shapley Summary Plot (Original Features) <dai-shapley-summary>
  -   Partial Dependence (PDP) and Individual Conditional Expectation (ICE) <pdp-ice>
  -   Disparate Impact Analysis <dai-dia>
  -   Time Series Explainer <dai-time-series>
  -   Sensitivity Analysis <dai-sa>
  -   NLP LOCO <dai-nlp-loco>
  -   Permutation Feature Importance <dai-permutation-feature-importance>
[]
NOTES:
  -   When running DIA on an external model, the DAI MODEL tab on the
      the interpretation page is renamed to 3RD PARTY MODEL. -   Shapley plots are not supported for RuleFit, FTRL, and TensorFlow
      models.

To enable the
      calculations using KERNEL EXPLAINER method, enable Original Kernel
      SHAP explainer in recipes <mli_default_recipes>. -   Shapley plots are only supported for those BYOR (custom) models
      that implement the has_pred_contribs method (and return True) and
      implement proper handling of the argument pred_contribs=True in
      the predict method. -   The Permutation-based feature importance plot is only available
      when the autodoc_include_permutation_feature_importance
      configuration option is enabled when starting Driverless AI or
      when starting the MLI experiment (enable AutoDoc from the recipe
      tab and include_permutation_feature_importance from MLI AutoDoc
      expert settings when launching the MLI job).

-   NumToCatTE is the transformation type. -   BILL_AMT1:EDUCATION:MARRIAGE:SEX represent original features
            used. -   0 represents the likelihood encoding for target[0] after
            grouping by features (shown here as BILL_AMT1, EDUCATION,
            MARRIAGE and SEX) and making out-of-fold estimates. For
            multiclass experiments, this value is > 0. For binary
            experiments, this value is always 0. Interpretations Using Surrogate Model (Surrogate Model Tab)
A surrogate model is a data mining and engineering technique in which a
generally simpler model is used to explain another, usually more
complex, model or phenomenon. For example, the decision tree surrogate
model is trained to predict the predictions of the more complex
Driverless AI model using the original model inputs.

The Surrogate Model tab is organized into tiles for each interpretation
want to view. For binary classification and regression experiments, this
tab includes K-LIME/LIME-SUP and Decision Tree plots as well as Feature
Importance, Partial Dependence, and LOCO plots for the Random Forest
surrogate model. For more information on these plots, see
surrogate-model-plots. The following is a list of the interpretation plots from Surrogate
Models:
  -   K-LIME and LIME-SUP <klime-LimeSup>
  -   Random Forest Feature Importance <rf-feature-importance>
  -   Random Forest Partial Dependence and Individual Conditional Expectation <rf-pdp-ice>
  -   Random Forest LOCO <rf-loco>
  -   Decision Tree <decision-tree>
  -   NLP Surrogate <nlp-surrogate>
[]
NOTE: For multiclass classification experiments, only the Decision Tree
and Random Forest Feature Importance plots are available in this tab.

To
view a specific plot, click the tile for the plot that you want to view
The following is a list of the interpretation plots available from the
NLP tab:
  -   dai-nlp-loco
  -   mli-nlp-pdp
  -   mli-nlp-tokens
  -   mli-nlp-vlm
[]

Surrogate Models Dashboard
To view a dashboard with an overview of the interpretations built using
surrogate models, click the SURROGATE MODELS DASHBOARD button. It is
located in the upper-right corner of the interpretations page. []
For binary classification and regression experiments, the Surrogate
Models Dashboard page provides a single page with the following
surrogate plots. Note that the PDP and Feature Importance plots on this
page are based on the Random Forest surrogate model. -   Global Interpretable Model Explanations
  -   Feature Importance
  -   Decision Tree
  -   Partial Dependence
You can also view explanations from this page by clicking the
EXPLANATIONS button located in the upper-right corner.

NOTE: The Surrogate Models Dashboard is only available for binary
classification and regression experiments. []

Actions Button
The Actions button can be used to download REASON CODES, SCORING
PIPELINES for productionization, and LOGS. Click this button to view the
following options:
  -   MLI DOCS: View the Machine Learning Interpretability section of
      the Driverless AI documentation. -   DISPLAY MLI JAVA LOGS: View MLI Java logs for the interpretation. -   DISPLAY MLI PYTHON LOGS: View MLI Python logs for the
      interpretation. -   EXPERIMENT: View the experiment that was used to generate the
      interpretation. -   DOWNLOAD MLI LOGS: Download a ZIP file of the logs that were
      generated during the interpretation. -   PYTHON SCORING PIPELINE: For binomial and regression experiments,
      download the Python scoring pipeline for the interpretation. This
      option is not available for multiclass experiments.

For more info, see klime-mojo. -   DOWNLOAD FORMATTED TRANSFORMED SHAPLEY REASON CODES: For
      regression, binary, and multiclass experiments, download a CSV
      file of formatted Shapley reason codes on transformed data. -   DOWNLOAD FORMATTED LIME REASON CODES: For binomial experiments,
      download a CSV file of formatted LIME reason codes. -   DOWNLOAD LIME REASON CODES: For binomial experiments, download a
      CSV file of LIME reason codes. -   DOWNLOAD FORMATTED ORIGINAL SHAPLEY REASON CODES (NAIVE SHAPLEY):
      For regression, binary, and multiclass experiments, download a CSV
      file of formatted Shapley reason codes for original data. []

The Task Bar
The MLI explainers tasks and logs for non-time-series experiments can be
viewed from the status manager:
[]

DAI Model Plots
This section describes the plots that are available in the DAI Model
Tab.

This plot shows the Driverless AI feature importance. Driverless AI
feature importance is a measure of the contribution of an input variable
to the overall predictions of the Driverless AI model. []
Shapley (Original and Transformed Features)
This plot is not available for RuleFit or TensorFlow models. For all
other models, this plot is available for binary classification,
multiclass classification, and regression experiments. Shapley explanations are a technique with credible theoretical support
that presents consistent global and local variable contributions. Local
numeric Shapley values are calculated by tracing single rows of data
through a trained tree ensemble and aggregating the contribution of each
input variable as the row of data moves through the trained ensemble. For regression tasks, Shapley values sum to the prediction of the
Driverless AI model.

Global Shapley values are the average of the absolute Shapley
values over every row of a dataset. Shapley values for TRANSFORMED FEATURES are calculated by tracing
individual rows of data through a trained tree ensemble and then
aggregating the contribution of each input variable as the row of data
moves through the trained ensemble. More information about Shapley for
tree-based models is available at https://arxiv.org/abs/1706.06060. Driverless AI directly calls XGBoost and LightGBM SHAP functions to get
contributions for transformed features. Shapley values for ORIGINAL FEATURES are approximated from the
accompanying Shapley values for transformed features with the NAIVE
SHAPLEY (even split) method. This method makes the assumption that input
features to a transformer are independent and splits the contribution
evenly amongst the contributing features.

incomewill be 2.5 each. For ensembles, Shapley values (in the link space) are blended as per the model weights in the ensemble. Driverless AI :ref:`MOJO <quick-run>` for productionization supports Naive Shapley (even split) approach for original features. Shapley values for original features can also be calculated with the **Kernel Explainer** method, which uses a special weighted linear regression to compute the importance of each feature. This can be enabled by using the :ref:`recipe <mli_default_recipes>` Original Kernel SHAP explainer. More information about Kernel SHAP is available at http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf.

If there are a significant amount of features, they will be organized in numbered pages that can be viewed individually. **Note**: The provided original values are approximations derived from the accompanying transformed values. For example, if the transformed feature :math:`feature1\_feature2` has a value of 0.5, then the value of the original features (:math:`feature1` and :math:`feature2`) will be 0.25. .. _dai-shapley-summary:  Shapley Summary Plot (Original Features) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  The Shapley Summary Plot shows original features versus their local Shapley values on a sample of the dataset. Feature values are binned by Shapley values, and the average normalized feature value for each bin is plotted. To see the Shapley value, number of rows, and average normalized feature value for a particular feature bin, hold the pointer over the bin. The legend corresponds to numeric features and maps to their normalized value.

You can click on numeric features to see a scatter plot of the actual feature values versus their corresponding Shapley values. Categorical features are shown in grey and do not provide an actual-value scatter plot. .. raw:: html     <img src="_static/shapley_summary_plot.gif" alt="Shapley Summary Plot" data-linktype="relative-path">  **Notes**:  -  The Shapley Summary Plot only shows original features that are used    in the Driverless AI model. -  The dataset sample size and the number of bins can be updated in the    Interpretation Expert Settings. -  For a list of Shapley Summary Plot explainer expert settings, see    :ref:`interpretation-expert-settings-shapley`. .. _pdp-ice:  Partial Dependence (PDP) and Individual Conditional Expectation (ICE) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  A Partial Dependence and ICE plot is available for both Driverless AI and surrogate models.

Partial dependence plots display how machine-learned response functions change based on the values of an input variable of interest while taking nonlinearity into consideration and averaging out the effects of all other input variables. Partial dependence plots are described in the Elements of Statistical Learning (Hastie et al, 2001). Partial dependence plots enable increased transparency in Driverless AI models and the ability to validate and debug Driverless AI models by comparing a variable's average predictions across its domain to known standards, domain knowledge, and reasonable expectations. The ICE Technique ^^^^^^^^^^^^^^^^^  This plot is available for binary classification and regression models. A newer adaptation of partial dependence plots called Individual conditional expectation (ICE) plots can be used to create more localized explanations for a single individual by using the same basic ideas as partial dependence plots.

ICE values are disaggregated partial dependence, but ICE is also a type of nonlinear sensitivity analysis in which the model predictions for a single row are measured while a variable of interest is varied over its domain. ICE plots enable a user to determine whether the model's treatment of an individual row of data is outside one standard deviation from the average model behavior, whether the treatment of a specific row is valid in comparison to average model behavior, known standards, domain knowledge, and reasonable expectations, and how a model will behave in hypothetical situations where one variable in a selected row is varied across its domain.

Large differences in partial dependence and ICE are an indication that strong variable interactions may be present. In this case partial dependence plots may be misleading because average model behavior may not accurately reflect local behavior. .. _partial-dependence-plot:  Partial Dependence Plot (PDP) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  This plot is available for binary classification and regression models. Overlaying ICE plots onto partial dependence plots allow the comparison of the Driverless AI model's treatment of certain examples or individuals to the model's average predictions over the domain of an input variable of interest. This plot shows the partial dependence when a variable is selected and the ICE values when a specific row is selected. Users may select a point on the graph to see the specific value at that point. You can also focus the PDP plot on a specific subset of data by using the slider in the middle of the screen.

ICE (grey) displays the prediction behavior for an individual row of data when an input variable is toggled across its domain. Currently, partial dependence and ICE plots are only available for the top ten most important original input variables. Categorical variables with 20 or more unique values are never included in these plots. .. figure:: images/mli-pdp.png    :alt:   **Notes**:  -  To use dynamic switching between PDP numeric and categorical binning    and UI chart selection in cases where features were used both as    numeric and categorical by the experiment, enable themli_pd_numcat_num_chart:ref:`config.toml <config_file>` setting. (This setting is enabled by default.) When this setting is enabled,    you can specify the threshold for PDP binning and chart selection    with themli_pd_numcat_thresholdsetting, which defaults to 11.

.. _pdp-on-demand:  Partial Dependence On-Demand ''''''''''''''''''''''''''''  The PD on-demand option lets you choose an existing or new feature for which to compute Driverless AI PD/ICE from an existing interpretation. With this method, PD/ICE is calculated by an ad hoc explainer, then run and merged to the original DAI PD/ICE representation. To use the PD on-demand option, click the interpretation you want to use, then click **DAI Partial Dependence Plot** from the **DAI Model** tab. On the PD plot page, click the **Add Feature** button and select the feature(s) you want to calculate PD for. Click **Done** to confirm your selection. A notification appears at the bottom of the screen once Driverless AI has finished the on-demand computation. To view the computed PD values for a particular feature, click **Feature** on the PD plot page, then select the feature you want to view PD values for.

DIA is a technique that is used to evaluate fairness. Bias can be introduced to models during the process of collecting, processing, and labeling data—as a result, it is important to determine whether a model is harming certain users by making a significant number of biased decisions. DIA typically works by comparing aggregate measurements of unprivileged groups to a privileged group. For instance, the proportion of the unprivileged group that receives the potentially harmful outcome is divided by the proportion of the privileged group that receives the same outcome—the resulting proportion is then used to determine whether the model is biased. Refer to the **Summary** section to determine if a categorical level (for example, Fairness Female) is fair in comparison to the specified reference level and user-defined thresholds. **Fairness All** is a true or false value that is only true if every category is fair in comparison to the reference level.

The average group metrics reported in most cases by DIA may miss cases of local discrimination, especially with complex, unconstrained models that can treat individuals very differently based on small changes in their data attributes. DIA lets you specify a **disparate impact variable** (the group variable that is analyzed), a **reference level** (the group level that other groups are compared to), and **user-defined thresholds for disparity**. Several tables are provided as part of the analysis:  -  **Group metrics**: The aggregated metrics calculated per group. For    example, true positive rates per group. -  **Group disparity**: This is calculated by dividing themetric_for_groupby thereference_group_metric. Disparity is    observed if this value falls outside of the user-defined thresholds. -  **Group parity**: This builds on Group disparity by converting the    above calculation to a true or false value by applying the    user-defined thresholds to the disparity values.

These thresholds will generally detect if the model is (on average) treating the non-reference group 20% more or less favorably than the reference group. Users are encouraged to set the user-defined thresholds to align with their organization's guidance on fairness thresholds. Run DIA on external datasets ^^^^^^^^^^^^^^^^^^^^^^^^^^^^  You can run DIA on a dataset that has predictions from an external source instead of getting predictions within Driverless AI. This process is similar to running MLI on an external dataset. Note that when running DIA on an external model, the **DAI Model** tab on the the interpretation page is renamed to **3rd Party Model**. 1. In the main navigation, click **MLI**. The **Interpreted Models**    page is displayed. 2. Click the **New Interpretation** button, and then click **New    Interpretation** from the list of available options. 3.

4. In the **Interpretation Settings** section, click **Recipes**. Click    the **Uncheck all** button, and then select only **Disparate Impact    Analysis**. To confirm your selection, click **Done**. .. figure:: images/dia-external-select-recipe.png    :alt:   5. In the **Interpretation Target** section, click **Select target    column**, and then specify the target column. 6. In the **Interpretation Target** section, click **Select prediction    column**, and then specify the prediction column. 7. Click the **Launch MLI** button. .. figure:: images/dia-external-launch.png    :alt:   Metrics - Binary Classification ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  The following are formulas for error metrics and parity checks utilized by binary DIA.

-  **ME** is the difference between the percent of the control group    members receiving a favorable outcome and the percent of the    protected class members receiving a favorable outcome:     .. math:: \text{ME} \equiv 100 \cdot (\text{PR} (\hat{y} = 1 \vert X_c = 1) - \text{Pr}(\hat{y} = 1 \vert X_p = 1))  ..     Where:     -  :math:`\hat{y}` is the model decisions. -  :math:`X_c` and :math:`X_p` are binary markers created from some       demographic attribute. -  :math:`c` is the control group. -  :math:`p` is the protected group. -  :math:`Pr(\cdot)` is the operator for conditional probability. -  **AIR** is equal to the ratio of the proportion of the protected    class that receives a favorable outcome and the proportion of the    control class that receives a favorable outcome:     .. math:: \text{AIR} \equiv \frac{Pr(\hat{y} \; = 1 \vert X_p = 1)}{Pr(\hat{y} \; = 1 \vert X_c = 1)}  ..     Where:     -  :math:`\hat{y}` is the model decisions.

-  :math:`c` is the control group. -  :math:`p` is the protected group. -  :math:`Pr(·)` is the operator for conditional probability. -  **SMD** is used to assess disparities in continuous features such as    income differences in employment analyses or interest rate    differences in lending:     .. math:: \text{SMD} \equiv \frac{\bar{\hat y_p} - \bar{\hat y_c}}{\sigma_{\hat y}}  ..     Where:     -  :math:`\bar{\hat y_p}` is the difference in the average protected       class outcome. -  :math:`\bar{\hat y_c}` is the control class outcome. -  :math:`\sigma_{\hat y}` is a measure of the standard deviation of       the population. .. note::     - For more information on how DIA is implemented in Driverless AI,    see    https://www.frontiersin.org/articles/10.3389/frai.2021.695301/full. -    Although the process of DIA is the same for both classification and    regression experiments, the returned information is dependent on the    type of experiment being interpreted.

-    Users are encouraged to consider the explanation dashboard to    understand and augment results from disparate impact analysis. In    addition to its established use as a fairness tool, users may want to    consider disparate impact for broader model debugging purposes. For    example, users can analyze the supplied confusion matrices and group    metrics for important, non-demographic features in the Driverless AI    model. - For a list of DIA Summary Plot explainer expert settings,    see :ref:`interpretation-expert-settings-dia`. - The mean prediction    disparity is the average prediction for the group being considered    divided by the average prediction for the reference group. - For more    information on group disparity and parity, refer to    https://h2oai.github.io/tutorials/disparate-impact-analysis/#5.

For example, if your DAI    experiment was optimized for root-mean-square error (RMSE), then an    RMSE graph is made available on this page. Note that you can use the    accompanying slider to view a specific range of dates. .. raw:: html        <img src="_static/interpret-time-series-slider.gif" alt="Using the accompanying slider to view a specific range of dates" data-linktype="relative-path">  -  **Actual vs. Predicted:** View a graph that contrasts actual and    predicted values. Note that this graph also features an accompanying    slider that you can use to view a specific range of dates. In addition to the preceding graphs, the following additional information is provided:  -  **Group metrics:** Grouped metrics are based on an aggregation by    group. For example, aggregate by store and department and get counts    per group. You can also get the metric of interest, for example    aggregate RMSE, etc.

-  **Shapley values:** Based on the selected date, Shapley values for    each feature are provided in this section. To view Value + Bias for    each feature and definitions of the transformed feature, click the    **Details** button. You can also download Shapley values for the    selected date by clicking the download button. Note that you can select a specific group and / or date by clicking **Group** or **Date**. .. figure:: images/interpret-time-series.png    :alt:   .. _dai-sa:  Sensitivity Analysis (SA) ~~~~~~~~~~~~~~~~~~~~~~~~~  Overview ^^^^^^^^  **Note**: Sensitivity Analysis (SA) is only available for binary classification and regression experiments. Sensitivity Analysis (or "What if?") is a simple and powerful model debugging, explanation, fairness, and security tool.

Beyond traditional assessment practices, sensitivity analysis of machine learning model predictions is perhaps the most important validation technique for machine learning models. Sensitivity analysis investigates whether model behavior and outputs remain stable when data is intentionally perturbed or other changes are simulated in the data. Machine learning models can make drastically differing predictions for only minor changes in input variable values. For example, when looking at predictions that determine financial decisions, SA can be used to help you understand the impact of changing the most important input variables and the impact of changing socially sensitive variables (such as Sex, Age, Race, etc.) in the model. If the model changes in reasonable and expected ways when important variable values are changed, this can enhance trust in the model.

This page utilizes the `What If Tool <https://pair-code.github.io/what-if-tool/>`__ for displaying the SA information. The top portion of this page includes:  -  A summary of the experiment -  Predictions for a specified column. Change the column on the Y axis    to view predictions for that column. -  The current working score set. This updates each time you rescore. The bottom portion of this page includes:  -  A filter tool for filtering the analysis. Choose a different column,    predictions, or residuals. Set the filter type (<,>, etc.). Choose to filter by False Positive, False Negative, True Positive, or    True Negative. -  Scoring chart. Click the **Rescore** button after applying a filter    to update the scoring chart. This chart also lets you add or remove    variables, toggle the main chart aggregation, reset the data, and    delete the global history while resetting the data. -  The current history of actions taken on this page.

.. figure:: images/sensitivity_analysis.png    :alt:   Column actions ^^^^^^^^^^^^^^  When clicking a column in SA, the following actions are available:  -  **Absolute:** Change a column to a specific value for all rows. For    example, you can set a column to have the value 5 for all    observations. This is also possible for categorical columns. For    example, you can set a categorical column to have the value "foobar"    for all observations. -  **Relative:** Change a numeric column by adding a specific value to    all rows. For example, you can add 9 to all observations in a    numerical column. You can also pass in a negative number, for    example, -9. The input must be numeric. -  **Percentage:** Change a numeric column by some percentage. For    example, passing 9 to this field changes all values to be 9% of its    original value. For example, if the value is 2 and you pass in 9 as    the percentage, then the value changes to be 0.18. The input must be    an integer.

-  **Randomize:** Randomly change the values in a column, irrespective    of what is in the textbox. The change itself is absolute and based on    the domain of the column. .. figure:: images/sa-column-actions.png    :alt:   Save and download filtered datasets in Sensitivity Analysis ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  The following steps describe how to save and download a filtered dataset in SA. 1. Navigate to SA. 2. Edit a dataset, and then click the save icon to save the dataset. 3. After saving the filtered dataset, it is made available on the    Datasets page. .. figure:: images/filtered-dataset-sa.png    :alt:   Understand residuals ^^^^^^^^^^^^^^^^^^^^  Residuals are differences between observed and predicted values. In Sensitivity Analysis, the method used to calculate residuals varies depending on the type of problem. For classification problems, logloss residuals are calculated for the class of interest.

Use cases ^^^^^^^^^  **Use Case 1: Using SA on a Single Row or on a Small Group of Rows**  This section describes scenarios for using SA for explanation, debugging, security, or fairness when scoring a trained model on a single row or on a small group of rows. -  **Explanation**: Change values for a variable, and then rescore the    model. View the difference between the original prediction and the    new model prediction. If the change is big, then the changed variable    is locally important. -  **Debugging**: Change values for a variable, and then rescore the    model. View the difference between the original prediction and the    new model prediction and determine whether the change to variable    made the model more or less accurate. -  **Security**: Change values for a variable, and then rescore the    model. View the difference between the original prediction and the    new model prediction.

-  **Fairness**: Change values for a demographic variable, and then    rescore the model. View the difference between the original    prediction and the new model prediction. If change is big, then the    user can consider using a different model, regularizing the model    more, or applying post-hoc bias remediation techniques. -  **Random**: Set variables to random values, and then rescore the    model. This can help you look for things the you might not have    thought of. **Use Case 2: Using SA on an Entire Dataset and Trained Model**  This section describes scenarios for using SA for explanation, debugging, security, or fairness when scoring a trained model for an entire dataset and trained predictive model.

Change the values of the variables in their    entire dataset, and look at the **Percentage Change** in the average    model score (default probability) on the original and new data. They    can then use this discovered information along with external    information and processes to understand whether their institution has    enough cash on hand to be prepared for the simulated crisis. -  **Random**: Set variables to random values, and then rescore the    model. This lets users look for things they may not have otherwise    considered. Additional Resources ^^^^^^^^^^^^^^^^^^^^  `Sensitivity Analysis on a Driverless AI Model <https://github.com/h2oai/driverlessai-tutorials/blob/master/interpretable_ml/MLISensitivityAnalysis.ipynb>`__: This ipynb uses the `UCI credit card default data <https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients>`__ to perform sensitivity analysis and test model performance.

- When permutation importance is enabled for    interpretations, it is run as part of the interpretation process,    regardless of whether it was run for the original experiment or    AutoDoc. Permutation-based feature importance shows how much a model's performance would change if a feature's values were permuted. If the feature has little predictive power, shuffling its values should have little impact on the model's performance. If a feature is highly predictive, however, shuffling its values should decrease the model's performance. The difference between the model's performance before and after permuting the feature provides the feature's absolute permutation importance. .. figure:: images/permutation_feature_importance.png    :alt:   Surrogate Model Plots ---------------------  This section describes the plots that are available in the Surrogate Model Tab.

A K-LIME graph is available by default when you interpret a model from the experiment page. When you create a new interpretation, you can instead choose to use LIME-SUP as the LIME method. Note that these graphs are essentially the same, but the K-LIME/LIME-SUP distinction provides insight into the LIME method that was used during model interpretation. .. _klime_technique:  The K-LIME Technique ^^^^^^^^^^^^^^^^^^^^  This plot is available for binary classification and regression models. **Summary**  K-LIME creates one global surrogate GLM on the entire training data and also creates numerous local surrogate GLMs on samples formed from *k*-means clusters in the training data. The parameters of the global K-LIME model give an indication of overall linear feature importance and the overall average direction in which an input variable influences the Driverless AI model predictions.

**Additional details**  K-LIME is a variant of the LIME technique proposed by Ribeiro at al (2016). K-LIME generates global and local explanations that increase the transparency of the Driverless AI model, and allow model behavior to be validated and debugged by analyzing the provided plots, and comparing global and local explanations to one-another, to known standards, to domain knowledge, and to reasonable expectations. As described in the preceding section, K-LIME creates numerous local surrogate GLMs on samples formed from *k*-means clusters in the training data. The features used for *k*-means are selected from the Random Forest surrogate model's variable importance. The number of features used for *k*-means is the minimum of the top 25% of variables from the Random Forest surrogate model's variable importance and the max number of variables that can be used for *k*-means, which is set by the user in the config.toml setting formli_max_number_cluster_vars``.

use_all_columns_klime_kmeansin the config.toml file totrue. All penalized GLM surrogates are trained to model the predictions of the Driverless AI model. The number of clusters for local explanations is chosen by a grid search in which the :math:`R^2` between the Driverless AI model predictions and all of the local K-LIME model predictions is maximized. The global and local linear model's intercepts, coefficients, :math:`R^2` values, accuracy, and predictions can all be used to debug and develop explanations for the Driverless AI model's behavior. In addition to the usage described in the preceding section, the global model is also used to generate explanations for very small clusters (:math:`N < 20`) where fitting a local linear model is inappropriate.

For a point within a cluster, the sum of the local linear model intercept and the products of each coefficient with their respective input variable value are the K-LIME prediction. By disaggregating the K-LIME predictions into individual coefficient and input variable value products, the local linear impact of the variable can be determined. This product is sometimes referred to as a reason code and is used to create explanations for the Driverless AI model's behavior. .. raw:: html     <img src="_static/reason-codes-page.gif" alt="Recipe expert settings" data-linktype="relative-path">  **Reason codes in K-LIME**  The K-LIME plot includes a **Reason codes** page that can be accessed by clicking the **Explanations** button. From the **Reason codes** page, you can view information about both cluster-specific reason codes and global reason codes. In K-LIME, reason code values are calculated by determining each coefficient-feature product.

In the following example, reason codes are created by evaluating and disaggregating a local linear model.

By taking into consideration the value of each contribution, reason codes for the Driverless AI decision can be derived. debt_to_income_ratio and credit_score would be the two largest negative reason codes, followed by savings_acct_balance. The local linear model intercept and the products of each coefficient and corresponding value sum to the K-LIME prediction. Moreover it can be seen that these linear explanations are reasonably representative of the nonlinear model's behavior for this individual because the K-LIME predictions are within 5.5% of the Driverless AI model prediction. This information is encoded into English language rules which can be viewed by clicking the **Explanations** button. Like all LIME explanations based on linear models, the local explanations are linear in nature and are offsets from the baseline prediction, or intercept, which represents the average of the penalized linear model residuals.

When K-LIME accuracy for a given point or set of points is quite low, this can be an indication of extremely nonlinear behavior or the presence of strong or high-degree interactions in this local region of the Driverless AI response function. In cases where K-LIME linear models are not fitting the Driverless AI model well, nonlinear LOCO feature importance values may be a better explanatory tool for local model behavior. As K-LIME local explanations rely on the creation of *k*-means clusters, extremely wide input data or strong correlation between input variables may also degrade the quality of K-LIME local explanations. .. _limesup_technique:  The LIME-SUP Technique ^^^^^^^^^^^^^^^^^^^^^^  This plot is available for binary classification and regression models. LIME-SUP explains local regions of the trained Driverless AI model in terms of the original variables.

For each local region, a local GLM model is trained on the original inputs and the predictions of the Driverless AI model. Then the parameters of this local GLM can be used to generate approximate, local explanations of the Driverless AI model. **The Global Interpretable Model Explanation Plot**  This plot shows Driverless AI model predictions and LIME model predictions in sorted order by the Driverless AI model predictions. This graph is interactive. Hover over the **Model Prediction**, **LIME Model Prediction**, or **Actual Target** radio buttons to magnify the selected predictions. Or click those radio buttons to disable the view in the graph. You can also hover over any point in the graph to view LIME reason codes for that value. By default, this plot shows information for the global LIME model, but you can change the plot view to show local results from a specific cluster.

The closer the local linear model approximates the Driverless AI model predictions, the more linear the Driverless AI model and the more accurate the explanation generated by the LIME local linear models. .. figure:: images/global_interpretable.png    :alt:   .. _decision-tree:  Surrogate Decision Tree ~~~~~~~~~~~~~~~~~~~~~~~  The decision tree surrogate model increases the transparency of the Driverless AI model by displaying an *approximate* flow-chart of the complex Driverless AI model's decision making process. It also displays the most important variables in the Driverless AI model and the most important interactions in the Driverless AI model. The decision tree surrogate model can be used for visualizing, validating, and debugging the Driverless AI model by comparing the displayed decision-process, important variables, and important interactions to known standards, domain knowledge, and reasonable expectations. It is known to date back at least to 1996 (Craven and Shavlik).

Given our learned function :math:`g` and set of predictions, :math:`g(X) = \hat{Y}`, we can train a surrogate model :math:`h`: :math:`X,\hat{Y} \xrightarrow{\mathcal{A}_{\text{surrogate}}} h`, such that :math:`h(X)` is approximately equal to :math:`g(X)`. To preserve interpretability, the hypothesis set for :math:`h` is often restricted to linear models or decision trees. For the purposes of interpretation in Driverless AI, :math:`g` is considered to represent the entire pipeline, including both the feature transformations and model, and the surrogate model is a decision tree (:math:`h_{\text{tree}}`). Users must also note that there exist few guarantees that :math:`h_{\text{tree}}` accurately represents :math:`g`. The RMSE for :math:`h_{\text{tree}}` is displayed for assessing the fit between :math:`h_{\text{tree}}` and :math:`g`.

:math:`h_{\text{tree}}` can be used for visualizing, validating, and debugging :math:`g` by comparing the displayed decision-process, important features, and important interactions to known standards, domain knowledge, and reasonable expectations. The preceding image displays the decision tree surrogate, :math:`h_{\text{tree}}`, for an example probability of default model, :math:`g`, created with Driverless AI using the UCI repository credit card default data (see https://www.kaggle.com/uciml/default-of-credit-card-clients-dataset). ThePAY_0feature is likely the most important feature in :math:`g` due to its place in the initial split in :math:`h_{\text{tree}}` and its second occurrence on the third level of :math:`h_{\text{tree}}`. First level interactions betweenPAY_0andPAY_2and betweenPAY_0andPAY_5are visible along with several second level interactions.

The thickness of the edges in this path indicate that this is a very common decision path through :math:`h_{\text{tree}}`. Following the decision path to the highest probability leaf node in :math:`h_{\text{tree}}` (second from right in the preceding image) shows that customers who are late on their first (PAY_0) and fifth (PAY_5) month bills and who pay less than 16520 in their sixth payment (PAY_AMT6) are the most likely to default according to :math:`h_{\text{tree}}`. The thinness of the edges in this path indicate that this is a relatively rare decision path through :math:`h_{\text{tree}}`. When an observation of data is selected using the k-LIME plot, :math:`h_{\text{tree}}` can also provide a degree of local interpretability. When a single observation, :math:`x^{(i)}`, is selected, its path through :math:`h_{\text{tree}}` is highlighted. The path of :math:`x^{(i)}` through :math:`h_{\text{tree}}` can be helpful when analyzing the logic or validity of :math:`g(x^{(i)})`.

-          (2) The attributes of a decision tree are used to explain global           attributes of a complex Driverless AI model such as important           features, interactions, and decision processes. -  **Appropriate Response Function Complexity**: Decision tree surrogate    models can create explanations for models of nearly any complexity. -  **Understanding and Trust**:     -          (1) Decision tree surrogate models foster understanding and           transparency because they provide insight into the internal           mechanisms of complex models. -          (2) They enhance trust, accountability, and fairness when their           important features, interactions, and decision paths are in           line with human domain knowledge and reasonable expectations. -  **Application Domain**: Decision tree surrogate models are model    agnostic.

In the Decision Tree plot, the highlighted row shows the path to the highest probability leaf node and indicates the globally important variables and interactions that influence the Driverless AI model prediction for that row. You can view rules for a specific path by clicking the path's terminal node. **Note**: For a list of Surrogate Decision Tree explainer expert settings, see :ref:`interpretation-expert-settings-surrogate-dt`. .. raw:: html     <img src="_static/mli_surrogate_dt_plot.gif" alt="Surrogate Decision Tree Plot" data-linktype="relative-path">  For multiclass models, decision trees are created for each class. To view a decision tree for a specific class, click **Class** in the upper-left corner of the page and select the class you want to view a decision tree for.

**Global Feature Importance vs Local Feature Importance**  Global feature importance (yellow) is a measure of the contribution of an input variable to the overall predictions of the Driverless AI model. Global feature importance is calculated by aggregating the improvement in splitting criterion caused by a single variable across all of the decision trees in the Random Forest surrogate model. Local feature importance (grey) is a measure of the contribution of an input variable to a single prediction of the Driverless AI model. Local feature importance values for regression and binomial cases are calculated by tracing single rows of data through the random forest surrogate model and returning the absolute LOCO values. For the multiclass case, local feature importance values are calculated by re-scoring the trained supervised model and measuring the impact of setting each variable to missing.

Both global and local variable importance are scaled so that the largest contributor has a value of 1. **Note**: Engineered features are used for MLI when a time series experiment is built. This is because munged time series features are more useful features for MLI than raw time series features, as raw time series features are not IID (Independent and Identically Distributed). .. figure:: images/rf_feature_importance.png    :alt:   .. _rf-pdp-ice:  Random Forest Partial Dependence and Individual Conditional Expectation ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  A Partial Dependence and ICE plot is available for both Driverless AI and surrogate models. Refer to the previous :ref:`pdp-ice` section for more information about this plot. .. _rf-loco:  Random Forest LOCO ~~~~~~~~~~~~~~~~~~  This plot is available for binary and multiclass classification models as well as regression models.

Local feature importance values reported in this plot are based on a variant of the leave-one-covariate-out (LOCO) method (Lei et al, 2017). The LOCO-variant method for binary and regression models is calculated by traversing the random forest surrogate model and removing the prediction contribution of any rule containing the variable of interest for every tree from the original prediction. Local LOCO values are calculated by tracing single rows of data through the random forest surrogate model. Global LOCO values are the average of the LOCO values over every row of a dataset. The LOCO-variant method for multiclass models differs slightly in that it calculates row-wise local feature importance values by re-scoring the trained supervised model and measuring the impact of setting each variable to missing. The sum of the absolute value of differences across classes is then calculated for each dropped or replaced column.

First, the modified predictions are calculated:     :math:`F_{~debt\_to\_income\_ratio} = F(NA, 600, 1000) = 0.99`     :math:`F_{~credit\_score} = F(30, NA, 1000) = 0.73`     :math:`F_{~savings\_acct\_balance} = F(30, 600, NA) = 0.82`  Second, the original prediction is subtracted from each modified prediction to generate the unscaled local feature importance values:     :math:`\text{LOCO}_{debt\_to\_income\_ratio} = F_{~debt\_to\_income\_ratio} - 0.85 = 0.99 - 0.85 = 0.14`     :math:`\text{LOCO}_{credit\_score} = F_{~credit\_score} - 0.85 = 0.73 - 0.85 = -0.12`     :math:`\text{LOCO}_{savings\_acct\_balance} = F_{~savings\_acct\_balance} - 0.85 = 0.82 - 0.85 = -0.03`  Finally LOCO values are scaled between 0 and 1 by dividing each value for the row by the maximum value for the row and taking the absolute magnitude of this quotient.

.. figure:: images/loco_plot.png       :alt:   .. _nlp-surrogate:  NLP Surrogate Models ~~~~~~~~~~~~~~~~~~~~  These plots are available for natural language processing (NLP) models. For NLP surrogate models, Driverless AI creates a TF-IDF matrix by tokenizing all text features. The resulting frame is appended to numerical or categorical columns from the training dataset, and the original text columns are removed. This frame is then used for training surrogate models that have prediction columns consisting of tokens and the original numerical or categorical features. **Notes**:  -  MLI support for NLP is only available for binary classification and    regression experiments. -  Each row in the TF-IDF matrix contains :math:`N` columns, where    :math:`N` is the total number of tokens in the corpus with values    that are appropriate for that row (0 if absent). -  Driverless AI does not currently generate a K-LIME scoring pipeline    for MLI NLP problems.

The method used to calculate residuals varies depending on the type of problem. For classification problems, logloss residuals are calculated for a specified class. For regression problems, residuals are determined by calculating the square of the difference between targeted and predicted values. To run MLI surrogate models on residuals, enable the **Debug Model Residuals** interpretation expert setting. For classification experiments, specify a class to use as an outcome of interest with the **Class for Debugging Classification Model Logloss Residuals** interpretation expert setting (not visible for regression problems). You can view the models by clicking the **Surrogate Models on Residuals** tab once the interpretation is complete. .. figure:: images/mli_surrogate_residuals.png    :alt:   .. _mli-nlp-plots:  NLP Plots ---------  This section describes the plots that are available in the NLP tab.

.. _dai-nlp-loco:  NLP Leave-One-Covariate-Out (LOCO) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  This plot is available for binomial, multiclass, and regression natural language processing (NLP) models. It is located in the **NLP** tab on the Model Interpretation page, which is only visible for NLP models. .. raw:: html     <img src="_static/nlp_loco.gif" alt="NLP LOCO" data-linktype="relative-path">  This plot applies a leave-one-covariate-out (LOCO) styled approach to NLP models by removing a specific token, which is obtained by TF-IDF, from only a single column where the token is occurring. For example, if there is a tokenfooin bothcolumn1andcolumn2, LOCO is computed for both columns separately, even though the token is the same. Note that the TF-IDF for the token differs in both columns. In addition, if a token does **not** exist in a row, then it is appended before calculating LOCO to ensure the token was evaluated across all rows.

Driverless AI fits a separate TF-IDF vectorizer for each individual column and concatenates the results. The terms (tokens) in the resulting importance frames are then wrapped with column names:  .. table:: Column Names Example     +-----------------------+-----------------------+-----------------------+    | column1('and')        | column1('apple')      | column2('and')        |    +=======================+=======================+=======================+    | 0.1                   | 0.0005                | 0.412512              |    +-----------------------+-----------------------+-----------------------+  The NLP LOCO plot lets you view text for a specific row by specifying a row number. Each token in the row is highlighted by importance. You can switch between different text features and view their respective importances globally and locally. .. note::     - Due to computational complexity, the global importance value is    only calculated for :math:`N` (20 by default) tokens.

- A    specific token selection method can be used by specifying one of the    following options for themli_nlp_min_token_modeconfiguration    option:     -linspace: Selects :math:`N` evenly spaced tokens according to       their TF-IDF score (Default)    -top: Selects top :math:`N` tokens by TF-IDF score    -bottom: Selects bottom :math:`N` tokens by TF-IDF score  ..     -  Local values for NLP LOCO can take a significant amount of time to       calculate depending on the specifications of your hardware. -  Driverless AI does not currently generate a K-LIME scoring       pipeline for MLI NLP problems. .. _mli-nlp-pdp:  NLP Partial Dependence Plot ~~~~~~~~~~~~~~~~~~~~~~~~~~~  This plot is available for binomial, multiclass, and regression natural language processing (NLP) models. It is located in the **NLP** tab on the Model Interpretation page, which is only visible for NLP models.

ICE (grey) displays the prediction behavior for an individual row of data when an input text token is left in its respective text and not included in its respective text. The text tokens are generated from TF-IDF. .. raw:: html     <img src="_static/nlp_pdp.gif" alt="NLP Partial Dependence Plot" data-linktype="relative-path">  .. _mli-nlp-tokens:  NLP Tokenizer ~~~~~~~~~~~~~  This plot is available for natural language processing (NLP) models. It is located in the **NLP** tab on the Model Interpretation page, which is only visible for NLP models. .. raw:: html     <img src="_static/nlp_tokens.gif" alt="NLP Tokens" data-linktype="relative-path">  This plot shows both the global and local importance values of each token in a corpus (a large and structured set of texts). The corpus is automatically generated from text features used by Driverless AI models prior to the process of tokenization.

The TF-IDF increases proportionally to the number of times a token appears in a given document and is offset by the number of documents in the corpus that contain the token. Specify the row that you want to view, then click the **Search** button to see the local importance of each token in that row. Global importance values are calculated by using the inverse document frequency (IDF), which measures how common or rare a given token is across all documents. (Default View)  You can download an archive of files relating to the NLP Tokenizer plot by clicking "NLP Tokenizer ZIP Archive" in the NLP tab. .. note::     - MLI for NLP does not currently feature the option to remove stop    words. - By default, up to 10,000 tokens are created during the    tokenization process. This value can be changed in the configuration. - By default, Driverless AI uses up to 10,000 documents to extract    tokens from. This value can be changed with theconfig.mli_nlp_sample_limitparameter.

- Driverless    AI does not currently generate a K-LIME scoring pipeline for MLI NLP    problems. - With the LOCO method, a specific token is removed from    only a single column where the token is occurring. For example, if    there is a tokenfooin bothcolumn1andcolumn2``, LOCO is
    computed for both columns separately, even though the token is the
    same. The TF-IDF for the token differs in both columns. NLP VECTORIZER + LINEAR MODEL (VLM) TEXT FEATURE IMPORTANCE

This plot is available for binomial and regression natural language
processing (NLP) models. It is located in the NLP tab on the Model
Interpretation page, which is only visible for NLP models.

MODEL INSIGHTS

For time series and Automatic Image Model experiments, you can view
detailed insights while an experiment is running or after an experiment
is complete by clicking on the INSIGHTS option. []

Time Series Insights
The Insights page for time series experiments includes the following
graphs:
-   Time Based Validation Splits for Model Tuning and Feature Evolution
-   Training Data for Final Pipeline
-   Time Based Back Testing of Final Pipeline on Training Data
You can hover over a point in the graphs to view the dates in the
graphs.

AWS ROLE-BASED AUTHENTICATION

In Driverless AI, it is possible to enable role-based authentication via
the IAM role. This is a two-step process that involves setting up AWS
IAM and then starting Driverless AI by specifying the role in the
config.toml file or by setting the AWS_USE_EC2_ROLE_CREDENTIALS
environment variable to

True. AWS IAM Setup -------------  1. Create an IAM role. This IAM role should have a Trust Relationship    with Principal Trust Entity set to your Account ID. For example:    trust relationship for Account ID 524466471676 would look like:  ..     .. code:: bash        {         "Version": "2012-10-17",         "Statement": [           {             "Effect": "Allow",             "Principal": {               "AWS": "arn:aws:iam::524466471676:root"             },             "Action": "sts:AssumeRole"           }         ]       }     |image|  2. Create a new policy that lets users assume the role:  ..     |image|  3. Assign the policy to the user. ..     |image|  4. Test role switching here: https://signin.aws.amazon.com/switchrole. (Refer to    https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoot_roles.html#troubleshoot_roles_cant-assume-role.)

AWS_USE_EC2_ROLE_CREDENTIALS`` environment variable.

RESOURCES

1.  Granting a User Permissions to Switch Roles:
    https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_permissions-to-switch.html
2.  Creating a Role to Delegate Permissions to an IAM User:
    https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user.html
3.  Assuming an IAM Role in the AWS CLI:
    https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-role.html

CUSTOM RECIPE MANAGEMENT

The following sections describe custom recipe management in Driverless
AI. Understanding Custom Recipes
required. Custom recipes can be provided for transformers, models, and
scorers. During training of a supervised machine learning modeling
pipeline, Driverless AI can use these code snippets as building blocks
in combination with or in place of built-in code pieces. When selecting
recipes for an experiment in the expert-settings panel, only custom
recipes that are currently active are visible. New datasets can be created by
modifying an existing dataset with a data recipe <modify_by_recipe>. You
can also apply data recipes as standalone recipes. Additionally, the set
of MLI techniques and methodologies used in Driverless AI can be
extended with recipes. For more information on MLI explainer recipes,
see mli-byor.

You may be required to build several dependencies before
running scorers that use custom recipes and to be able to compile. For
complete examples that demonstrate how to download these dependencies
and run the Python Scoring Pipeline, see Python_Pipeline. -   In most cases, and especially for complex recipes, MOJO for model
    deployment is not available out of the box. However, _it is
    possible_ to get the MOJO. Contact support@h2o.ai for more
    information about creating MOJOs for custom recipes. -   To enable Shapley calculations in MLI, custom model recipes must use
    the has_pred_contribs method. Refer to the model recipe template for
    more info.

Select one of the following options from the drop-down menu that
appears:
[]
    local file system. -   FROM URL: Add one or more custom recipes from a URL that points to
    one of the following locations:
      -   A GitHub repository. For example, you can enter
          https://github.com/h2oai/driverlessai-recipes/ to add all the
          custom recipes contained in the official Recipes for
          Driverless AI repository. -   A GitHub tree. For example, you can enter
          https://github.com/h2oai/driverlessai-recipes/tree/master/models
          to add only the custom model recipes contained in the official
          Recipes for Driverless AI repository, or enter
          https://github.com/h2oai/driverlessai-recipes/tree/master/models/algorithms
          to add only the custom algorithm recipes contained in the
          repository. -   A file system path.

-   FROM BITBUCKET: Add a custom recipe from a Bitbucket repository. To
    provided along with the custom recipe Bitbucket URL. -   WITH EDITOR: Add a custom recipe with a built-in code editor. []
Note
When uploading a new recipe, any recipes that share either the same
class name or _display_name attribute are deactivated. Note
If you set the _global_modules_needed_by_name parameter in a custom
recipe, then ensure that it is set on a single line before uploading it. Using line breaks when setting the _global_modules_needed_by_name
parameter results in a syntax error when attempting to upload the custom
recipe. Managing Recipes
Two distinct views are available on this page:
-   LIST VIEW: This view displays all available custom recipes. Only
    active recipes are listed by default, but deactivated recipes can
    also be viewed. For more information, see list-view.

The detail view is available
    for both active and deactivated recipes. For more information, see
    detail-view. List View
The following is a list of actions that you can take from the recipe
list view:
_General actions_:
-   View deactivated recipes by selecting INCLUDE INACTIVE RECIPES. -   Deactivate a recipe by selecting it and clicking DEACTIVATE x
    ITEM(S). You can use this option to deactivate multiple recipes at
    one time. Note that recipes can only be deactivated, not deleted. -   Search and sort recipes. Note that if enough recipes are uploaded,
    they are listed on multiple pages. -   Select which columns are visible on the list view. _Recipe-specific actions_:
-   OPEN: View a specific recipe in detail. -   EDIT NOTE: Create or edit a note for a recipe to keep track of its
    functionality. -   DEACTIVATE: Deactivate the selected recipe. -   APPLY ON DATASET (_For data recipes only_): Apply an existing data
    recipe to the dataset.

-   APPLY WITHOUT DATASET (FOR DATA RECIPES ONLY): Apply the selected
    data recipe as a standalone recipe. Detail View
detail view:
-   Edit custom recipe code:
      -   You can toggle an in-code search feature by pressing Control+F
          (or Command+F on Mac). -   To save the edited recipe, click the SAVE AS NEW RECIPE AND
          ACTIVATE button. -   If you change both the ClassName and _display_name defined in
          the recipe, then the old version of the recipe remains active. If you don't change both the ClassName and _display_name
          defined in the recipe, the old version of the recipe is
          automatically deactivated when a new version is saved and
          activated. New versions of existing recipes keep references to
          the original recipes, letting you keep track of changes
          throughout multiple versions.

-   View the recipe's name, type, ID, filename, creation date, and
    whether the recipe is currently active. -   (_For data recipes only_) Apply the data recipe on a dataset or as a
    standalone recipe. -   If a recipe was downloaded from an external URL, the link is
    displayed under ORIGINAL URL. -   (_For Individual recipes only_) View a link to the experiment from
    which the Individual recipe was derived from. -   MORE ACTIONS drop-down:
      -   (_For Individual recipes only_) To create a new experiment
          using the Individual recipe, click USE IN NEW EXPERIMENT. The
          new experiment is set up based on the settings used in the
          parent experiment (equivalent to using the NEW WITH SAME
          SETTINGS option), and the individual recipe is also applied. -   Download the recipe by clicking DOWNLOAD. -   Deactivate the recipe by clicking DEACTIVATE. Note that
          recipes can only be deactivated, not deleted.

NATIVE INSTALLATION

This section provides instructions for installing Driverless AI in
native Linux environments.

install/x86-64

For instructions on installing the Driverless AI Docker image, refer to
docker_installs.

ENSEMBLE LEARNING IN DRIVERLESS AI

This section describes Driverless AI's ensemble learning capabilities. Ensemble Method
An ensemble is a hierarchical composition of multiple models, where
every level in the hierarchy uses the output of the previous level as
input. The simplest ensemble is a 2-layer architecture with a single
linear model (the meta model or meta learner) combining the predictions
from several first layer models (base models). This is the default
ensemble model in Driverless AI due to its robustness and linear
properties that allow Shapley contributions to be fully interpretable
even for ensembles. By default, the meta learner is a linear blender that assigns
non-negative weights (that sum to 1) to all the base models. The weights
are assigned at the model level and obtained using cross-validation (to
avoid overfitting of the meta learner). When making prediction on a test
set, the predictions from all cross-validation models are averaged.

All cross-validation
models are averaged, so for 4-fold CV, each XGBoost fold model has an
effective weight of 0.63/4. When Driverless AI ensembles a single model
(level 1), then it is simply taking the average of the CV model
predictions (the model itself is assigned a weight of 1). Ensemble Levels
Driverless AI has multiple ensemble levels that are tied to the accuracy
knob. As accuracy increases, the ensemble level increases. Ensemble level can also be controlled using
Ensemble Level for Final Modeling Pipeline <fixed_ensemble_level> from
the Model settings of the expert settings panel. The following is a
description of each ensemble level:
-   LEVEL 0: No ensemble, only a final single model. Cross validation is
    only used to determine the model validation performance. The final
    model is trained on the whole dataset. -   LEVEL 1: Cross validation is performed for 1 model and the CV model
    predictions are ensembled.

PASSING ADDITIONAL PIP INSTALL OPTIONS

You can use the pip_install_options TOML option <understanding-configs>
to pass additional pip install options formatted as a list. The
following are two examples that demonstrate how this option can be used.

-   When installing Python packages, you can use this TOML option to
    specify your organization's internal Python package index as
    follows:

-   You can use this TOML option to install Python packages with a proxy
    server as follows:

Passing multiple pip install options to DAI

The following example demonstrates how to correctly pass multiple pip
install options to DAI.

    pip_install_options="['--extra-index-url', 'http://my-own-repo1:port','--extra-index-url', 'http://my-own-repo2:port']"

MLI OVERVIEW

Driverless AI provides robust interpretability of machine learning
models to explain modeling results in a human-readable format. In the
Machine Learning Interpretability (MLI) view, Driverless AI employs a
host of different techniques and methodologies for interpreting and
explaining the results of its models. A number of charts are generated
automatically (depending on experiment type), including K-LIME, Shapley,
Variable Importance, Decision Tree Surrogate, Partial Dependence,
Individual Conditional Expectation, Sensitivity Analysis, NLP Tokens,
NLP LOCO, and more. Additionally, you can download a CSV of LIME,
Shapley, and Original (Kernel SHAP) Shapley reason codes as well as text
and Python files of Decision Tree Surrogate model rules from this view. The techniques and methodologies used by Driverless AI for model
interpretation can be extended with recipes (Python code snippets).

This chapter describes Machine Learning Interpretability (MLI) in
Driverless AI for both regular and time-series experiments. Refer to the
following sections for more information:
-   interpreted-model-page
-   interpret-regular
-   interpret-ts
-   mli-byor
Note
MIGRATION INFORMATION
-   Interpretations made in version 1.9.0 are supported in 1.9.x and
    later. -   Interpretations made in version 1.8.x aren't supported in 1.9.x and
    later. However, interpretations made in 1.8.x can still be viewed
    and rerun. Note
- MLI is not supported for unsupervised learning models. - MLI is not
supported for Image or multiclass Time Series experiments. - MLI does
not require an Internet connection to run on current models. - To
specify a port of a specific H2O instance for use by MLI, use the
h2o_port config.toml <sample-configtoml> setting. You can also specify
an IP address for use by MLI with the h2o_ip setting.

INSTALLING THE PYTHON CLIENT

The Driverless AI Python Client is available from the RESOURCES
drop-down in Driverless AI GUI. The latest release of the client is also
available from Python Package Index and Anaconda Cloud. For Driverless AI Python client API documentation, see

The Python Client requires Python version 3.6 or greater. In addition,
the following Python packages must be installed:
-   requests
-   tabulate
-   toml
-   tornado

Download and Installation
The Python Client can be downloaded from either the RESOURCES drop-down
in Driverless AI or the command-line. Download from Resources Drop-Down
In the Driverless AI top menu, select the RESOURCES > PYTHON CLIENT link
to download the DRIVERLESSAI Python wheel. []
Download from Command-Line
The Driverless AI Python client is exposed as the /clients/py HTTP end
point.

pip
installcommand. Once installed, you can launch a Jupyter notebook and begin using the Driverless AI Python client.  Installing from Python Package Index (PyPI) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  The latest release of the client is available on PyPI and can be installed to your desired Python environment withpip``.
The following command installs the latest version of the Python Client:

    pip install driverlessai

To upgrade when new versions of the client are released, run the
following command:

    pip install --upgrade driverlessai

INSTALLING FROM ANACONDA CLOUD

To install the Python Client as a conda package, use the following
command:

    conda install -c h2oai driverlessai

-   (2) Replace <REDIS_PASSWORD> with default Redis password or new one. -   (3) Replace various ports with alternative values if required. -   (4) Replace docker_image with the image (include repository if
        remote image). -   (5) For GPU usage, --runtime nvidia is required. Systems without
        GPUs should remove this line. -   (6) Dask on cluster can be disabled by passing
        dai_enable_dask_cluster=0. If Dask on cluster is disabled, then
        dai_dask_server_ip does not need to be set. -   (7) Dask dashboard ports (for example, 8787 and 8791) and H2O-3
        ports 12348, 50361, and 50362 are not required to be exposed. These are for user-level access to H2O-3 or Dask behavior. -   (8) Jupyter can be disabled by passing dai_enable_jupyter_server=0
        and dai_enable_jupyter_server_browser=0. -   (9) Dask requires the host network be used so scheduler can tell
        workers where to find other workers, so a subnet on new IP
        cannot be used, e.g.

-   (10) To isolate user access to single user, instead of doing
        -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro one
        can map to user files with the same required information. These
        options ensure container knows who user is. -   (11) Directories created should have not existed or should be from a
        prior run by same user. Pre-existing directories should be moved
        or names changed to avoid conflicts. -   (12) Services like the Procsy server, H2O-3 MLI and Recipe servers,
        and Vis-data server are only used internally for each node. -   (13) The options -p 12400:12400 is only required to LightGBM Dask. -   (14) NCCL_SOCKET_IFNAME should specify the actual hardware device to
        use, as required due to issues with NCCL obtaining the correct
        device automatically from IP.

-   (2) NCCL_SOCKET_IFNAME should specify actual hardware name, in
        general different on each node. Dask Multinode Example running tar
On main server with public IP address 172.16.2.210:
    export DRIVERLESS_AI_LICENSE_FILE=/home/$$USER/.driverlessai/license.sig
    export server=172.16.2.210
    NCCL_SOCKET_IFNAME="enp5s0" \
    NCCL_DEBUG=WARN \
    NCCL_P2P_DISABLE=1 \
    dai_dask_server_ip=$server dai_redis_ip=$server dai_redis_port=6379 \
    dai_main_server_minio_address=$server:9001 dai_ip=$server dai_main_server_redis_password="<REDIS_PASSWORD>" \
    dai_worker_mode='multinode' dai_enable_dask_cluster=1 \
    dai_enable_jupyter_server=1 dai_enable_jupyter_server_browser=1 \
    /opt/h2oai/dai/dai-env.sh python -m h2oai &> multinode_main.txt
On each worker node, run the exact same command but with --worker added
at the end, i.e.

EXPERIMENT SETTINGS

This section includes settings that can be used to customize the
experiment like total runtime, reproducibility level, pipeline building,
feature brain control, adding config.toml settings and more.

MAX_RUNTIME_MINUTES``

MAX RUNTIME IN MINUTES BEFORE TRIGGERING THE FINISH BUTTON

Specify the maximum runtime in minutes for an experiment. This is
equivalent to pushing the FINISH button once half of the specified time
value has elapsed. Note that the overall enforced runtime is only an
approximation.

This value defaults to 1440, which is the equivalent of a 24 hour
approximate overall runtime. The FINISH button will be automatically
selected once 12 hours have elapsed, and Driverless AI will subsequently
attempt to complete the overall experiment in the remaining 12 hours.
Set this value to 0 to disable this setting.

Note that this setting applies to per experiment so if building
leaderboard models(n) it will apply to each experiment separately(i.e
total allowed runtime will be n*24hrs. This time estimate assumes
running each experiment one at a time, sequentially)

MAX_RUNTIME_MINUTES_UNTIL_ABORT``

MAX RUNTIME IN MINUTES BEFORE TRIGGERING THE ABORT BUTTON

Specify the maximum runtime in minutes for an experiment before
triggering the abort button. This option preserves experiment artifacts
that have been generated for the summary and log zip files while
continuing to generate additional artifacts. This value defaults to
10080 mins (7 days).

leaderboard models( say n), it will apply to each experiment
separately(i.e total allowed runtime will be n*7days. This time estimate
assumes running each experiment one at a time, sequentially). Also see
time_abort <time_abort>.

TIME_ABORT``

TIME TO TRIGGER THE 'ABORT' BUTTON
If the experiment is not done by this time, push the abort button. Note
that this applies to the leaderboard as well i.e if all leaderboard
experiments are not done, then entire leaderboard is aborted after this
actual time. Also see
max_runtime_minutes_until_abort <max_runtime_minutes_until_abort> for
control over per experiment abort times. This accepts time in format given by time_abort_format (defaults to
%Y-%m-%d %H:%M:%S).This assumes a timezone set by time_abort_timezone in
config.toml(defaults to UTC). User can also specify integer seconds
since 1970-01-01 00:00:00 UTC. This will apply to the time on a DAI worker that runs the experiments. Similar to
max_runtime_minutes_until_abort <max_runtime_minutes_until_abort>, time
abort will preserves experiment artifacts made so far for summary and
log zip files.

pipeline-building-recipe----------------------------  .. container:: dropdown     **Pipeline Building Recipe**     Specify the Pipeline Building recipe type (overrides GUI settings). Select from the following:     -  **Auto**: Specifies that all models and features are automatically       determined by experiment settings, config.toml settings, and the       feature engineering effort. (Default)    -  **Compliant**: Similar to **Auto** except for the following:           -  Interpretability is set to 10. -  Only uses GLM or booster as 'giblinear'. -  :ref:`Fixed ensemble level <fixed_ensemble_level>` is set to             0. -  :ref:`Feature brain level <feature_brain1>` is set to 0. -  Max feature interaction depth is set to 1 i.e no             interactions. -  Target transformers is set to 'identity' for regression. -  Does not use             :ref:`distribution shift <check_distribution_shift_drop>`             detection.

-  **monotonic_gbm**: Similar to **Auto** except for the following:           -  Enables monotonicity constraints          -  Only uses LightGBM model. -  Drops features that are not correlated with target by at             least 0.01. See             :ref:`monotonicity-constraints-drop-low-correlation-features <monotonicity-constraints-drop-low-correlation-features>`             and             :ref:`monotonicity-constraints-correlation-threshold <monotonicity-constraints-correlation-threshold>`. -  Does not build an ensemble model i.e setfixed_ensemble_level=0-  No :ref:`feature brain <feature_brain1>` is used to ensure             every restart is identical. -  :ref:`Interaction depth <max-feature-interaction-depth>` is             set to 1 i.e no multi-feature interactions done to avoid             complexity. -  No target transformations applied for regression problems             i.e sets :ref:`target_transformer <target_transformer>` to             'identity'.

-  :ref:`num_as_cat <num_as_cat>` feature transformation is             disabled.

-  **Kaggle**: Similar to **Auto** except for the following:           -  Any external validation set is concatenated with the train             set, with the target marked as missing. -  The test set is concatenated with the train set, with the             target marked as missing          -  Transformers that do not use the target are allowed tofit_transform`` across the entirety of the train,
    validation, and test sets. -   Has several config.toml expert options open-up limits. -   NLP_MODEL: Only enable NLP BERT models based on PyTorch to process
      pure text. To avoid slowdown when using this recipe, enabling one
      or more GPUs is strongly recommended. For more information, see
      nlp-in-dai.

To avoid slowdown when using this recipe,
      enabling one or more GPUs is strongly recommended. For more
      information, see nlp-in-dai. -   included_transformers = ['BERTTransformer']
        -   excluded_models = ['TextBERTModel',
            'TextMultilingualBERTModel', 'TextXLNETModel',
            'TextXLMModel','TextRoBERTaModel', 'TextDistilBERTModel',
            'TextALBERTModel', 'TextCamemBERTModel',
            'TextXLMRobertaModel']
        -   enable_pytorch_nlp_transformer = 'on'
        -   enable_pytorch_nlp_model = 'off'
  -   IMAGE_MODEL: Only enable image models that process pure images
      (ImageAutoModel). To avoid slowdown when using this recipe,
      information, see image-model. NOTES:
        -   This option disables the Genetic Algorithm <ga> (GA). -   Image insights are only available when this option is
            selected.

For more information, see
      image-embeddings. -   UNSUPERVISED: Only enable unsupervised transformers, models and
      scorers. See <unsupervised_algos> for reference. -   GPUS_MAX: Maximize use of GPUs (e.g. use XGBoost, RAPIDS, Optuna
      hyperparameter search, etc. that run on GPUs <gpu_in_dai>). Each pipeline building recipe mode can be chosen, and then fine-tuned
  using each expert settings. Changing the pipeline building recipe will
  reset all pipeline building recipe options back to default and then
  re-apply the specific rules for the new mode, which will undo any
  fine-tuning of expert options that are part of pipeline building
  recipe rules. If choose to do new/continued/refitted/retrained experiment from
  parent experiment, the recipe rules are not re-applied and any
  fine-tuning is preserved. To reset recipe behavior, one can switch
  between 'auto' and the desired mode.

enable_genetic_algorithm----------------------------  .. container:: dropdown     **Enable Genetic Algorithm for Selection and Tuning of Features and    Models**     Specify whether to enable :ref:`genetic algorithm <ga>` for selection    and hyper-parameter tuning of features and models:     -  **auto**: Default value is 'auto'. This is same as 'on' unless it       is a pure NLP or Image experiment. -  **on**: Driverless AI genetic algorithm is used for feature       engineering and model tuning and selection. -  **Optuna**: When 'Optuna' is selected, model hyperparameters are       tuned with :ref:`Optuna <num_inner_hyperopt_trials_prefinal>` and       Driverless AI genetic algorithm is used for feature engineering. In the Optuna case, the scores shown in the iteration panel are       the best score and trial scores. Optuna mode currently only uses       Optuna for XGBoost, LightGBM, and CatBoost (custom recipe).

TOURNAMENT_STYLE``

TOURNAMENT MODEL FOR GENETIC ALGORITHM
Select a method to decide which models are best at each iteration. This
is set to AUTO by default. Choose from the following:
-   AUTO: Choose based upon accuracy and interpretability
-   UNIFORM: all individuals in population compete to win as best (can
    lead to all, e.g. LightGBM models in final ensemble, which may not
    improve ensemble performance due to lack of diversity)
-   FULLSTACK: Choose from optimal model and feature types
-   FEATURE: individuals with similar feature types compete (good if
    target encoding, frequency encoding, and other feature sets lead to
    good results)
-   MODEL: individuals with same model type compete (good if multiple
    models do well but some models that do not do as well still
    contribute to improving ensemble)
For each case, a round robin approach is used to choose best scores
among type of models to choose from.

MAKE_PYTHON_SCORING_PIPELINE``

MAKE PYTHON SCORING PIPELINE

Specify whether to automatically build a Python Scoring Pipeline for the
experiment. Select ON or AUTO (default) to make the Python Scoring
Pipeline immediately available for download when the experiment is
finished. Select OFF to disable the automatic creation of the Python
Scoring Pipeline.

MAKE_MOJO_SCORING_PIPELINE``

MAKE MOJO SCORING PIPELINE

Specify whether to automatically build a MOJO (Java) Scoring Pipeline
for the experiment. Select ON to make the MOJO Scoring Pipeline
immediately available for download when the experiment is finished. With
this option, any capabilities that prevent the creation of the pipeline
are dropped. Select OFF to disable the automatic creation of the MOJO
Scoring Pipeline. Select AUTO (default) to attempt to create the MOJO
Scoring Pipeline without dropping any capabilities.

mojo_for_predictions------------------------  .. container:: dropdown     **Allow Use of MOJO for Making Predictions**     Specify whether to use MOJO for making fast, low-latency predictions    after the experiment has finished. When this is set to **Auto**    (default), the MOJO is only used if the number of rows is equal to or    below the value specified bymojo_for_predictions_max_rows``.

reduce_mojo_size--------------------  .. container:: dropdown     **Attempt to Reduce the Size of the MOJO (Small MOJO)**     Specify whether to attempt to create a small MOJO scoring pipeline    when the experiment is being built. A smaller MOJO leads to less    memory footprint during scoring. This setting attempts to reduce the    mojo size by limiting experiment's maximum    :ref:`interaction depth <max-feature-interaction-depth>` to **3**,    setting :ref:`ensemble level <fixed_ensemble_level>` to **0** i.e no    ensemble model for final pipeline and limiting the    :ref:`maximum number of features <nfeatures_max>` in the model to    **200**. Note that these settings in some cases can affect the    overall model's predictive accuracy as it is limiting the complexity    of the feature engineering and model building space.     This is disabled by default. The equivalent config.toml setting isreduce_mojo_size``

MAKE_PIPELINE_VISUALIZATION``

MAKE PIPELINE VISUALIZATION

Specify whether to create a visualization of the scoring pipeline at the
end of an experiment. This is set to AUTO by default. Note that the
VISUALIZE SCORING PIPELINE feature is experimental and is not available
for deprecated models. Visualizations are available for all newly
created experiments.

BENCHMARK_MOJO_LATENCY``

MEASURE MOJO SCORING LATENCY

Specify whether to measure the MOJO scoring latency at the time of MOJO
creation. This is set to AUTO by default. In this case, MOJO scoring
latency will be measured if the pipeline.mojo file size is less than 100
MB.

MOJO_BUILDING_TIMEOUT``

TIMEOUT IN SECONDS TO WAIT FOR MOJO CREATION AT END OF EXPERIMENT

Specify the amount of time in seconds to wait for MOJO creation at the
end of an experiment. If the MOJO creation process times out, a MOJO can
still be made from the GUI or the R and Python clients (the timeout
constraint is not applied to these). This value defaults to 1800 sec (30
minutes).

MOJO_BUILDING_PARALLELISM``

NUMBER OF PARALLEL WORKERS TO USE DURING MOJO CREATION

Specify the number of parallel workers to use during MOJO creation.
Higher values can speed up MOJO creation but use more memory. Set this
value to -1 (default) to use all physical cores.

KAGGLE_USERNAME``

KAGGLE USERNAME

Optionally specify your Kaggle username to enable automatic submission
and scoring of test set predictions. If this option is specified, then
you must also specify a value for the KAGGLE KEY option. If you don't
have a Kaggle account, you can sign up at https://www.kaggle.com.

KAGGLE_KEY``

KAGGLE KEY

Specify your Kaggle API key to enable automatic submission and scoring
of test set predictions. If this option is specified, then you must also
specify a value for the KAGGLE USERNAME option. For more information on
obtaining Kaggle API credentials, see
https://github.com/Kaggle/kaggle-api#api-credentials.

KAGGLE_TIMEOUT``

KAGGLE SUBMISSION TIMEOUT IN SECONDS

Specify the Kaggle submission timeout in seconds. This value defaults to
120 sec.

MIN_NUM_ROWS``

MIN NUMBER OF ROWS NEEDED TO RUN AN EXPERIMENT

Specify the minimum number of rows that a dataset must contain in order
to run an experiment. This value defaults to 100.

REPRODUCIBILITY_LEVEL``

REPRODUCIBILITY LEVEL

Specify one of the following levels of reproducibility. Note that this
setting is only used when the reproducible option is enabled in the
experiment:

-   1 = Same experiment results for same O/S, same CPU(s), and same
    GPU(s) (Default)
-   2 = Same experiment results for same O/S, same CPU architecture, and
    same GPU architecture
-   3 = Same experiment results for same O/S, same CPU architecture
    (excludes GPUs)
-   4 = Same experiment results for same O/S (best approximation)

This value defaults to 1.

SEED``

RANDOM SEED

Specify a random seed for the experiment. When a seed is defined and the
reproducible button is enabled (not by default), the algorithm will
behave deterministically.

ALLOW_DIFFERENT_CLASSES_ACROSS_FOLD_SPLITS``

ALLOW DIFFERENT SETS OF CLASSES ACROSS ALL TRAIN/VALIDATION FOLD SPLITS

(NOTE: Applicable for multiclass problems only.) Specify whether to
enable full cross-validation (multiple folds) during feature evolution
as opposed to a single holdout split. This is enabled by default.

SAVE_VALIDATION_SPLITS``

STORE INTERNAL VALIDATION SPLIT ROW INDICES

Specify whether to store internal validation split row indices. This
includes pickles of (train_idx, valid_idx) tuples (numpy row indices for
original training data) for all internal validation folds in the
experiment summary ZIP file. Enable this setting for debugging purposes.
This setting is disabled by default.

MAX_NUM_CLASSES``

MAX NUMBER OF CLASSES FOR CLASSIFICATION PROBLEMS

Specify the maximum number of classes to allow for a classification
problem. A higher number of classes may make certain processes more
time-consuming. Memory requirements also increase with a higher number
of classes. This value defaults to 200.

max_num_classes_compute_roc-------------------------------  .. container:: dropdown     **Max Number of Classes to Compute ROC and Confusion Matrix for    Classification Problems**     Specify the maximum number of classes to use when computing the ROC    and CM. When this value is exceeded, the reduction type specified byroc_reduce_type`` is applied. This value defaults to 200 and cannot

    be lower than 2.

max_num_classes_client_and_gui----------------------------------  .. container:: dropdown     **Max Number of Classes to Show in GUI for Confusion Matrix**     Specify the maximum number of classes to show in the GUI for CM,    showing firstmax_num_classes_client_and_gui`` labels. This value

    defaults to 10, but any value beyond 6 will result in visually
    truncated diagnostics. Note that if this value is changed in the
    config.toml and the server is restarted, then this setting will only
    modify client-GUI launched diagnostics. To control experiment plots,
    this value must be changed in the expert settings panel.

roc_reduce_type-------------------  .. container:: dropdown     **ROC/CM Reduction Technique for Large Class Counts**     Specify the ROC confusion matrix reduction technique used for large    class counts:     -  **Rows** (Default): Reduce by randomly sampling rows    -  **Classes**: Reduce by truncating classes to no more than the       value specified bymax_num_classes_compute_roc``

MAX_ROWS_CM_GA``

MAXIMUM NUMBER OF ROWS TO OBTAIN CONFUSION MATRIX RELATED PLOTS DURING
FEATURE EVOLUTION

Specify the maximum number of rows to obtain confusion matrix related
plots during feature evolution. Note that this doesn't limit final model
calculation.

USE_FEATURE_BRAIN_NEW_EXPERIMENTS``

WHETHER TO USE FEATURE BRAIN FOR NEW EXPERIMENTS

Specify whether to use feature_brain results even if running new
experiments. Feature brain can be risky with some types of changes to
experiment setup. Even rescoring may be insufficient, so by default this
is False. For example, one experiment may have training=external
validation by accident, and get high score, and while
feature_brain_reset_score='on' means we will rescore, it will have
already seen during training the external validation and leak that data
as part of what it learned from. If this is False, feature_brain_level
just sets possible models to use and logs/notifies, but does not use
these feature brain cached models.

FEATURE_BRAIN_LEVEL``

MODEL/FEATURE BRAIN LEVEL
Specify whether to use H2O.ai brain, which enables local caching and
smart re-use (checkpointing) of prior experiments to generate useful
features and models for new experiments. It can also be used to control
checkpointing for experiments that have been paused or interrupted. When enabled, this will use the H2O.ai brain cache if the cache file:
  -   has any matching column names and types for a similar experiment
      type
  -   has classes that match exactly
  -   has class labels that match exactly
  -   has basic time series choices that match
  -   the interpretability of the cache is equal or lower
  -   the main model (booster) is allowed by the new experiment
-   -1: Don't use any brain cache (default)
-   0: Don't use any brain cache but still write to cache. Use case:
    Want to save the model for later use, but we want the current model
    to be built without any brain models.

Use case:
    Want to use the latest matching model. The match may not be precise,
    so use with caution. -   2: Smart checkpoint if the experiment matches all column names,
    column types, classes, class labels, and time series options
    identically. Use case: Driverless AI scans through the H2O.ai brain
    cache for the best models to restart from. -   3: Smart checkpoint like level #1 but for the entire population. Tune only if the brain population is of insufficient size. Note that
    this will re-score the entire population in a single iteration, so
    it appears to take longer to complete first iteration. -   4: Smart checkpoint like level #2 but for the entire population. Tune only if the brain population is of insufficient size. Note that
    it appears to take longer to complete first iteration.

FEATURE_BRAIN2``

FEATURE BRAIN SAVE EVERY WHICH ITERATION
Save feature brain iterations every iter_num %
feature_brain_iterations_save_every_iteration == 0, to be able to
restart/refit with which_iteration_brain >= 0. This is disabled (0) by
default. -   -1: Don't use any brain cache. -   0: Don't use any brain cache but still write to cache. -   1: Smart checkpoint if an old experiment_id is passed in (for
    example, via running "resume one like this" in the GUI). -   2: Smart checkpoint if the experiment matches all column names,
    identically. (default)
-   3: Smart checkpoint like level #1 but for the entire population. Tune only if the brain population is of insufficient size. -   4: Smart checkpoint like level #2 but for the entire population. Tune only if the brain population is of insufficient size.

FEATURE_BRAIN3``

FEATURE BRAIN RESTART FROM WHICH ITERATION

When performing restart or re-fit of type feature_brain_level with a
resumed ID, specify which iteration to start from instead of only last
best. Available options include:

-   -1: Use the last best
-   1: Run one experiment with
    feature_brain_iterations_save_every_iteration=1 or some other number
-   2: Identify which iteration brain dump you wants to restart/refit
    from
-   3: Restart/Refit from the original experiment, setting
    which_iteration_brain to that number here in expert settings.

NOTE: If restarting from a tuning iteration, this will pull in the
entire scored tuning population and use that for feature evolution. This
value defaults to -1.

FEATURE_BRAIN4``

FEATURE BRAIN REFIT USES SAME BEST INDIVIDUAL

Specify whether to use the same best individual when performing a refit.
Disabling this setting allows the order of best individuals to be
rearranged, leading to a better final result. Enabling this setting lets
you view the exact same model or feature with only one new feature
added. This is disabled by default.

FEATURE_BRAIN5``

FEATURE BRAIN ADDS FEATURES WITH NEW COLUMNS EVEN DURING RETRAINING OF
FINAL MODEL

Specify whether to add additional features from new columns to the
pipeline, even when performing a retrain of the final model. Use this
option if you want to keep the same pipeline regardless of new columns
from a new dataset. New data may lead to new dropped features due to
shift or leak detection. Disable this to avoid adding any columns as new
features so that the pipeline is perfectly preserved when changing data.
This is enabled by default.

FORCE_MODEL_RESTART_TO_DEFAULTS``

RESTART-REFIT USE DEFAULT MODEL SETTINGS IF MODEL SWITCHES

When restarting or refitting, specify whether to use the model class's
default settings if the original model class is no longer available. If
this is disabled, the original hyperparameters will be used instead.
(Note that this may result in errors.) This is enabled by default.

MIN_DAI_ITERATIONS``

MIN DAI ITERATIONS

Specify the minimum number of Driverless AI iterations for an
experiment. This can be used during restarting, when you want to
continue for longer despite a score not improving. This value defaults
to 0.

target_transformer----------------------  .. container:: dropdown     **Select Target Transformation of the Target for Regression    Problems**     Specify whether to automatically select target transformation for    regression problems. Available options include:     -  auto    -  identity    -  identity_noclip    -  center    -  standardize    -  unit_box    -  log    -  log_noclip    -  square    -  sqrt    -  double_sqrt    -  inverse    -  logit    -  sigmoid     If set to **auto** (default), Driverless AI will automatically pick    the best target transformer if the **Accuracy** is set to the value    of thetune_target_transform_accuracy_switchconfiguration option    (defaults to 5) or larger. Selecting **identity_noclip**    automatically turns off any target transformations.

FIXED_NUM_FOLDS_EVOLUTION``

NUMBER OF CROSS-VALIDATION FOLDS FOR FEATURE EVOLUTION

Specify the fixed number of cross-validation folds (if >= 2) for feature
evolution. Note that the actual number of allowed folds can be less than
the specified value, and that the number of allowed folds is determined
at the time an experiment is run. This value defaults to -1 (auto).

FIXED_NUM_FOLDS``

NUMBER OF CROSS-VALIDATION FOLDS FOR FINAL MODEL

Specify the fixed number of cross-validation folds (if >= 2) for the
final model. Note that the actual number of allowed folds can be less
than the specified value, and that the number of allowed folds is
(auto).

FIXED_ONLY_FIRST_FOLD_MODEL``

FORCE ONLY FIRST FOLD FOR MODELS

Specify whether to force only the first fold for models. Select from
AUTO (Default), ON, or OFF. Set "on" to force only first fold for
models.This is useful for quick runs regardless of data

FEATURE_EVOLUTION_DATA_SIZE``

MAX NUMBER OF ROWS TIMES NUMBER OF COLUMNS FOR FEATURE EVOLUTION DATA
SPLITS

Specify the maximum number of rows allowed for feature evolution data
splits (not for the final pipeline). This value defaults to 100,000,000.

FINAL_PIPELINE_DATA_SIZE``

MAX NUMBER OF ROWS TIMES NUMBER OF COLUMNS FOR REDUCING TRAINING DATASET

Specify the upper limit on the number of rows times the number of
columns for training the final pipeline. This value defaults to
500,000,000.

MAX_VALIDATION_TO_TRAINING_SIZE_RATIO_FOR_FINAL_ENSEMBLE``

MAXIMUM SIZE OF VALIDATION DATA RELATIVE TO TRAINING DATA

Specify the maximum size of the validation data relative to the training
data. Smaller values can make the final pipeline model training process
quicker. Note that final model predictions and scores will always be
provided on the full dataset provided. This value defaults to 2.0.

FORCE_STRATIFIED_SPLITS_FOR_IMBALANCED_THRESHOLD_BINARY``

PERFORM STRATIFIED SAMPLING FOR BINARY CLASSIFICATION IF THE TARGET IS
MORE IMBALANCED THAN THIS

For binary classification experiments, specify a threshold ratio of
minority to majority class for the target column beyond which stratified
sampling is performed. If the threshold is not exceeded, random sampling
is performed. This value defaults to 0.01. You can choose to always
perform random sampling by setting this value to 0, or to always perform
stratified sampling by setting this value to 1.

config_overrides--------------------  .. container:: dropdown     **Add to config.toml via TOML String**     Specify any additional configuration overrides from the config.toml    file that you want to include in the experiment. (Refer to the    :ref:`sample-configtoml` section to view options that can be    overridden during an experiment.) Setting this will override all    other settings. Separate multiple config overrides withn. For    example, the following enables Poisson distribution for LightGBM and    disables Target Transformer Tuning. Note that in this example double    quotes are escaped ("
"``).

LAST_RECIPE``

LAST_RECIPE

Internal helper to allow memory of if changed recipe

FEATURE_BRAIN_RESET_SCORE``

WHETHER TO RE-SCORE MODELS FROM BRAIN CACHE
Specify whether to smartly keep score to avoid
re-munging/re-training/re-scoring steps brain models ('auto'), always
force all steps for all brain imports ('on'), or never rescore ('off'). 'auto' only re-scores if a difference in current and prior experiment
warrants re-scoring, like column changes, metric changes, etc. 'on' is
useful when smart similarity checking is not reliable enough. 'off' is
useful when know want to keep exact same features and model for final
model refit, despite changes in seed or other behaviors in features that
might change the outcome if re-scored before reaching final model. If
set off, then no limits are applied to features during brain ingestion,
while can set brain_add_features_for_new_columns to false if want to
ignore any new columns in data.

FEATURE_BRAIN_SAVE_EVERY_ITERATION``

Specify whether to save feature brain iterations every iter_num %
restart/refit with which_iteration_brain >= 0. Set to 0 to disable this
setting.

WHICH_ITERATION_BRAIN``

When performing restart or re-fit type feature_brain_level with
resumed_experiment_id, choose which iteration to start from, instead of
only last best -1 means just use last best.

Usage:

  -   1)  Run one experiment with
          feature_brain_iterations_save_every_iteration=1 or some other
          number

  -   2)  Identify which iteration brain dump one wants to restart/refit
          from

  -   3)  Restart/Refit from original experiment, setting
          which_iteration_brain to that number in expert settings

Note: If restart from a tuning iteration, this will pull in entire
scored tuning population and use that for feature evolution.

REFIT_SAME_BEST_INDIVIDUAL``

When doing re-fit from feature brain, if change columns or features,
population of individuals used to refit from may change order of which
was best, leading to better result chosen (False case). But sometimes
you want to see exact same model/features with only one feature added,
and then would need to set this to True case. That is, if refit with
just 1 extra column and have interpretability=1, then final model will
be same features, with one more engineered feature applied to that new
original feature.

RESTART_REFIT_REDO_ORIGFS_SHIFT_LEAK``

FOR RESTART-REFIT, SELECT WHICH STEPS TO DO
When doing restart or re-fit of experiment from feature brain, sometimes
user might change data significantly and then warrant redoing reduction
of original features by feature selection, shift detection, and leakage
detection. However, in other cases, if data and all options are nearly
(or exactly) identical, then these steps might change the features
slightly (e.g. due to random seed if not setting reproducible mode),
leading to changes in features and model that is refitted. By default,
restart and refit avoid these steps assuming data and experiment setup
have no changed significantly. If check_distribution_shift is forced to
on (instead of auto), then this option is ignored.

BRAIN_ADD_FEATURES_FOR_NEW_COLUMNS``

FEATURE BRAIN ADDS FEATURES WITH NEW COLUMNS EVEN DURING RETRAINING

Whether to take any new columns and add additional features to pipeline,
even if doing retrain final model. In some cases, one might have a new
dataset but only want to keep same pipeline regardless of new columns,
in which case one sets this to False. For example, new data might lead
to new dropped features, due to shift or leak detection. To avoid change
of feature set, one can disable all dropping of columns, but set this to
False to avoid adding any columns as new features, so pipeline is
perfectly preserved when changing data.

If restart/refit and no longer have the original model class available,
be conservative and go back to defaults for that model class. If False,
then try to keep original hyperparameters, which can fail to work in
general.

DUMP_MODELPARAMS_EVERY_SCORED_INDIV``

ENABLE DETAILED SCORED MODEL INFO

Whether to dump every scored individual's model parameters to
csv/tabulated/json file produces files. For example:
individual_scored.params.[txt, csv, json]

fast_approx_num_trees-------------------------  .. container:: dropdown     **Max number of trees to use for fast approximation**     Whenfast_approx=True, specify the maximum number of trees to    use. By default, this value is 250.        .. note::           By default,fast_approx`` is enabled for MLI and AutoDoc and

    disabled for Experiment predictions.

fast_approx_do_one_fold---------------------------  .. container:: dropdown     **Whether to use only one fold for fast approximation**     Whenfast_approx=True, specify whether to speed up fast    approximation further by using only one fold out of all    cross-validation folds. By default, this setting is enabled.        .. note::           By default,fast_approx`` is enabled for MLI and AutoDoc and

fast_approx_do_one_model----------------------------  .. container:: dropdown     **Whether to use only one model for fast approximation**     Whenfast_approx=True, specify whether to speed up fast    approximation further by using only one model out of all ensemble    models. By default, this setting is disabled.        .. note::           By default,fast_approx`` is enabled for MLI and AutoDoc and

fast_approx_contribs_num_trees----------------------------------  .. container:: dropdown     **Maximum number of trees to use for fast approximation when making    Shapley predictions**     Whenfast_approx_contribs=True, specify the maximum number of    trees to use for 'Fast Approximation' in GUI when making Shapley    predictions and for AutoDoc/MLI. By default, this value is 50.        .. note::           By default,fast_approx_contribs`` is enabled for MLI and

    AutoDoc.

fast_approx_contribs_do_one_fold------------------------------------  .. container:: dropdown     **Whether to use only one fold for fast approximation when making    Shapley predictions**     Whenfast_approx_contribs=True, specify whether to speed upfast_approx_contribsfurther by using only one fold out of all    cross-validation folds for 'Fast Approximation' in GUI when making    Shapley predictions and for AutoDoc/MLI. By default, this setting is    enabled.        .. note::           By default,fast_approx_contribs`` is enabled for MLI and

    AutoDoc.

fast_approx_contribs_do_one_model-------------------------------------  .. container:: dropdown     **Whether to use only one model for fast approximation when making    Shapley predictions**     Whenfast_approx_contribs=True, specify whether to speed upfast_approx_contribsfurther by using only one model out of all    ensemble models for 'Fast Approximation' in GUI when making Shapley    predictions and for AutoDoc/MLI. By default, this setting is enabled.        .. note::           By default,fast_approx_contribs`` is enabled for MLI and

    AutoDoc.

AUTOVIZ_RECOMMENDED_TRANSFORMATION``

AUTOVIZ RECOMMENDED TRANSFORMATIONS

Key-value pairs of column names and transformations that
Autoviz <autoviz_reco> recommended. Also see
Autoviz Recommendation Transformer
<autoviz_transformer>.

INSTALL ON AWS

Driverless AI can be installed on Amazon AWS using the AWS Marketplace
AMI or the AWS Community AMI.

choose-AWS aws-marketplace-ami aws-community-ami

When installing via AWS, you can also enable role-based authentication.

aws-role-based-authentication

USING DRIVERLESS AI CONFIGURATION OPTIONS

This page describes how to use DRIVERLESS AI (DAI) configuration
options. -   understanding-configs
-   understanding-expert-settings
-   toml_editor_using
-   expert-settings-use-case

Understanding DAI configuration options
Driverless AI features many different kinds of configuration options
that you can use to configure various aspects of your DAI environment,
including authentication, data connectors, UI, experiments, and MLI. The
following methods can be used to control the available DAI configuration
options:
-   Administrators can edit the config.toml file, which is a
    configuration file that uses the TOML v0.5.0 file format. The
    config.toml file lets you control all of the configuration options
    documented in the dai_config page. For more information, see
    config_file. -   Using the Expert Settings window, which is accessible from the
    EXPERIMENT SETUP page by clicking EXPERT SETTINGS.

-   Setting options as environment variables. Note
Some configuration options, such as those related to authentication and
data connectors, are applied when starting the DAI server and cannot be
changed without restarting the DAI server. Understanding Expert Settings
When creating an experiment, you can specify basic
settings for the experiment <experiment_settings> such as whether to
drop specific columns or whether to include a validation dataset. However, you may want to customize the experiment in a manner that is
beyond the scope of these basic settings—in this case, EXPERT SETTINGS
can be used to further fine-tune the experiment. For example, you can
use Expert Settings to include specific models or transformers as part
of the experiment. To open the Expert Settings window, click EXPERT
SETTINGS on the EXPERIMENT SETUP page. []
NOTES:
-   For supervised experiments, the Expert Settings window cannot be
    accessed until a target column has been selected.

-   If you change one or more settings in the EXPERT SETTINGS window,
    the experiment preview on the EXPERIMENT SETUP page automatically
    updates to reflect your changes. Navigating the Expert Settings window
The following sections describe how to navigate the Expert Settings
window. Tabbed view
When the TABBED view is selected, the available Expert Settings are
organized into the following tabs and sub-tabs. For each sub-tab in the
following list, the available settings are organized into COMMON and
ADVANCED settings. -   TRAINING: Configure settings related to the model training process. -   General
      -   Data
      -   Feature Engineering
      -   Models
      -   Genetic Algorithm
      -   Validation
      -   Deployment
-   DOCUMENTATION: Configure settings related to AutoDoc, model
    performance, and model interpretation.

(This tab has only one
    sub-tab that is also called System.) []
Tabbed view: sub-tabs
The following is a list of sub-tab level categories:
-   Common
-   Advanced
-   Image
-   NLP
-   Time Series
-   Unsupervised
Flat view
You can also select the FLAT view to view all of the available settings
in a single searchable window. This window uses the same sub-tab level
categorization described in the preceding section. Searching for specific settings
To locate a specific Expert Setting, click the search box and type the
configuration name of the Expert Setting you want to locate. For some
Expert Settings, additional results for related Expert Settings are also
displayed. Filtering settings by tags
To filter the list of available settings by specific tags, click the
FILTER BY TAGS button and select the checkbox next to the tag(s) that
you want to filter the list of available settings by. Note that both
global and sub-tab level filtering are supported.

The
information dialog provides the config name and a description of the
setting. _static/setting-info.mp4
Adding custom recipes
You can add custom recipes from the Expert Settings window by clicking
the ADD CUSTOM RECIPES button. Select one of the following options:
          Driverless AI repository. -   A GitHub tree.

-   A file system path. This option is equivalent to the FILE
          SYSTEM option when adding datasets. -   FROM BITBUCKET: Add a custom recipe from a Bitbucket repository. To
Note that you can also view the official Recipes for Driverless AI
repository from the Expert Settings window by clicking the OFFICIAL
RECIPES button. Using the built-in TOML config editor
The TOML configuration editor lets you manually add, remove, or edit
Expert Setting parameters. Note that for a given experiment, you can
only change the settings that are visible in the Expert Settings window,
not application-level settings. To open the built-in TOML configuration
editor, click EDIT TOML in the EXPERT SETTINGS window.

[]
The built-in TOML editor is synchronized with the EXPERT SETTINGS
window. This means that if you change the default value of an expert
setting from the EXPERT SETTINGS window, that change is displayed in the
TOML configuration editor. For example, if you set the MAKE MOJO SCORING
PIPELINE setting in the Experiment tab to OFF, then the line
make_mojo_scoring_pipeline = "off" is displayed in the TOML editor. Conversely, if you make changes using the TOML editor, those changes are
also visible from the EXPERT SETTINGS window. You can confirm that your
changes have been correctly entered into the editor by checking whether
the relevant settings have also changed in the EXPERT SETTINGS window. To confirm your changes, click SAVE. The experiment preview updates to
reflect your specified configuration changes. Warning
Do not edit the last_recipe setting or the section below the
[recipe_activation] line.

This is important for keeping experiments comparable when performing
retrain / refit operations. Note
- The settings listed in the dai_config page cannot be edited from the
built-in TOML editor unless they are exposed in the EXPERT SETTINGS
window. -   For information on TOML, see TOML v0.5.0. Order of settings in the TOML editor
When using the built-in TOML editor, ensure that settings are added in
the following order:
1. Booleans, integers, strings, and lists
2. Unprocessed dictionaries, which are automatically processed after
    clicking the SAVE button
3. Processed dictionaries
Checking TOML validity
The TOML Python library can be used to check the validity of your TOML
to avoid errors when using the built-in TOML editor. To install the TOML
Python library, run the following command:
    pip install toml
The following examples demonstrate how the TOML Python library can be
used to check whether your TOML is valid.

The toml.loads() function is then used to
    convert the string into a dictionary. -   ENTERING AN INVALID STRING: In the following example, an error is
    returned after attempting to convert the entered TOML string into a
    dictionary, which means that the entered string is not valid. Sample use case: Hyperparameter tuning
The following steps describe how to perform hyperparameter tuning by
using the params_tune_lightgbm Expert Setting. 1. On the EXPERIMENTS page, click the NEW EXPERIMENT button and select
    a training dataset to use for the experiment. 2. Select a target column and specify a test dataset to use for the
    experiment. 3. Click EXPERT SETTINGS to open the Expert Settings window. 4. Go to the RECIPES tab. For the INCLUDE SPECIFIC MODELS setting,
    click UNCHECK ALL and select LightGBM from the list of available
    models. Click DONE to confirm your selection. Completing this step
    lets you view how only LightGBM mutates. 5.

MODIFYING DATASETS WITH RECIPES

Driverless AI lets you create a new dataset by
modifying an existing dataset with a data recipe <modify_by_recipe>. This example shows you how to create a new dataset with the LIVE CODE
option. 1. Navigate to the Datasets page, then click on the dataset you want to
    modify. 2. Click DETAILS from the submenu that appears to open the Dataset
    Details page. 3. Click the MODIFY BY RECIPE button in the top right portion of the
    UI, then click LIVE CODE from the submenu that appears. 4. Enter the code for the data recipe you want to use to modify the
    dataset. Click the GET PREVIEW button to see a preview of how the
    data recipe will modify the dataset. In this example, the data
    recipe modifies the number of rows and columns in the dataset. 5. To download the entered code script as a .py file, click the
    DOWNLOAD button. 6. Click the APPLY button to confirm the changes and create a new
    dataset.

BEFORE YOU BEGIN

data-sampling missing-values-handling imputation-in-dai reproducibility
transformations internal-validation ensemble-learning
monotonicity-constraints leakage-shift-detection vi imbalanced-modeling
wide gpu-dai queuing dai-free-space ts_bestpractices tips-n-tricks
simple_configs

SUPPORTED ENVIRONMENTS

The following tables list the environments that support Driverless AI.

If your CPUs do not support AVX, then TensorFlow can
be turned off by setting the following configuration option:
enable_tensorflow_import=false. For install instructions, refer to linux. Windows 10 Pro, Enterprise, or Education
CAUTION: Windows computers (laptops in particular) should only be used
with small datasets for the purpose of exploring the software. For
serious use, server hardware is required. Consider spinning up a more
powerful instance in the cloud instead of using a laptop. Avoid laptops
with less than 16 GB of RAM. GPUs are not supported on Windows. --------------------------------------------------------------------
  Package    OS                              GPU        CPU   Min
  Type                                       Support?

OVERVIEW

H2O Driverless AI is an artificial intelligence (AI) platform for
automatic machine learning. Driverless AI automates some of the most
difficult data science and machine learning workflows, such as feature
engineering, model validation, model tuning, model selection, and model
deployment. It aims to achieve the highest predictive accuracy,
comparable to expert data scientists, but in a much shorter time thanks
to end-to-end automation. Driverless AI also offers automatic
visualization and machine learning interpretability (MLI). Especially in
regulated industries, model transparency and explanation are just as
important as predictive performance. Modeling pipelines (feature
engineering and models) are exported (in full fidelity, without
approximations) both as Python modules and as Java standalone scoring
artifacts.

Driverless AI runs on commodity hardware. It was also specifically
designed to take advantage of graphics processing units (GPUs),
including multi-GPU workstations and servers such as the NVIDIA DGX-1
for orders of magnitude faster training. This document describes how to install and use Driverless AI. For more
information about Driverless AI, see
https://www.h2o.ai/products/h2o-driverless-ai/. For a third-party review, see
https://www.infoworld.com/article/3236048/machine-learning/review-h2oai-automates-machine-learning.html. HAVE QUESTIONS? If you have questions about using Driverless AI, post them on Stack
Overflow using the driverless-ai tag at
http://stackoverflow.com/questions/tagged/driverless-ai. You can also post questions on the H2O.ai Community Slack workspace in
the #DRIVERLESSAI channel. If you have not signed up for the H2O.ai
Community Slack workspace, you can do so here:
https://www.h2o.ai/community/.

TRANSFORM DATASETS

When a training dataset is used in an experiment, Driverless AI
transforms the data into an improved, feature engineered dataset. (For
more information on the transformations that are provided in Driverless
AI, see Transformations.) But what happens when new rows are added to
your dataset? In this case, you can specify to transform the new dataset
after adding it to Driverless AI, and the same transformations that
Driverless AI applied to the original dataset are applied to these new
rows. The following sections describe the two options for transforming
datasets that are available in Driverless AI:
-   transform_dataset
-   fit_and_transform_dataset
NOTES:
-   To avoid leakage, the result of transformations should not be used
    for training unless enable_target_encoding='off'. []

Transform dataset
The following steps describe how to transform a dataset with the
TRANSFORM DATASET option, which transforms the dataset without fitting.

-   This transformation uses the experiment's full model pipeline,
    except instead of generating predictions, it generates the
    transformation before the model is applied. -   For ensembles of folds or models, for each row, numeric values are
    averaged, and one (last) value is taken for non-numeric values. 1. Select the dataset that you want to transform. 2. Select the columns you want to include in the transformation frame. To confirm your selection, click DONE. The dataset transformation
    job is added to the pending jobs queue. 3. When the transformed dataset is ready, click DOWNLOAD TRANSFORMED
    DATASET. Specify a filename for the dataset, then click the DOWNLOAD
    button to download the transformed dataset. Fit and transform dataset
The following steps describe how to transform a dataset with the FIT &
TRANSFORM DATASET option, which both fits and transforms the dataset. NOTES:
-   Don't use this option for training unless
    enable_target_encoding='off'.

(That is, when the lag-based recipe is
    used.) -   This functionality provides the pipeline (engineered features) of
    the best individual model of the experiment, not the full pipeline
    of all models and folds. 1. On the completed experiment page for the original dataset, click
    MODEL ACTIONS -> FIT & TRANSFORM DATASET. 2. Select the new training dataset that you want to transform. Note
    that this must have the same number of columns as the original
    dataset. 3. Select one of the following options:
      -   DEFAULT: The validation split ratio is set to 0. -   WITH VALIDATION DATASET: Specify a validation dataset to use
          with this dataset. The validation split ratio is set to 0.2. -   WITH TRAINING DATA SPLIT: Split the training data. The
          validation split ratio is set to 0.2. NOTE: To ensure that the transformed dataset respects the row
      order, choose a validation dataset instead of splitting the
      training data.

  then you will not be able to view MLI on that model after upgrading. Before upgrading, be sure to run MLI jobs on models that you want to
  be retained after upgrading.

Before upgrading, be sure to build MOJO
  tmp directory. NOTE: Stop Driverless AI if it is still running. REQUIREMENTS

series drivers. For reference on CUDA Toolkit and Minimum Required
Driver Versions and CUDA Toolkit and Corresponding Driver Versions, see
here . Note
If you are using K80 GPUs, the minimum required NVIDIA driver version is
450.80.02. UPGRADE STEPS

    machine:
3.

BLUEDATA DATATAP SETUP

This section provides instructions for configuring Driverless AI to work
with BlueData DataTap.

docker run --runtime=nvidia(>= Docker 19.03) ornvidia-docker(< Docker 19.03) command when starting the Driverless AI Docker image. Usedocker versionto check which version of Docker you are using. Description of Configuration Attributes ---------------------------------------  -dtap_auth_type: Selects DTAP authentication. Available values    are:        -noauth: No authentication needed       -principal: Authenticate with DataTap with a principal user       -keytab: Authenticate with a Key tab (recommended). If          running Driverless AI as a service, then the Kerberos keytab          needs to be owned by the Driverless AI user. -keytabimpersonation: Login with impersonation using a          keytab  -dtap_config_path: The location of the DTAP (HDFS) config folder    path. This folder can contain multiple config files.

For    use whendtap_auth_type=principal. -dtap_app_principal_user: The Kerberos app principal user    (recommended). -dtap_app_login_user: The user ID of the current user (for    example, user@realm). -dtap_app_jvm_args: JVM args for DTap distributions. Separate each    argument with spaces. -dtap_app_classpath: The DTap classpath. -dtap_init_path: Specifies the starting DTAP path displayed in the    UI of the DTAP browser. -enabled_file_systems: The file systems you want to enable. This    must be configured in order for data connectors to function properly. Example 1: Enable DataTap with No Authentication ------------------------------------------------  .. container:: tabs     .. group-tab:: Docker Image Installs     This example enables the DataTap data connector and disables    authentication. It does not pass any configuration file; however it    configures Docker DNS by passing the name and IP of the DTap name    node.

(**Note**: The trailing slash is    currently required for directories.) .. code:: bash        nvidia-docker run \         --pid=host \         --init \         --rm \         --shm-size=2g --cap-add=SYS_NICE --ulimit nofile=131071:131071 --ulimit nproc=16384:16384 \         --add-host name.node:172.16.2.186 \         -e DRIVERLESS_AI_ENABLED_FILE_SYSTEMS="file,dtap" \         -e DRIVERLESS_AI_DTAP_AUTH_TYPE='noauth'  \         -p 12345:12345 \         -v /etc/passwd:/etc/passwd \         -v /tmp/dtmp/:/tmp \         -v /tmp/dlog/:/log \         -v /tmp/dlicense/:/license \         -v /tmp/ddata/:/data \         -u $(id -u):$(id -g) \         h2oai/dai-ubi8-x86_64:|tag|  ..     .. container:: group-tab        Docker Image with the config.toml        This example shows how to configure DataTap options in the       config.toml file, and then specify that file when starting       Driverless AI in Docker. Note that this example enables DataTap       with no authentication. 1.

Set the following          configuration options:        ..           -enabled_file_systems = "file, upload, dtap"2. Mount the config.toml file into the Docker container.

This allows users to       reference data stored in DataTap directly using the name node       address, for example:dtap://name.node/datasets/iris.csvordtap://name.node/datasets/. (**Note**: The trailing slash is       currently required for directories.) 1. Export the Driverless AI config.toml file or add it to          ~/.bashrc. For example:        ..           ::              # DEB and RPM             export DRIVERLESS_AI_CONFIG_FILE="/etc/dai/config.toml"              # TAR SH             export DRIVERLESS_AI_CONFIG_FILE="/path/to/your/unpacked/dai/directory/config.toml"         2. Specify the following configuration options in the config.toml          file. ..           ::              # File System Support             # upload : standard upload feature             # dtap : Blue Data Tap file system, remember to configure the DTap section below             enabled_file_systems = "file, dtap"        3.

Example 2: Enable DataTap with Keytab-Based Authentication ----------------------------------------------------------  **Notes**:  -  If using Kerberos Authentication, the the time on the Driverless AI    server must be in sync with Kerberos server. If the time difference    between clients and DCs are 5 minutes or higher, there will be    Kerberos failures. -  If running Driverless AI as a service, then the Kerberos keytab needs    to be owned by the Driverless AI user; otherwise Driverless AI will    not be able to read/access the Keytab and will result in a fallback    to simple authentication and, hence, fail. .. container:: tabs     .. group-tab:: Docker Image Installs     This example:     -  Places keytabs in the/tmp/dtmpfolder on your machine and       provides the file path as described below. -  Configures the environment variableDRIVERLESS_AI_DTAP_APP_PRINCIPAL_USERto reference a user for       whom the keytab was created (usually in the form of user@realm).

-  Configures the optiondtap_app_prinicpal_userto reference          a user for whom the keytab was created (usually in the form of          user@realm). 1. Configure the Driverless AI config.toml file. Set the following          configuration options:        ..           -enabled_file_systems = "file, upload, dtap"-dtap_auth_type = "keytab"-dtap_key_tab_path = "/tmp/<keytabname>"-dtap_app_principal_user = "<user@kerberosrealm>"2. Mount the config.toml file into the Docker container.

-  Configures the optiondtap_app_prinicpal_userto reference          a user for whom the keytab was created (usually in the form of          user@realm). 1. Export the Driverless AI config.toml file or add it to          ~/.bashrc. For example:        ..           ::              # DEB and RPM             export DRIVERLESS_AI_CONFIG_FILE="/etc/dai/config.toml"              # TAR SH             export DRIVERLESS_AI_CONFIG_FILE="/path/to/your/unpacked/dai/directory/config.toml"         2. Specify the following configuration options in the config.toml          file. ..           ::              # File System Support             # file : local file system/server file system             # dtap : Blue Data Tap file system, remember to configure the DTap section below             enabled_file_systems = "file, dtap"              # Blue Data DTap connector settings are similar to HDFS connector settings.

If running             #             DAI as a service, then the Kerberos keytab needs to             #             be owned by the DAI user. #   keytabimpersonation : Login with impersonation using a keytab             dtap_auth_type = "keytab"              # Path of the principal key tab file             dtap_key_tab_path = "/tmp/<keytabname>"              # Kerberos app principal user (recommended)             dtap_app_principal_user = "<user@kerberosrealm>"        3. Save the changes when you are done, then stop/restart          Driverless AI. Example 3: Enable DataTap with Keytab-Based Impersonation ---------------------------------------------------------  **Notes**:  -  If using Kerberos, be sure that the Driverless AI time is synched    with the Kerberos server. -  If running Driverless AI as a service, then the Kerberos keytab needs    to be owned by the Driverless AI user.

-  Configures theDRIVERLESS_AI_DTAP_APP_PRINCIPAL_USERvariable,       which references a user for whom the keytab was created (usually       in the form of user@realm). -  Configures theDRIVERLESS_AI_DTAP_APP_LOGIN_USERvariable,       which references a user who is being impersonated (usually in the       form of user@realm).

-  Configures thedtap_app_principal_uservariable, which          references a user for whom the keytab was created (usually in          the form of user@realm). -  Configures thedtap_app_login_uservariable, which          references a user who is being impersonated (usually in the          form of user@realm). 1. Configure the Driverless AI config.toml file. Set the following          configuration options:        ..           -enabled_file_systems = "file, upload, dtap"-dtap_auth_type = "keytabimpersonation"-dtap_key_tab_path = "/tmp/<keytabname>"-dtap_app_principal_user = "<user@kerberosrealm>"-dtap_app_login_user = "<user@realm>"2. Mount the config.toml file into the Docker container.

INSTALL ON RHEL

This section describes how to install the Driverless AI Docker image on
RHEL. The installation steps vary depending on whether your system has
GPUs or if it is CPU only. Environment
  -------------------------------------------
  Operating System          GPUs? Min Mem
  ------------------------- ------- ---------
  RHEL with GPUs            Yes     64 GB
  RHEL with CPUs            No      64 GB

Install on RHEL with GPUs
NOTE: Refer to the following links for more information about using RHEL
with GPUs. These links describe how to disable automatic updates and
specific package updates. This is necessary in order to prevent a
mismatch between the NVIDIA driver and the kernel, which can lead to the
GPUs failures. -   https://access.redhat.com/solutions/2372971
  -   https://www.rootusers.com/how-to-disable-specific-package-updates-in-rhel-centos/
WATCH THE INSTALLATION VIDEO here.

Note
As of this writing, Driverless AI has been tested on RHEL versions 7.4,
8.3, and 8.4. Open a Terminal and ssh to the machine that will run Driverless AI. Once
you are logged in, perform the following steps. 1. Retrieve the Driverless AI Docker image from
    https://www.h2o.ai/download/. 2. Install and start Docker EE on RHEL (if not already installed). Follow the instructions on
    https://docs.docker.com/engine/installation/linux/docker-ee/rhel/. 3. Install nvidia-docker2 (if not already installed). More information
    is available at
    https://github.com/NVIDIA/nvidia-docker/blob/master/README.md. 4. Verify that the NVIDIA driver is up and running. If the driver is
    not up and running, log on to
    http://www.nvidia.com/Download/index.aspx?lang=en-us to get the
    latest NVIDIA Tesla V/P/K series driver. 5. Set up a directory for the version of Driverless AI on the host
    machine:
6.

Enable persistence of the GPU. Note that this needs to be run once
    http://docs.nvidia.com/deploy/driver-persistence/index.html. 8. Set up the data, log, and license directories on the host machine
    (within the new directory):
9. At this point, you can copy data into the data directory on the host
    machine. The data will be visible inside the Docker container. 10. Run docker images to find the image tag. 11. Start the Driverless AI Docker image and replace TAG below with the
    Docker 19.03) command. Note that from version 1.10 DAI docker image
    runs with internal tini that is equivalent to using --init from
    docker, if both are enabled in the launch command, tini will print a
    (harmless) warning message.

    http://Your-Driverless-AI-Host-Machine:12345. Install on RHEL with CPUs
This section describes how to install and start the Driverless AI Docker
image on RHEL. Note that this uses docker and not nvidia-docker. WATCH THE INSTALLATION VIDEO here. Note that some of the images in this
same. Note
you are logged in, perform the following steps. 1. Install and start Docker EE on RHEL (if not already installed). Follow the instructions on
    https://docs.docker.com/engine/installation/linux/docker-ee/rhel/. 2. On the machine that is running Docker EE, retrieve the Driverless AI
    Docker image from https://www.h2o.ai/download/. 3.

Load the Driverless AI Docker image inside the new directory:
5. Set up the data, log, license, and tmp directories (within the new
    directory):
6. Copy data into the DATA directory on the host. The data will be
    visible inside the Docker container at /<USER-HOME>/DATA. 7. Run docker images to find the image tag. 8. Start the Driverless AI Docker image. Note that GPU support will not
    be available. Note that from version 1.10 DAI docker image runs with
    internal tini that is equivalent to using --init from docker, if
    both are enabled in the launch command, tini will print a (harmless)
    warning message. 9. Connect to Driverless AI with your browser at
    http://Your-Driverless-AI-Host-Machine:12345. Stopping the Docker Image
Driverless AI Docker image.

These steps ensure that
  that model after upgrading.

NOTE: Stop Driverless AI if it is still running. Requirements
here . Note
Upgrade Steps
    machine:
5.

HIVE SETUP

Driverless AI lets you explore Hive data sources from within the
Driverless AI application. This section provides instructions for
configuring Driverless AI to work with Hive.

docker run --runtime=nvidia(>= Docker 19.03) ornvidia-docker(< Docker 19.03) command when starting the Driverless AI Docker image. Usedocker versionto check which version of Docker you are using. Description of Configuration Attributes ---------------------------------------  -enabled_file_systems: The file systems you want to enable. This    must be configured in order for data connectors to function properly. -hive_app_configs: Configuration for Hive Connector. Inputs are    similar to configuring the HDFS connector. Important keys include:     -hive_conf_path: The path to Hive configuration. This can have       multiple files (e.g. hive-site.xml, hdfs-site.xml, etc.)

However, if a non-edge node is being used, then there could be    issues (e.g. missing classes, dependencies, authorization errors). -  Ensure the core-site.xml file (from e.g Hadoop conf) is also       present in the Hive conf with the rest of the files       (hive-site.xml, hdfs-site.xml, etc.). The core-site.xml file       should have proxyuser configured (e.g.hadoop.proxyuser.hive.hosts&hadoop.proxyuser.hive.groups). -  If you have tez as the Hive execution engine, make sure that the       required tez dependencies (classpaths, jars, etc.) are available       on the DAI node. Alternatively, you can use internal engines that       come with DAI by changing yourhive.execution.enginevalue in       the hive-site.xml file tomrorspark. The configuration should be JSON/Dictionary String with multiple keys.

Double    quotation marks ("...") must be used to denote keys and values    *within* the JSON dictionary, and *outer* quotations must be    formatted as either""",''', or'. Depending on how the    configuration value is applied, different forms of outer quotations    may be required. The following examples show two unique methods for    applying outer quotations. -  Configuration value applied with the config.toml file:        ..           ::              hive_app_configs = """{"my_json_string": "value", "json_key_2": "value2"}"""        -  Configuration value applied with an environment variable:        ..           ::              DRIVERLESS_AI_HIVE_APP_CONFIGS='{"my_json_string": "value", "json_key_2": "value2"}'  -hive_app_jvm_args: Optionally specify additional Java Virtual    Machine (JVM) args for the Hive connector. Each arg must be separated    by a space.

nvidia-docker run`` command or by editing the configuration options in
the config.toml file and then specifying that file in the

nvidia-docker run`` command.

Docker Image Installs

1.  Start the Driverless AI Docker Image.

ADDING DATASETS USING HIVE

After the Hive connector is enabled, you can add datasets by selecting
HIVE from the ADD DATASET (OR DRAG AND DROP) drop-down menu.

1.  Select the Hive configuraton that you want to use.

2.  Specify the following information to add your dataset.

DRIVERLESS AI RELEASE BLOGS

Looking for the latest news on H2O Driverless AI releases? Find it here
in a single convenient location. Driverless AI 1.10.4
Version 1.10.4 brings several new features that make it simpler for you
to take advantage of the predictive modeling capabilities of DAI. For a
full list of changes and accompanying documentation, see version_1104. READ MORE: WHAT'S NEW IN VERSION 1.10.4
DRIVERLESS AI GUI-BASED WIZARDS
Several new GUI-based wizards have been added to DAI as part of this
release. -   _Experiment wizard:_ This wizard guides you step-by-step through to
    process of setting up and starting an experiment. For users who
    aren't already familiar with using DAI, the experiment wizard is a
    great way to start running experiments without having to worry about
    whether you've set up your experiment correctly.

Finally, if you're
      used to only setting up a particular type of experiment, you can
      use this wizard to confidently try creating different types of
      experiments, like time series or NLP experiments. To access the experiment wizard, go to the EXPERIMENTS page and
      click NEW EXPERIMENT -> WIZARD SETUP. -   _Dataset join wizard:_ The process of joining two datasets together
    can sometimes be difficult, depending on the size and complexity of
    the datasets. This wizard guides you through this process so that
    you can be sure that the datasets are joined correctly. To access the Dataset Join Wizard, go to the DATASETS page and
      click on the name of the dataset, then click JOIN WIZARD from the
      list of options. -   _Leaderboard wizard:_ This wizard helps you set up and perform a
    business value analysis of all models in a project. To access the
    Leaderboard wizard, go to a project and click the ANALYZE RESULTS
    button.

By clicking the FILTER BY TAGS button, you can now also
filter the list of available settings by specific tags. This is a useful
way of ensuring that the settings being displayed are relevant to your
particular use case. []
IMPROVED MODEL ACCURACY
H2O.ai developers conducted extensive research on the model accuracy of
Driverless AI for multiple open source datasets, with a particular
emphasis on the effectiveness of each phase of the experiment process. This research resulted in the identification of several improvements to
the experiment process to make the experiment runtime considerably more
effective. Driverless AI now consistently drops models that have poor performance
so that time is not wasted on unproductive models. We have also found
more productive uses for certain feature transformers and algorithms,
resulting in improved model performance across the board. These changes
are especially helpful when working with smaller datasets.

LEADERBOARDS

Driverless AI provides a feature to automatically create leaderboards. The CREATE LEADERBOARD feature runs multiple diverse experiments that
provide an overview of the dataset. This feature also provides you with
relevant information for deciding on complexity, accuracy, size, and
time tradeoffs when putting models into production. Refer to the
expert-settings topic for information on expert settings that can be
used to control this feature. For more information on the default models
built for a leaderboard, see leaderboard_models. The built models are placed under the projects page and can be
simultaneously scored on the test dataset and compared. Creating a Leaderboard
Creating a Leaderboard is similar to running a
new experiment <new_experiment>. Refer to the experiment_settings,
expert-settings, and scorers topics for more information about options
you can set when running an experiment. 1.

2. Specify whether to include dropped columns, a validation dataset,
    and a testing dataset. 3. Specify the Target column and optionally a fold column, weight
    column, and time column. 4. Optionally specify expert-settings. 5. Optionally adjust the Accuracy/Time/Interpretability knobs. 6. Optionally override the default scorer. 7. Optionally override the Classification/Regression setting. 8. Optionally specify to make the experiments reproducible and/or
    whether to enable GPUs. 9. Click the CREATE LEADERBOARD button. []
Driverless AI creates a new, randomly named project and begins
automatically training models using the queuing mechanism. The new
project is given the description "Automatic Leader Board". After all
models have been built, you can
score each experiment <leaderboard_scoring> and
compare experiments <comparing_experiments>, as described in the
projects topic.

Regular Experiments
By default, the following models are built when creating a leaderboard
for regular (non-time-series) experiments. You can omit models from
being built by disabling those models in the expert-settings.

time_series_leaderboard_modeTOML config. Specify one of the following options:  -diverse: Explore a diverse set of models built using various    expert settings. Note that you can rerun another diverse leaderboard    on top of the best-performing model(s). -sliding_window: If the forecast horizon is *N* periods, create a    separate model for each of the (gap, horizon) pairs of (0,n), (n,n),    (2*n,n), ..., (2*N-1, n) in units of time periods. The number of    periods to predict per model *n* is controlled by the expert settingtime_series_leaderboard_periods_per_model, which defaults to 1. **Notes**:  -  Lag-based features are always included for time-series baseline    models. -  Unlike in regular experiments, the leaderboard will only build models    using the specified Accuracy/Time/Interpretability settings. -  For information about each transformer, see :ref:`Transformations`.

PYTHON CLIENT

This section describes how to install the Driverless AI Python client.
Several end-to-end examples that demonstrate how to use the client are
also provided. Additional examples are available in the Driverless AI
Code Samples and Tutorials GitHub repository.

For more information on the Python client, see the Driverless AI Python
client documentation.

Note

The Python client does not currently support the following Driverless AI
features:

-   Diagnostics
-   Deployments
-   MLI Bring Your Own Recipe (BYOR)
-   mTLS authentication

python_install_client python_client_admin
examples/credit_card/credit_card_default.ipynb
examples/walmart_timeseries_experiment/training_timeseries_model.ipynb
examples/stock_timeseries_experiment/demo_stock_timeseries.ipynb
examples/nlp_airline_sentiment/demo_nlp_airline_sentiment.ipynb

USING AUTODOC

The following sections describe Driverless AI's AutoDoc feature. -   understanding-autodoc
-   generate-autodoc
-   configure-autodoc
-   autodoc-custom

Understanding AutoDoc
The AutoDoc feature is used to generate automated machine learning
documentation for individual Driverless AI experiments. This editable
document contains an overview of the experiment and includes other
significant details like feature engineering and final model
performance. To download and view a sample experiment report in Word format,
click here <sample_report.docx>. AutoDoc Support
AutoDoc only supports resumed experiments for certain Driverless AI
versions. See the following table to check the types of resumed
experiments that are supported for your version:
    ---------------------------------------------------------------------
    AutoDoc Support for Resumed        1.7.0 and    1. 1.9.0 and later
    Experiments Via                    older        7.

- AutoDoc
does not support experiments that were built off of previously aborted
or failed experiments. - Reports for unsupported resumed experiments
will still build, but they will only include the following text:
"AutoDoc not yet supported for resumed experiments." Custom AutoDocs
All Driverless AI experiments can generate either a standard or custom
AutoDoc. A standard AutoDoc uses the default AutoDoc template that is
included with Driverless AI, while a custom AutoDoc uses a
customer-specific template that Driverless AI automatically populates. If you are interested in creating a custom AutoDoc, contact
support@h2o.ai. If you have already purchased a custom AutoDoc template
and want to learn how to generate custom AutoDocs from your experiments,
see autodoc-custom. Note
- For a list of custom AutoDoc placeholders, see autodoc_placeholders. -
Custom AutoDocs are Driverless AI version-specific.

Custom scorers and transformers are documented
the same as Driverless AI scorers and transformers. If Driverless AI
used a custom transformer, is is included in the Feature Transformations
table under its display name; otherwise it is only included in the
Feature Evolution section. (NOTE: custom-transformer descriptions are
currently shown as "None" in this section.) For custom models, the
standard performance metrics and plots are included; however,
information that Driverless AI cannot access is not included, or is
shown as "custom", "unavailable", or "auto." For example, in the Model
Tuning table, the booster is listed as "custom", and in the Alternative
Models section, the model package documentation is listed as
"unavailable."

-   These approaches also apply to custom AutoDocs. For more
    information, see autodoc-custom. Experiment UI
Navigate to the EXPERIMENTS page and click on the completed experiment
you want to generate an AutoDoc for. If AutoDoc was not previously enabled for the experiment, click the
BUILD AUTODOC button. []
If AutoDoc was previously enabled for the experiment, click the DOWNLOAD
AUTODOC button. []
MLI UI
Navigate to the MLI page and click on the completed experiment you want
to generate an AutoDoc for. Select AutoDoc from the MLI RECIPES's menu and optionally select
explainers that can be included in the AutoDoc (the standard AutoDoc
supports the k-LIME Explainer and DT Surrogate Explainer).

',
        mli_key:str='',
        individual_rows:list=[], 
        external_dataset_keys:list=[])
To save an AutoDoc to the DAI experiment directory (recommended if local
downloads are disabled):
    make_autodoc_sync(
        model_key:str,
        template_path:str='',
        config_overrides:str='',
-   model_key: The experiment key string. -   template_path: The full path the custom AutoDoc template. -   config_overrides: The TOML string format with configurations
    overrides for the AutoDoc. -   dest_path: The local path where the AutoDoc should be saved. -   mli_key: The mli key string. -   individual_rows: List of row indices for rows of interest in the
    training dataset, for which additional information can be shown
    (ICE, LOCO, KLIME). -   external_dataset_keys: List of DAI dataset keys.

',
    )

Configuring AutoDoc
The plots, tables, and sections of an AutoDoc can be configured through
four different workflows:
-   config-experiment-expert
-   config-mli-expert
-   config-python-client
-   config.toml file <config_file>
You can also configure the font for the AutoDoc plots <autodoc-font> by
setting the H2O_AUTODOC_PLOTS_FONT_FAMILY environment variable. Experiment UI Expert Settings
[]
MLI UI Expert Settings
[]
Python Client
-   driverlessai <config-driverlessai>
All configuration options for the AutoDoc are listed in the
config.toml file <config_file>.

',
    )
Configuring AutoDoc font
The following sections describe how to configure the AutoDoc font
environment variable. NOTE: The following steps assume that DAI has been installed on an EC2
instance or an Ubuntu lab machine. These steps still apply if you are
using H2O Enterprise Puddle to run a DAI instance—just log in to the EC2
instance where the DAI service is running using the provided SSH key. If the DAI service has not been started
1. Create an EC2 instance with enough memory and storage to run DAI. 2. Install the font you want to use. In this example, the font
    TakaoPGothic is used. 3. Create and install the DAI debian file. 4. Set the font setting environment variable by adding the following
    line to the EnvironmentFile.conf file. 5. Start the DAI service. If the DAI service has already been started
1. Ensure that the font is available on your system. In this example,
    the font TakaoPGothic is used. 2. Stop the DAI service. 3.

config.tomlsettings:  -autodoc_template: Specify the path for the main template file. -autodoc_additional_template_folder: If you have additional custom    sub-templates, use this setting to specify the location of additional    AutoDoc templates. Note that if this field is left empty, only the    default sub-templates folder is used. To generate custom AutoDocs, Driverless AI must have access to the custom template(s). To make sure that Driverless AI has access, update the path in the following example with your own path:  .. code::      autodoc_template="/full/path/to/your/custom_autodoc_template.docx"     # Required if you have additional custom sub-templates.

FEATURE COUNT CONTROL

This page describes how to control feature counts during the feature
selection process in H2O Driverless AI (DAI). -   original_feature_control
-   transformed_feature_control
-   individuals_control
-   feature_count_use_case

Original Feature Control
To control the count of original features when creating an experiment,
use one of the following methods:
-   On the EXPERIMENT SETUP page, click DROPPED COLUMNS to manually
    select specific columns to drop. -   Use the Features to Drop <features_to_drop> Expert Setting to enter
    a list of features to drop.

-   max_orig_cols_selected: Specify the maximum number of columns to
        be selected from the set of original columns using FS by
        permutation importance. -   max_orig_numeric_cols_selected: This option has the same
        functionality as max_orig_cols_selected, but for numeric
        columns. -   max_orig_nonnumeric_cols_selected: This option has the same
        functionality as max_orig_cols_selected, but for non-numeric
        columns. -   To view a report about original features without any action, set
    orig_features_fs_report = true.

In some cases, however, too few or too many features
are made. To control the number of transformed features that are made during an
experiment, use the nfeatures_max and ngenes_max settings. The term
_gene_ in ngenes_max refers to the number of _transformers_ used, while
the term _feature_ in nfeatures_max refers to the _transformed
features_. These settings can be used to control the number of allowed
transformers and transformed features by setting a limit beyond which
transformed features or transformers are removed. (The transformed
features or transformers with the lowest variable importance are removed
first.) In some cases, specifying nfeatures_max and ngenes_max may be sufficient
to get a restricted model. However, the best practice when using these
settings is to first run an experiment without specifying any
restrictions, and then retrain the final pipeline with the restrictions
enabled.

ngenes_min`` parameter forces DAI to search more actively at the cost of
more features per model. DAI automatically sets limits on the number of features that are made.

enable_genetic_algorithm='off'.) .. _individuals_control:  Individuals Control -------------------  You can control the number or type of individuals that are tuned or evolved by using the following config.toml parameters:  .. code::      parameter_tuning_num_models    fixed_num_individuals  .. _feature_count_use_case:  Sample Use Case ---------------  The following is a sample use case for controlling feature counts. **Example**:  You want to limit the number of features used for scoring to 14. **Solution A**:  -  For transformed features, setnfeatures_max
=
14in the    :ref:`Expert Settings window <understanding-expert-settings>`. -  For original features, set the following parameters:  ..     .. code::         max_orig_cols_selected       max_orig_numeric_cols_selected       max_orig_nonnumeric_cols_selected  **Solution B**  Without changing any parameters, let DAI complete the experiment.

KEY FEATURES

Below are some of the key features available in Driverless AI. Flexibility of Data and Deployment
Driverless AI works across a variety of data sources, including Hadoop
HDFS, Amazon S3, and more. Driverless AI can be deployed everywhere,
including all clouds (Microsoft Azure, AWS, and Google Cloud),
on-premises, and can run on machines with only CPUs or machines with
CPUs and GPUs. NVIDIA GPU Acceleration
Driverless AI is optimized to take advantage of GPU acceleration to
achieve up to 40X speedups for automatic machine learning. It includes
multi-GPU algorithms for XGBoost, GLM, K-Means, and more. GPUs allow for
thousands of iterations of model features and optimizations and give
significant speedups for use cases involving images and/or text. For
more information, see gpu_in_dai.

These visualizations help users
get a quick understanding of their data prior to starting the model
building process. They are also useful for understanding the composition
of very large datasets and for seeing trends or even possible issues,
such as large numbers of missing values or significant outliers that
could impact modeling results. For more information, see
Visualizing Datasets <automatic-visualization>. Automatic Feature Engineering
Feature engineering is the secret weapon that advanced data scientists
use to extract the most accurate results from algorithms. H2O Driverless
AI employs a library of algorithms and feature transformations to
automatically engineer new, high-value features for a given dataset. (See transformations for more information.) Included in the interface is
a variable importance chart that shows the significance of original and
newly engineered features.

Driverless AI provides an AutoDoc for
each experiment, relieving the user from the time-consuming task of
documenting and summarizing their workflow used when building machine
learning models. The AutoDoc includes details about the data used, the
validation schema selected, model and feature tuning, and the final
model created. With this capability in Driverless AI, practitioners can
focus more on drawing actionable insights from the models and save weeks
or even months in development, validation, and deployment. Driverless AI also provides a number of autodoc_ configuration options,
giving users even more control over the output of the AutoDoc. (Refer to
the sample-configtoml topic for information about these configuration
options.) Click here <sample_report.docx> to download and view a sample experiment
report in Word format. Time Series Forecasting
Time series forecasting is one of the biggest challenges for data
scientists.

Driverless AI delivers superior time series capabilities to optimize for
almost any prediction time window. Driverless AI incorporates data from
numerous predictors, handles structured character data and
high-cardinality categorical variables, and handles gaps in time series
data and other missing values. For more information, see
time-series-in-dai. NLP with TensorFlow and PyTorch
Text data can contain critical information to inform better predictions. Driverless AI automatically converts text strings into features using
powerful techniques like TFIDF and Embeddings. With TensorFlow and
PyTorch, Driverless AI can process large text blocks and build models
using all the available data to solve business problems like sentiment
analysis, document classification, and content tagging. The Driverless
AI platform has the ability to support both standalone text and text
with other columns as predictive features. For more information, see
nlp-in-dai.

It
supports the use of both standalone images and images together with
other data types as predictive features. For more information, see
image-processing-in-dai. Machine Learning Interpretability (MLI)
MLI view, Driverless AI employs a host of different techniques and
methodologies for interpreting and explaining the results of its models. A number of charts are generated automatically (depending on experiment
type), including K-LIME, Shapley, Variable Importance, Decision Tree
Surrogate, Partial Dependence, Individual Conditional Expectation,
Sensitivity Analysis, NLP Tokens, NLP LOCO, and more. Additionally, you
can download a CSV of LIME and Shapley reasons codes from the MLI page. For more information, see interpreting_a_model.

Reason codes show the key positive and negative factors in a
model's scoring decision in a simple language. Reasons codes are also
useful in other industries, such as healthcare, because they can provide
insights into model decisions that can drive additional testing or
investigation. For more information, see mli-explanations. Custom Recipe Support
Driverless AI lets you import custom recipes for MLI algorithms, feature
engineering (transformers), scorers, and configuration. You can use your
custom recipes in combination with or instead of all the built-in
recipes. This lets you have greater influence over the Driverless AI
Automatic ML pipeline and gives you control over the optimization
choices that Driverless AI makes. For more information, see
custom-recipes.

EXPLAINER (RECIPES) EXPERT SETTINGS

The following is a list of the explainer-specific expert settings that
are available when setting up a new interpretation. These settings can
be accessed when running interpretation from the
MLI page <mli_expert_settings> under recipes <mli_default_recipes> tab.
For info on general MLI expert settings, see
interpretation-expert-settings.

-   interpretation-expert-settings-absolute-permutation
-   interpretation-expert-settings-autodoc
-   interpretation-expert-settings-dia
-   interpretation-expert-settings-nlp-pdp
-   interpretation-expert-settings-nlp-vectorizer
-   interpretation-expert-settings-pdp
-   interpretation-expert-settings-sa
-   interpretation-expert-settings-shapley
-   interpretation-expert-settings-shapley-values
-   interpretation-expert-settings-surrogate-dt

Absolute Permutation Feature Importance Explainer Settings

MLI_SAMPLE_SIZE``

SAMPLE SIZE

Specify the sample size for the absolute permutation feature importance
explainer. This value defaults to 100000.

missing_values~~~~~~~~~~~~~~~~~~  .. container:: dropdown     **List of values that should be interpreted as missing values**     Specify the list of values that should be interpreted as missing    values during data import. This applies to both numeric and string    columns. Note that 'nan' is always interpreted as a missing value for    numeric columns.     Example:"""['',
'?', 'None', 'nan', 'N/A', 'unknown', 'inf']""``

AUTODOC_FEATURE_IMPORTANCE_NUM_PERM``

NUMBER OF PERMUTATIONS FOR FEATURE IMPORTANCE

Specify the number of permutations to make per feature when computing
feature importance. This is set to 1 by default.

AUTODOC_FEATURE_IMPORTANCE_SCORER``

FEATURE IMPORTANCE SCORER

Specify the name of the scorer to be used when calculating feature
importance. Leave this setting unspecified to use the default scorer for
the experiment.

MLI AutoDoc Explainer Settings

AUTODOC_REPORT_NAME``

AUTODOC NAME

Specify the name of the AutoDoc.

AUTODOC_TEMPLATE``

AUTODOC TEMPLATE LOCATION

Specify the AutoDoc template path. Provide the full path to your custom
AutoDoc template. To generate the standard AutoDoc, leave this field
empty.

autodoc_output_type~~~~~~~~~~~~~~~~~~~~~~~  .. container:: dropdown     **AutoDoc File Output Type**     Specify the AutoDoc file output type. Choose fromdocx(the    default value) andmd``.

AUTODOC_SUBTEMPLATE_TYPE``

AUTODOC SUB-TEMPLATE TYPE

Specify the type of sub-templates to use. Choose from the following:

-   auto (Default)
-   md
-   docx

AUTODOC_MAX_CM_SIZE``

CONFUSION MATRIX MAX NUMBER OF CLASSES

Specify the maximum number of classes in the confusion matrix. This
value defaults to 10.

AUTODOC_NUM_FEATURES``

NUMBER OF TOP FEATURES TO DOCUMENT

Specify the number of top features to display in the document. To
disable this setting, specify -1. This is set to 50 by default.

AUTODOC_MIN_RELATIVE_IMPORTANCE``

MINIMUM RELATIVE FEATURE IMPORTANCE THRESHOLD

Specify the minimum relative feature importance in order for a feature
to be displayed. This value must be a float >= 0 and <= 1. This is set
to 0.003 by default.

AUTODOC_INCLUDE_PERMUTATION_FEATURE_IMPORTANCE``

PERMUTATION FEATURE IMPORTANCE

Specify whether to compute permutation-based feature importance. This is
disabled by default.

autodoc_pd_max_rows~~~~~~~~~~~~~~~~~~~~~~~  .. container:: dropdown     **PDP and Shapley Summary Plot Max Rows**     Specify the number of rows shown for the partial dependence plots    (PDP) and Shapley values summary plot in the AutoDoc. Random sampling    is used for datasets with more than theautodoc_pd_max_rows``

    limit. This value defaults to 10000.

AUTODOC_PD_MAX_RUNTIME``

PDP MAX RUNTIME IN SECONDS

Specify the maximum number of seconds Partial Dependency computation can
take when generating a report. Set to -1 for no time limit.

AUTODOC_OUT_OF_RANGE``

PDP OUT OF RANGE

Specify the number of standard deviations outside of the range of a
column to include in partial dependence plots. This shows how the model
reacts to data it has not seen before. This is set to 3 by default.

AUTODOC_NUM_ROWS``

ICE NUMBER OF ROWS

Specify the number of rows to include in PDP and ICE plots if individual
rows are not specified. This is set to 0 by default.

AUTODOC_POPULATION_STABILITY_INDEX``

POPULATION STABILITY INDEX

Specify whether to include a population stability index if the
experiment is a binary classification or regression problem. This is

AUTODOC_POPULATION_STABILITY_INDEX_N_QUANTILES``

POPULATION STABILITY INDEX NUMBER OF QUANTILES

Specify the number of quantiles to use for the population stability
index. This is set to 10 by default.

AUTODOC_PREDICTION_STATS``

PREDICTION STATISTICS

Specify whether to include prediction statistics information if the
experiment is a binary classification or regression problem. This value
is disabled by default.

AUTODOC_PREDICTION_STATS_N_QUANTILES``

PREDICTION STATISTICS NUMBER OF QUANTILES

Specify the number of quantiles to use for prediction statistics. This
is set to 20 by default.

AUTODOC_RESPONSE_RATE``

RESPONSE RATES PLOT

Specify whether to include response rates information if the experiment
is a binary classification problem. This is disabled by default.

AUTODOC_RESPONSE_RATE_N_QUANTILES``

RESPONSE RATES PLOT NUMBER OF QUANTILES

Specify the number of quantiles to use for response rates information.
This is set to 10 by default.

AUTODOC_GINI_PLOT``

SHOW GINI PLOT

Specify whether to show the GINI plot. This is disabled by default.

AUTODOC_ENABLE_SHAPLEY_VALUES``

ENABLE SHAPLEY VALUES

Specify whether to show Shapley values results in the AutoDoc. This is
enabled by default.

AUTODOC_GLOBAL_KLIME_NUM_FEATURES``

GLOBAL K-LIME NUMBER OF FEATURES

Specify the number of features to show in a k-LIME global GLM
coefficients table. This value must be an integer greater than 0 or -1.
To show all features, set this value to -1.

AUTODOC_GLOBAL_KLIME_NUM_TABLES``

GLOBAL K-LIME NUMBER OF TABLES

Specify the number of k-LIME global GLM coefficients tables to show in
the AutoDoc. Set this value to 1 to show one table with coefficients
sorted by absolute value. Set this value to 2 to show two tables - one
with the top positive coefficients and another with the top negative
coefficients. This value is set to 1 by default.

AUTODOC_DATA_SUMMARY_COL_NUM``

NUMBER OF FEATURES IN DATA SUMMARY TABLE

Specify the number of features to be shown in the data summary table.
This value must be an integer. To show all columns, specify any value
lower than 1. This is set to -1 by default.

AUTODOC_LIST_ALL_CONFIG_SETTINGS``

LIST ALL CONFIG SETTINGS

Specify whether to show all config settings. If this is disabled, only
settings that have been changed are listed. All settings are listed when
enabled. This is disabled by default.

AUTODOC_KERAS_SUMMARY_LINE_LENGTH``

KERAS MODEL ARCHITECTURE SUMMARY LINE LENGTH

Specify the line length of the Keras model architecture summary. This
value must be either an integer greater than 0 or -1. To use the default
line length, set this value to -1 (default).

AUTODOC_TRANSFORMER_ARCHITECTURE_MAX_LINES``

NLP/IMAGE TRANSFORMER ARCHITECTURE MAX LINES

Specify the maximum number of lines shown for advanced transformer
architecture in the Feature section. Note that the full architecture can
be found in the appendix.

AUTODOC_FULL_ARCHITECTURE_IN_APPENDIX``

APPENDIX NLP/IMAGE TRANSFORMER ARCHITECTURE

Specify whether to show the full NLP/Image transformer architecture in
the appendix. This is disabled by default.

AUTODOC_COEF_TABLE_APPENDIX_RESULTS_TABLE``

FULL GLM COEFFICIENTS TABLE IN THE APPENDIX

Specify whether to show the full GLM coefficient table(s) in the
appendix. This is disabled by default.

AUTODOC_COEF_TABLE_NUM_MODELS``

GLM COEFFICIENT TABLES NUMBER OF MODELS

Specify the number of models for which a GLM coefficients table is shown
in the AutoDoc. This value must be -1 or an integer >= 1. Set this value
to -1 to show tables for all models. This is set to 1 by default.

AUTODOC_COEF_TABLE_NUM_FOLDS``

GLM COEFFICIENT TABLES NUMBER OF FOLDS PER MODEL

Specify the number of folds per model for which a GLM coefficients table
is shown in the AutoDoc. This value must be be -1 (default) or an
integer >= 1 (-1 shows all folds per model).

AUTODOC_COEF_TABLE_NUM_COEF``

GLM COEFFICIENT TABLES NUMBER OF COEFFICIENTS

Specify the number of coefficients to show within a GLM coefficients
table in the AutoDoc. This is set to 50 by default. Set this value to -1
to show all coefficients.

AUTODOC_COEF_TABLE_NUM_CLASSES``

GLM COEFFICIENT TABLES NUMBER OF CLASSES

Specify the number of classes to show within a GLM coefficients table in
the AutoDoc. Set this value to -1 to show all classes. This is set to 9
by default.

AUTODOC_NUM_HISTOGRAM_PLOTS``

NUMBER OF HISTOGRAMS TO SHOW

Specify the number of top features for which to show histograms. This is
set to 10 by default.

Disparate Impact Analysis Explainer Settings

For information on Disparate Impact Analysis in Driverless AI, see
dai-dia. The following is a list of parameters that can be toggled from
the recipes tab of the MLI page when running a new interpretation.

DIA_COLS``

LIST OF FEATURES FOR WHICH TO COMPUTE DIA

Specify a list of specific features for which to compute DIA.

MAXIMIZE_METRIC``

MAXIMIZE METRIC

Specify a metric to use when computing DIA. Choose from the following:

-   F1
-   F05
-   F2
-   MCC

USE_HOLDOUT_PREDS``

USE INTERNAL HOLDOUT PREDICTIONS

Specify whether to use internal holdout predictions when computing DIA.

SAMPLE_SIZE``

SAMPLE SIZE FOR DISPARATE IMPACT ANALYSIS

Specify the sample size for Disparate Impact Analysis. By default, this
value is set to 100000.

MAX_CARD``

MAX CARDINALITY FOR CATEGORICAL VARIABLES

Specify the max cardinality for categorical variables. By default, this
value is set to 10.

MIN_CARD``

MINIMUM CARDINALITY FOR CATEGORICAL VARIABLES

Specify the minimum cardinality for categorical variables. By default,
this value is set to 2.

NUM_CARD``

MAX CARDINALITY FOR NUMERIC VARIABLES TO BE CONSIDERED CATEGORICAL

Specify the max cardinality for numeric variables to be considered
categorical. By default, this value is set to 25.

FAST_APPROX``

SPEED UP PREDICTIONS WITH A FAST APPROXIMATION

Specify whether to increase the speed of predictions with a fast
approximation. This is enabled by default.

NLP Partial Dependence Plot Explainer Settings

MAX_TOKENS``

NUMBER OF TEXT TOKENS

Specify the number of text tokens for the NLP Partial Dependence plot.
This value defaults to 20.

custom_tokens~~~~~~~~~~~~~~~~~  .. container:: dropdown     **List of custom text tokens**     Specify a list of custom text tokens for which to compute NLP partial    dependence. For example,["text_feature('word_1')"], wheretext_feature``
is the name of the model text feature.

NLP VECTORIZER + LINEAR MODEL TEXT FEATURE IMPORTANCE EXPLAINER SETTINGS

TXT_COLS``

TEXT FEATURE FOR WHICH TO COMPUTE EXPLANATION

Specify the text feature for which to compute explanation.

CUT_OFF``

CUT OFF FOR DECIPHERING BINARY CLASS OUTCOME

Specify the cut off for deciphering binary class outcome based on DAI
model predictions. Any DAI prediction greater than the cut off is the
target label and any DAI prediction less than the cut off is the
non-target label.

CUT OFF BASED ON A METRIC TO MAXIMIZE

Calculate cut off based on a metric to maximize, which will decipher
binary class outcome based on DAI model predictions. Any DAI prediction
greater than the cut off is the target label and any DAI prediction less
than the cut off is the non-target label. It should be noted that
specifying a cut off AND a max metric will give precedence to the cut
off.

Partial Dependence Plot Explainer Settings

For information on Partial Dependence Plots in Driverless AI, see
partial-dependence-plot. The following is a list of parameters that can
be toggled from the recipes tab of the MLI page when running a new
interpretation.

SAMPLE SIZE FOR PARTIAL DEPENDENCE PLOT

When number of rows is above this limit, sample for the Driverless AI
partial dependence plot.

MAX_FEATURES``

PARTIAL DEPENDENCE PLOT NUMBER OF FEATURES

Specify the number of features that can be viewed on the partial
dependence plot. By default, this is set to 10. To view all features,
set this value to -1.

FEATURES``

PARTIAL DEPENDENCE PLOT FEATURE LIST

Specify a list of features for the partial dependence plot.

OOR_GRID_RESOLUTION``

PDP NUMBER OF OUT OF RANGE BINS

Specify the number of out of range bins for the partial dependence plot.
By default, this is set to 0.

QTILE_GRID_RESOLUTION``

PDP QUANTILE BINNING

Specify the total quantile points used to create bins. By default, this
is set to 0.

GRID_RESOLUTION``

PDP OBSERVATIONS PER BIN

Specify the number of equally spaced points used to create bins. By
default, this is set to 20.

CENTER``

CENTER PDP USING ICE CENTERED AT 0

Specify whether center the partial dependence plot using ICE centered at
0. This is disabled by default.

SORT_BINS``

ENSURE BIN VALUES SORTING

Specify whether to ensure bin values sorting. This is enabled by
default.

HISTOGRAMS``

ENABLE HISTOGRAMS

Specify whether to enable histograms for the partial dependence plot.

qtile-bins~~~~~~~~~~~~~~  .. container:: dropdown     **Per-Feature Quantile Binning**     Specify per-feature quantile binning. For example, if you select    features F1 and F2, this parameter can be specified as'{"F1":
2,"F2": 5}'``.

  NOTE: You can set all features to use the same quantile binning with
  the quantile-bins parameter and then adjust the quantile binning for a
  subset of PDP features with this parameter.

1_FRAME``

ENABLE PDP CALCULATION OPTIMIZATION

Specify whether to enable PDP calculation optimization, which minimizes
the number of predictions by combining per-bin frames together. By
default, this is set to 'Auto'.

NUMCAT_NUM_CHART``

UNIQUE FEATURE VALUES COUNT-DRIVEN PDP BINNING AND CHART SELECTION

Specify whether to use dynamic switching between PDP numeric and
categorical binning and UI chart selection in cases where features were
used both as numeric and categorical by the experiment. This is enabled

numcat_threshold~~~~~~~~~~~~~~~~~~~~  .. container:: dropdown     **Threshold for PD/ICE Binning and Chart Selection**     Ifmli_pd_numcat_num_chart`` is enabled, and if the number of

    unique feature values is greater than the threshold, then numeric
    binning and chart is used. Otherwise, categorical binning and chart
    is used. The default threshold value is 11.

SENSITIVITY ANALYSIS EXPLAINER SETTINGS

SAMPLE SIZE FOR SENSITIVITY ANALYSIS (SA)

When the number of rows is above this limit, sample for Sensitivity
Analysis (SA). The default value is 500000.

Shapley Summary Plot Explainer Settings

For information on Shapley Summary Plots in Driverless AI, see
dai-shapley-summary. The following is a list of parameters that can be
toggled from the recipes tab of the MLI page when running a new

MAXIMUM NUMBER OF FEATURES TO BE SHOWN

Specify the maximum number of features that are shown in the plot. By
default, this value is set to 50.

Specify the sample size for the plot. By default, this value is set to
20000.

X_RESOLUTION``

X-AXIS RESOLUTION

Specify the number of Shapley value bins. By default, this value is set
to 500.

DRILLDOWN_CHARTS``

ENABLE CREATION OF PER-FEATURE SHAPLEY / FEATURE VALUE SCATTER PLOTS

Specify whether to enable the creation of per-feature Shapley or feature
value scatter plots. This is enabled by default.

Shapley Values for Original Features Settings

SAMPLE SIZE FOR NAIVE SHAPLEY

When the number of rows is above this limit, sample for Naive Shapley.
By default, this value is set to 100000.

Surrogate Decision Tree Explainer Settings

For information on Surrogate Decision Tree Plots in Driverless AI, see
decision-tree. The following is a list of parameters that can be toggled
from the recipes tab of the MLI page when running a new interpretation.

DT_TREE_DEPTH``

DECISION TREE DEPTH

Specify the depth of the decision tree. By default, this value is set to
3.

NFOLDS``

NUMBER OF CV FOLDS

Specify the number of CV folds to use. By default, this value is set to
0.

QBIN_COLS``

QUANTILE BINNING COLUMNS

Specify quantile binning columns.

QBIN_COUNT``

QUANTILE BINS COUNT

Specify the number of quantile bins. By default, this value is set to 0.

LINUX DEBS

For Linux machines that will not use the Docker image or RPM, a deb
installation is available for x86_64 Ubuntu 16.04/18.04/20.04/22.04. The following installation steps assume that you have a valid license
key for Driverless AI. For information on how to obtain a license key
for Driverless AI, visit https://www.h2o.ai/products/h2o-driverless-ai/. Once obtained, you will be prompted to paste the license key into the
Driverless AI UI when you first log in, or you can save it as a .sig
file and place it in the license folder that you will create during the
installation process. Note
- To ensure that AutoDoc <autodoc> pipeline visualizations are generated
correctly on native installations, installing fontconfig is recommended. -   When using systemd, remove the dai-minio, dai-h2o, dai-redis,
    dai-procsy, and dai-vis-server services.

Note that if you are
    using K80 GPUs, the minimum required NVIDIA driver version is
    450.80.02
-   OpenCL (Required for full LightGBM support on GPU-powered systems)
-   Driverless AI deb, available from https://www.h2o.ai/download/
NOTE: CUDA 11.2.2 (for GPUs) and cuDNN (required for TensorFlow support
on GPUs) are included in the Driverless AI package. About the Install
-   The 'dai' service user is created locally (in /etc/passwd) if it is
    not found by 'getent passwd'. You can override the user by providing
    the DAI_USER environment variable during rpm or dpkg installation. -   The 'dai' service group is created locally (in /etc/group) if it is
    not found by 'getent group'. You can override the group by providing
    the DAI_GROUP environment variable during rpm or dpkg installation.

-   /etc/dai/User.conf: systemd config file specifying the service
        user. -   /etc/dai/Group.conf: systemd config file specifying the service
        group. -   /etc/dai/EnvironmentFile.conf: systemd config file specifying
        (optional) environment variable overrides. -   Software files are placed in /opt/h2oai/dai and owned by the 'root'
    user
-   The following directories are owned by the service user so that they
    can be updated by the running software:
    -   /opt/h2oai/dai/home: The application's home directory (license
        key files are stored here). -   /opt/h2oai/dai/tmp: Experiments and imported data are stored
        here. -   /opt/h2oai/dai/log: Log files go here if you are NOT using
        systemd (if you are using systemd, then the use the standard
        _journalctl_ tool). -   By default, for Docker or DEB/RPM installs, Driverless AI looks for
    a license key in /opt/h2oai/dai/home/.driverlessai/license.sig.

For TAR SH installs, the
    equivalent location is <tar.sh dir>/home/.driverlessai, and after
    the license is imported, it is copied under ~/.driverlessai. If no
    license key is found, the application guides you through the process
    of adding one through the UI. -   systemd unit files are placed in /usr/lib/systemd/system. -   Symbolic links to the configuration files in /etc/dai files are
    placed in /etc/systemd/system. If your environment is running an operational systemd, that is the
preferred way to manage Driverless AI. The package installs the
following systemd services and a wrapper service:
-   DAI: Wrapper service that starts/stops the other three services. -   DAI-DAI: Main Driverless AI process. -   DAI-H2O: H2O-3 helper process used by Driverless AI. -   DAI-PROCSY: Procsy helper process used by Driverless AI. -   DAI-VIS-SERVER: Visualization server helper process used by
    Driverless AI.

Starting NVIDIA Persistence Mode (GPU only)
If you have NVIDIA GPUs, you must run the following NVIDIA command. This
command needs to be run every reboot. For more information:
http://docs.nvidia.com/deploy/driver-persistence/index.html. sudo nvidia-smi -pm 1

Installing OpenCL
OpenCL is required for full LightGBM support on GPU-powered systems. To
install OpenCL, run the following as root:
    mkdir -p /etc/OpenCL/vendors && echo "libnvidia-opencl.so.1" > /etc/OpenCL/vendors/nvidia.icd && chmod a+r /etc/OpenCL/vendors/nvidia.icd && chmod a+x /etc/OpenCL/vendors/ && chmod a+x /etc/OpenCL
Note
If OpenCL is not installed, then CUDA LightGBM is automatically used. CUDA LightGBM is only supported on Pascal-powered (and later) systems,
and can be enabled manually with the enable_lightgbm_cuda_support
config.toml setting. Installing the Driverless AI Linux DEB
Run the following commands to install the Driverless AI DEB. # Install Driverless AI.

You can optionally specify a different service user and
group as shown below. Replace <myuser> and <mygroup> as appropriate. # Temporarily specify service user and group when installing Driverless AI. # dpkg saves these for systemd in the /etc/dai/User.conf and /etc/dai/Group.conf files. sudo DAI_USER=myuser DAI_GROUP=mygroup dpkg -i |VERSION-deb-lin|
You may now optionally make changes to /ETC/DAI/CONFIG.TOML. Starting Driverless AI
To start Driverless AI, use the following command:
    # Start Driverless AI. sudo systemctl start dai
NOTE: If you don't have systemd, refer to linux-tarsh for install
instructions.

sudo systemctl stop dai
    # The processes should now be stopped. Verify. sudo ps -u dai
If you do not have systemd:
    # Stop Driverless AI. sudo pkill -U dai

WARNINGS:
upgrade.

Driverless AI ships with CUDA 11.2.2 for GPUs, but the driver
here . Note
450.80.02. Upgrade Steps
If you have systemd (preferred):
    # Stop Driverless AI. sudo systemctl stop dai
    # Make a backup of /opt/h2oai/dai/tmp directory at this time. # Upgrade Driverless AI. sudo dpkg -i |VERSION-deb-lin|
    sudo systemctl daemon-reload
    sudo systemctl start dai
    # Make a backup of /opt/h2oai/dai/tmp directory at this time. If you do not, all previous data will be lost. # Upgrade and restart.

    # Uninstall Driverless AI. sudo dpkg -r dai
    # Purge Driverless AI. sudo dpkg -P dai
CAUTION! At this point you can optionally completely remove all
remaining files, including the database (this cannot be undone):
    sudo rm -rf /opt/h2oai/dai
    sudo rm -rf /etc/dai
NOTE: The UID and GID are not removed during the uninstall process. These can be removed with userdel and usergroup. However, we DO NOT
recommend removing the UID and GID if you plan to re-install Driverless
AI.

COMPLETED EXPERIMENT PAGE

The following sections describe the completed experiment page. -   completed-actions
-   completed-insights-scores

Completed Experiment Actions
The following is a description of the actions that can be performed
after the status of an experiment changes from RUNNING to COMPLETE. []
-   INTERPRET THIS MODEL drop-down: To create an interpretation of your
    model with the default settings, click INTERPRET THIS MODEL > WITH
    DEFAULT SETTINGS. To create an interpretation with custom settings
    of your choosing, click INTERPRET THIS MODEL > WITH CUSTOM SETTINGS. For more information, see interpreting_a_model. -   DIAGNOSE MODEL ON NEW DATASET: For more information, see
    diagnosing_a_model. -   MODEL ACTIONS drop-down:
      -   ANALYZE RESULTS: Opens the Experiment Results Wizard. Note
          that this wizard is currently in a beta state. -   PREDICT: See Score_On_Another_Dataset. -   TRANSFORM DATASET: See transform_dataset.

(Not
          available for Time Series experiments.) -   SHAPLEY VALUES drop-down: Download
          Shapley values <dai-shapley> for original or transformed
          features. Driverless AI calls XGBoost and LightGBM SHAP
          functions to get contributions for TRANSFORMED FEATURES. Shapley for ORIGINAL FEATURES is approximated from transformed
          features by evenly splitting (naive method) the contribution
          amongst the contributing features. For more information, see
          Shapley values in DAI <dai-shapley>. Select FAST APPROXIMATION
          to make Shapley predictions using only a single fold and model
          from all of the available folds and models in the ensemble. For more information on the fast approximation options, refer
          to the fast_approx_num_trees and
          fast_approx_do_one_fold_one_model
          config.toml settings <sample-configtoml>.

          export_import. -   VISUALIZE SCORING PIPELINE (EXPERIMENTAL): View a visualization of
    the experiment scoring pipeline. For more information, refer to
    visualize_scoring_pipeline. -   DOWNLOAD SCORING PIPELINE drop-down:
      -   DOWNLOAD PYTHON SCORING PIPELINE: Download a standalone Python
          scoring pipeline for H2O Driverless AI. For more information,
          refer to Python_Pipeline. -   DOWNLOAD MOJO SCORING PIPELINE: A standalone Model Object,
          Optimized scoring pipeline. For more information, refer to
          mojo_scoring_pipelines. (Note that this option is not
          available for TensorFlow or RuleFit models.) -   DEPLOY: For information on deploying DAI models, see deployment. -   DOWNLOAD PREDICTIONS: For _regression_ experiments, output includes
    predictions with lower and upper bounds.

For binary problems, F1 is
    the default threshold_scorer, so if a validation set is provided,
    then the threshold for max F1 on the validation set is used to
    create the labels. When performing cross validation, the mean of
    thresholds for max F1 on internal validation folds is used to create
    the labels. For multiclass problems, argmax is used to create the
    labels. -   TRAINING (HOLDOUT) PREDICTIONS: In CSV format, available if a
          validation set was _not_ provided. -   VALIDATION SET PREDICTIONS: In CSV format, available if a
          validation set was provided. -   TEST SET PREDICTIONS: In CSV format, available if a test
          dataset is used. -   DOWNLOAD SUMMARY & LOGS: Download a zip file containing the
    following files. For more information, refer to the
    experiment_summary section.

This file is also available in
    the Experiment Summary ZIP file. Note that this option is not
    available for deprecated models. For more information, see autodoc. -   TUNE EXPERIMENT drop-down: Tune the completed experiment by using
    the following options:
      -   NEW / CONTINUE: Select one of the following options:
            -   WITH SAME SETTINGS: Create a new experiment that copies
                the setup of the original experiment. Selecting this
                option takes you to the EXPERIMENT SETUP page, where you
                can change any parameter of the original experiment. -   FROM LAST CHECKPOINT: Create a new experiment that
                copies the setup of the original experiment and
                continues from the last iteration's checkpoint of models
                and features. Selecting this option takes you to the
                EXPERIMENT SETUP page, where you can change any
                parameter of the original experiment.

For
          more information, see retrain. -   CREATE INDIVIDUAL RECIPE: Create the Individual Recipe for the
          experiment being viewed. After the Individual Recipe has been
          created, you can either download it to your local filesystem
          or upload it as a Custom Recipe. If you upload the Individual
          Recipe as a Custom Recipe, it can be found on the RECIPES page
          and in the Expert Settings under INCLUDE SPECIFIC INDIVIDUALS. For more information on the Individual Recipe, see
          individual_recipe. Note
The preceding "Download" options (with the exception of AutoDoc) appear
as "Export" options if artifacts were enabled when Driverless AI was
started. For more information, refer to export_artifacts.

GOOGLE CLOUD STORAGE SETUP

Driverless AI lets you explore Google Cloud Storage data sources from
within the Driverless AI application. This section provides instructions
for configuring Driverless AI to work with Google Cloud Storage. This
setup requires you to enable authentication. If you enable GCS or GBP
connectors, those file systems will be available in the UI, but you will
not be able to use those connectors without authentication. In order to enable the GCS data connector with authentication, you must:
1. Obtain a JSON authentication file from GCP. 2. Mount the JSON file to the Docker instance. 3. Specify the path to the /json_auth_file.json in the
    gcs_path_to_service_account_json config option. NOTES:
-   The account JSON includes authentications as provided by the system
    administrator. You can be provided a JSON file that contains both
    Google Cloud Storage and Google BigQuery authentications, just one
    or the other, or none at all.

Use docker version to check which version of Docker you are using. Description of Configuration Attributes
-   gcs_path_to_service_account_json: Specifies the path to the
    /JSON_AUTH_FILE.JSON file. -   gcs_init_path: Specifies the starting GCS path displayed in the UI
    of the GCS browser. Start GCS with Authentication
This example enables the GCS data connector with authentication by
passing the JSON authentication file. This assumes that the JSON file
contains Google Cloud Storage authentications.

LOCAL AUTHENTICATION EXAMPLE

This section describes how to enable local authentication in Driverless
AI.

To enable authentication in Docker images, specify the authentication
environment variable that you want to use. Each variable must be
prepended with DRIVERLESS_AI. The example below starts Driverless AI
with environment variables the enable the following:

-   Local authentication when starting Driverless AI
-   S3 and HDFS access (without authentication)

MONITORING PENDING JOBS

Driverless AI features a PENDING JOBS panel that lets you monitor the
progress of various long-running jobs that can be started from the
completed_experiment page. To view this panel, click the group of square
icons located in the upper-right corner. The following jobs are monitored in this panel:
-   Create AutoDoc
-   Create MOJO Scoring Pipeline
-   Create Python Scoring Pipeline
-   Create Test Set Predictions
-   Create Training Predictions
-   Score Model
-   Transform Data
The circular icon next to the description of a pending job indicates its
status:
+---------+------------+
| Icon    | Status     |
+=========+============+
| [logo]  |   Complet  |
|         |            |
|         | e          |
| [logo2] |   Failed   |
|         |   Running  |
Navigate to a completed job by clicking the OPEN icon.

R CLIENT TUTORIAL

This tutorial describes how to use the Driverless AI R client package to
use and control the Driverless AI platform. It covers the main
predictive data-science workflow, including:
1. Data load
2. Automated feature engineering and model tuning
3. Model inspection
4. Predicting on new data
5. Managing the datasets and models
NOTE: These steps assume that you have entered your license key in the
Driverless AI UI.

dai.upload_dataset()`` instead.

If you already have the data loaded into R data.frame, you can convert
it into a DAIFrame. For example:

    iris.dai <- as.DAIFrame(iris)
    #> 
      |                                                                       
      |                                                                 |   0%
      |                                                                       
      |=================================================================| 100%

    print(iris.dai)
    #> DAI frame '7c38cb84-5baa-11e9-a50b-b938de969cdb': 150 obs. of 5 variables
    #> File path: ./tmp/7c38cb84-5baa-11e9-a50b-b938de969cdb/iris9e1f15d2df00.csv.1554912339.9424415.bin

You can switch off the progress bar whenever it is displayed by setting

progress =
FALSE. Upon creation of the dataset, you can display the basic information and summary statistics by calling generics print and summary:  .. code:: r     print(creditcard)    #> DAI frame '7abe28b2-5baa-11e9-a50b-b938de969cdb': 23999 obs.

training and test sets directly in R:  .. code:: r     creditcard.splits <- dai.split_dataset(creditcard,                                            output_name1 = 'train',                                            output_name2 = 'test',                                            ratio = .8,                                           seed = 25,                                           progress = FALSE)  In this case the creditcard.splits is a list with two elements with names "train" and "test", where 80% of the data went into train and 20% of the data went into test. .. code:: r     creditcard.splits$train    #> DAI frame '7cf3024c-5baa-11e9-a50b-b938de969cdb': 19199 obs. of 25 variables    #> File path: ./tmp/7cf3024c-5baa-11e9-a50b-b938de969cdb/train.1554912341.0864356.bin     creditcard.splits$test    #> DAI frame '7cf613a6-5baa-11e9-a50b-b938de969cdb': 4800 obs.

See the function’s documentation for more details. Automated Feature Engineering and Model Tuning ----------------------------------------------  One of the main strengths of Driverless AI is the fully automated feature engineering along with hyperparameter tuning, model selection and ensembling. The functiondai.train()executes the experiment that results in a DAIModel instance that represents the model.

-  Download a scoring pipeline and embed that into your Python or Java    workflow. Predicting in R ~~~~~~~~~~~~~~~  Genericpredict()either directly returns an R data.frame with the results (by default) or it returns a URL pointing to a CSV file with the results (return_df=FALSE). The latter option may be useful when you predict on a large dataset.

For more information about how to use the pipelines, refer to the R Client documentation.

The dai package offers a few utility functions to find, reuse, and remove the existing datasets and models. If you already have the dataset loaded into DAI, you can get the DAIFrame object by eitherdai.get_frame(if you know the frame’s key) ordai.find_dataset(if you know the original path or at least a part of it):  .. code:: r     dai.get_frame(creditcard$key)    #> DAI frame '7abe28b2-5baa-11e9-a50b-b938de969cdb': 23999 obs. of 25 variables    #> File path: tests/smalldata/kaggle/CreditCard/creditcard_train_cat.csv     dai.find_dataset('creditcard')    #> DAI frame '7abe28b2-5baa-11e9-a50b-b938de969cdb': 23999 obs. of 25 variables    #> File path: tests/smalldata/kaggle/CreditCard/creditcard_train_cat.csv  The latter directly returns you the frame if there’s only one match. Otherwise it let you select which frame to return from all the matching candidates.

creditcard.splits$trainandcreditcard.splits$testobjects will not be removed from R session because they are actually function calls (recall that$``
is a function).

LAUNCHING H2O FLOW

If you opened port 12348 when starting Driverless AI, then you can
launch H2O Flow from within Driverless AI. Click the H2O-3 link in the
top menu.

[]

This launches Flow on port 12348.

[]

ENABLING NOTIFICATIONS

Driverless AI can be configured to trigger a user-defined script at the
beginning and end of an experiment. This functionality can be used to
send notifications to services like Slack or to trigger a machine
shutdown. The CONFIG.TOML file exposes the following variables:
-   listeners_experiment_start: Registers an absolute location of a
    script that gets executed at the start of an experiment. -   listeners_experiment_done: Registers an absolute location of a
    script that gets executed when an experiment is finished
    successfully. Driverless AI accepts any executable as a script. (For example, a script
can be implemented in Bash or Python.) There are only two requirements:
-   The specified script can be executed. (i.e., The file has executable
    flag.) -   The script should be able to accept command line parameters.

setuidbit set up together with executable bit. For more info, visit: https://unix.stackexchange.com/questions/85663/poweroff-or-reboot-as-normal-user.) Theon_startScript ~~~~~~~~~~~~~~~~~~~~~~~  This script increases the counter of running experiments. ::        #!/usr/bin/env bash        app_id="${1}"       experiment_id="${3}"       tmp_dir="${TMPDIR:-/tmp}/${app_id}"       exp_file="${tmp_dir}/${experiment_id}"        mkdir -p "${tmp_dir}"       touch "${exp_file}"  Theon_doneScript ~~~~~~~~~~~~~~~~~~~~~~  This script decreases the counter and executes machine shutdown when the counter reaches 0-value.

else           echo "No experiments running! Machine is going to shutdown!" # Use instance meta-data API to get instance ID and then use AWS CLI to shutdown the machine           # This expects, that AWS CLI is properly configured and has capability to shutdown instances enabled. aws ec2 stop-instances --instance-ids $(curl http://169.254.169.254/latest/meta-data/instance-id)       fi  .. container:: tabs     .. group-tab:: Docker Image Installs     1. Copy the config.toml file from inside the Docker image to your       local filesystem. (Changenvidia-docker runtodocker runfor non-GPU environments.) ..        .. code:: bash           # In your Driverless AI folder (for exmaple, dai_1.5.1),           # make config and scripts directories          mkdir config          mkdir scripts           # Copy the config.toml file to the new config directory.

Edit the **Notification scripts** section in the **config.toml**       file and save your changes. Note that in this example, the scripts       are saved to a **dai_VERSION/scripts** folder. ..        ::           # Notification scripts          # - the variable points to a location of script which is executed at given event in experiment lifecycle          # - the script should have executable flag enabled          # - use of absolute path is suggested          # The on experiment start notification script location          listeners_experiment_start = "dai_VERSION/scripts/on_start.sh"          # The on experiment finished notification script location          listeners_experiment_done = "dai_VERSION/scripts/on_done.sh"     3. Start Driverless AI with the DRIVERLESS_AI_CONFIG_FILE environment       variable. Make sure this points to the location of the edited       config.toml file so that the software finds the configuration       file.

S3 SETUP

Driverless AI lets you explore S3 data sources from within the
configuring Driverless AI to work with S3.

docker run
--runtime=nvidia(>= Docker 19.03) ornvidia-docker(< Docker 19.03) command when starting the Driverless AI Docker image. Usedocker
versionto check which version of Docker you are using. Description of Configuration Attributes ---------------------------------------  -aws_access_key_id: The S3 access key ID -aws_secret_access_key: The S3 access key -aws_role_arn: The Amazon Resource Name -aws_default_region: The region to use when the    aws_s3_endpoint_url option is not set. This is ignored when    aws_s3_endpoint_url is set. -aws_s3_endpoint_url: The endpoint URL that will be used to access    S3. -aws_use_ec2_role_credentials: If set to true, the S3 Connector    will try to to obtain credentials associated with the role attached    to the EC2 instance. -s3_init_path: The starting S3 path that will be displayed in UI    S3 browser. -enabled_file_systems: The file systems you want to enable. This    must be configured in order for data connectors to function properly.

It does not pass any S3 access key or secret; however    it configures Docker DNS by passing the name and IP of the S3 name    node. This allows users to reference data stored in S3 directly using    the name node address, for example: s3://name.node/datasets/iris.csv.

Note that this example enables S3 with no authentication. 1. Configure the Driverless AI config.toml file. Set the following          configuration options. ..           -enabled_file_systems
= "file, upload,
s3"2. Mount the config.toml file into the Docker container.

It does not pass any S3 access key or secret. 1. Export the Driverless AI config.toml file or add it to          ~/.bashrc. For example:        ..           ::              # DEB and RPM             export DRIVERLESS_AI_CONFIG_FILE="/etc/dai/config.toml"              # TAR SH             export DRIVERLESS_AI_CONFIG_FILE="/path/to/your/unpacked/dai/directory/config.toml"         2. Specify the following configuration options in the config.toml          file.

(jdbc_app_configs)             # hive: Hive Connector, remember to configure Hive below. (hive_app_configs)             # recipe_url: load custom recipe from URL             # recipe_file: load custom recipe from local file system             enabled_file_systems = "file, s3"        3. Save the changes when you are done, then stop/restart          Driverless AI. Example 2: Enable S3 with Authentication ----------------------------------------  .. container:: tabs     .. group-tab:: Docker Image Installs     This example enables the S3 data connector with authentication by    passing an S3 access key ID and an access key. It also configures    Docker DNS by passing the name and IP of the S3 name node. This    allows users to reference data stored in S3 directly using the name    node address, for example: s3://name.node/datasets/iris.csv.

AUTOMATED MODEL DOCUMENTATION (AUTODOC)

This section describes Driverless AI's AutoDoc feature.

autodoc-using autodoc-placeholders

LINUX RPMS

For Linux machines that will not use the Docker image or DEB, an RPM
installation is available for the following environments:
-   x86_64 RHEL 7 / RHEL 8
-   CentOS 7 / CentOS 8
The installation steps assume that you have a license key for Driverless
visit https://www.h2o.ai/products/h2o-driverless-ai/. Once obtained, you
will be prompted to paste the license key into the Driverless AI UI when
you first log in, or you can save it as a .sig file and place it in the
license folder that you will create during the installation process. Note

Note that if you are using
    K80 GPUs, the minimum required NVIDIA driver version is 450.80.02
-   Driverless AI RPM, available from https://www.h2o.ai/download/
    the DAI_GROUP environment variable during rpm or dpkg installation. -   Configuration files are placed in /etc/dai and owned by the 'root'
    user:
    -   /etc/dai/config.toml: Driverless AI config file (See config_file
        section for details).

-   /etc/dai/Group.conf: systemd config file specifying the service
    user
    a license key in /opt/h2oai/dai/home/.driverlessai/license.sig. If
    you are installing Driverless AI programmatically, you can copy a
    license key file to that location.

If no
    Driverless AI. If you don't have systemd, refer to linux-tarsh for install
instructions. Installing OpenCL
OpenCL is required for full LightGBM support on GPU-powered systems.

CUDA LightGBM is only supported on Pascal-powered (and later) systems,
config.toml setting. Installing Driverless AI
Run the following commands to install the Driverless AI RPM. # Install Driverless AI. sudo rpm -i |VERSION-rpm-lin|
NOTE: For RHEL 7.5, it is necessary to upgrade library glib2:
    sudo yum upgrade glib2
By default, the Driverless AI processes are owned by the 'dai' user and
'dai' group. You can optionally specify a different service user and
group as shown below. Replace <myuser> and <mygroup> as appropriate. # Temporarily specify service user and group when installing Driverless AI. # rpm saves these for systemd in the /etc/dai/User.conf and /etc/dai/Group.conf files. sudo DAI_USER=myuser DAI_GROUP=mygroup rpm -i |VERSION-rpm-lin|
    # Start Driverless AI.

sudo -H -u dai /opt/h2oai/dai/run-dai.sh

Starting NVIDIA Persistence Mode

Looking at Driverless AI log files
    sudo systemctl status dai-dai
    sudo journalctl -u dai-dai
    sudo less /opt/h2oai/dai/log/dai.log
    sudo less /opt/h2oai/dai/log/h2o.log
    sudo less /opt/h2oai/dai/log/procsy.log
    sudo less /opt/h2oai/dai/log/vis-server.log

Stopping Driverless AI
    # The processes should now be stopped. Verify.

-   Experiments, MLIs, and MOJOs reside in the Driverless AI tmp
upgrade. Requirements

Note
    # Make a backup of /opt/h2oai/dai/tmp directory at this time. # Upgrade and restart. sudo rpm -U |VERSION-rpm-lin|
    sudo -H -u dai /opt/h2oai/dai/run-dai.sh

Uninstalling Driverless AI
    # Uninstall.

SUPPORTED FILE TYPES

Driverless AI supports the following dataset file formats:
-   arff
-   avro
-   bin
-   bz2
-   csv (See note below)
-   dat
-   feather
-   gz
-   jay (See note below)
-   orc (See notes below)
-   parquet (See notes below)
-   pickle / pkl (See note below)
-   tgz
-   tsv
-   txt
-   xls
-   xlsx
-   xz
-   zip
Note
- Compressed Parquet files are typically the most efficient file type to
use with Driverless AI. - CSV in UTF-16 encoding is only supported when
implemented with a byte order mark (BOM). If a BOM is not present, the
dataset is read as UTF-8. - For ORC and Parquet file formats, if you
select to import multiple files, those files will be imported as
multiple datasets. If you select a folder of ORC or Parquet files, the
folder will be imported as a single dataset. Tools like Spark/Hive
export data as multiple ORC or Parquet files that are stored in a
directory with a user-defined name.

SAMPLING IN DRIVERLESS AI

Data Sampling

Driverless AI does not perform any type of data sampling unless the
dataset is big or highly imbalanced (for improved accuracy). What is
considered big is dependent on your accuracy setting and the

statistical_threshold_data_size_largeparameter in the :ref:`config.toml file <sample-configtoml>` or in the Expert Settings. You can see if the data will be sampled by viewing the Experiment Preview when you set up the experiment. In the experiment preview below, I can see that my data was sampled down to 5 million rows for the final model, and to 100k rows for the feature evolution part of the experiment.

ImbalancedLightGBMModelandImbalancedXGBoostGBMModel. Both perform repeated stratified sampling (bagging) inside their fit() method in an attempt to speed up modeling and to improve the resolution of the decision boundary between the two classes. Because these models are presented a training dataset with a different prior than the original data, they require a probability correction that is performed as part of postprocessing in the predict() method. When imbalanced sampling is enabled, no sampling is performed at the start of the experiment for either the feature evolution phase or the final model pipeline. Instead, sampling (with replacement) is performed during model fitting, and the model is presented a more balanced target class distribution than the original data. Because the sample is usually much smaller than the original data, this process can be repeated many times and each internal model's prediction can be averaged to improve accuracy (bagging).

"over_under_sampling", each bag can have a slightly different balance between minority and majority classes. There are multiple settings for imbalanced sampling:  -  Disabled (imbalance_sampling_method="off", the default) -  Automatic (imbalance_sampling_method="auto"). A combination of    the two methods below. -  Under- and over-sample both minority and majority classes to reach    roughly class balance in each sampled bag    (imbalance_sampling_method="over_under_sampling"). If original    data has 500:10000 imbalance, this method could sample 1000:1500    samples for the first bag, 500:400 samples for the second bag, and so    on. -  Under-sample the majority class to reach exact class balance in each    sampled bag (imbalance_sampling_method="under_sampling"). Would    create 500:500 samples per bag for the same example imbalance ratio . Each bag would then sample the 500 rows from each class with    replacement, so each bag is still different.

In this    case, imbalanced models are added to the list of models for the    experiment to select from. -  By default, heavily imbalanced is defined as when the majority class    is 25 times more common than the minority class    (heavy_imbalance_ratio_sampling_threshold=25, configurable). In    highly imbalanced cases, imbalanced models are used exclusively. Notes:  -  The binary imbalanced sampling techniques and settings described in    this section apply only to the **Imbalanced Model** types listed    above. -  The data has to be large enough to enable imbalanced sampling: by    default,imbalance_sampling_threshold_min_rows_originalis set to    100,000 rows. -  Ifimbalance_sampling_number_of_bags=-1(automatic) andimbalance_sampling_method="auto", the number of bags will be    automatically determined by the experiment's accuracy settings and by    the total size of all bags together, controlled byimbalance_sampling_max_multiple_data_size, which defaults to1.

For an imbalance of 1:19, each balanced 1:1 sample    would be as large as 10% of the data, so it would take up to 10 such    1:1 bags (or approximately 10 if the balance is different or slightly    random) to reach that limit. That's still a bit much for feature    evolution and could cost too much slowdown. That's why the other    limit of 3 (by default) for feature evolution exists. Feel free to    adjust to your preferences. -  Ifimbalance_sampling_number_of_bags=-1(automatic) andimbalance_sampling_method="over_under_sampling"or"under_sampling", the number of bags will be equal to the    experiment's accuracy settings (accuracy 7 will use 7 bags). -  The upper limit for the number of bags can be specified separately    for feature evolution    (imbalance_sampling_max_number_of_bags_feature_evolution) and    globally (i.e., final model) set by    (imbalance_sampling_max_number_of_bags) and both will be strictly    enforced.

So if the data starts with a 1:1000 imbalance and you    wish to model with a 1:9 imbalance, specifyimbalance_sampling_target_minority_fraction=0.1. -  To force the number of bags to 15 for all models, setimbalance_sampling_number_of_bags=imbalance_sampling_max_number_of_bags_feature_evolution=imbalance_sampling_max_number_of_bags=15andimbalance_sampling_methodto either    "over_under_sampling"or\ under_sampling, based on your
    preferences. - By default, Driverless AI will notify you if the imbalance ratio
    (majority to minority) is larger than 2, controlled by
    imbalance_ratio_notification_threshold. - Each parameter mentioned above is described in detail in the
    config.toml file, and also displayed in the expert options (on
    mouse-over hover), and each actual Imbalanced Model displays its
    parameter in the experiment logs. Summary:
There are many options to control imbalanced sampling in Driverless AI.

DRIVERLESS AI WORKFLOW

A typical Driverless AI workflow is to:

1.  Load data
2.  Visualize data
3.  Run an experiment
4.  Interpret the model
5.  Deploy the scoring pipeline

In addition, you can diagnose a model, transform another dataset, score
the model against another dataset, and manage your data in Projects.

Also see the dai_wizard, a question and answer workflow that helps
automatically set up use case specific experiment settings.

The image below describes a typical workflow.

[]

INTRODUCTION TO H2O DRIVERLESS AI

H2O Driverless AI is a high-performance, GPU-enabled, client-server
application for the rapid development and deployment of state-of-the-art
predictive analytics models. It reads tabular data from various sources
and automates data visualization, grand-master level automatic feature
engineering, model validation (overfitting and leakage prevention),
model parameter tuning, model interpretability, and model deployment. H2O Driverless AI is currently targeting common regression, binomial
classification, and multinomial classification applications, including
loss-given-default, probability of default, customer churn, campaign
response, fraud detection, anti-money-laundering, and predictive asset
maintenance models. It also handles time-series problems for individual
or grouped time-series, such as weekly sales predictions per store and
department, with time-causal feature engineering and validation schemes.

-   Multinomial classification ("negative"/"neutral"/"positive" or
    0/1/2/3 or 0.5/1.0/2.0 for categorical target variables, for
    prediction of membership type, next-action, product recommendation,
    sentiment analysis, etc.) Data types supported:
-   Tabular structured data, rows are observations, columns are
    fields/features/variables
-   Numeric, categorical and textual fields
-   Images
-   Missing values are allowed
-   i.i.d.

UNSUPERVISED ALGORITHMS IN DRIVERLESS AI (EXPERIMENTAL)

Starting with version 1.10, Driverless AI exposes unsupervised
transformers that you can use for unsupervised model building. The
following sections describe several unsupervised transformers and
contain information on support for custom recipes and expert control of
unsupervised experiments. 1. Isolation Forest Anomaly detection <isolation_forest>
2. K-Means Clustering <clustering>
3. Truncated SVD (Dimensionality Reduction) <svd>
4. Full support for custom recipes <unsup_custom_recipes>
5. Expert control over Unsupervised Experiments <unsup_expert_control>
Conceptually, the overall pipeline of an unsupervised experiment is
similar to the pipeline of a regular supervised experiment. However,
there are a few notable differences:
1. Only one unsupervised algorithm (model, pipeline) can be chosen
    (that is, either clustering or anomaly detection, but not both).

2. Each such unsupervised modeling pipeline consists of exactly one
    pretransformer, one transformer and one model. No labels (y) are
    required. 3. The unsupervised model has only one function: To list the included
    pretransformer, the included transformer and any applicable scorers. The model itself is a pure pass-through function, the
    models.predict() method returns the output of the transformer
    pipeline (any features the transformers makes). This also means that
    the variable importance of the model is ill-defined, and uniformly
    spread across features. For clustering, there will be only 1 feature
    (the assigned cluster label), and it will have variable importance
    of 1.0. 4. Automatic Machine Learning is only possible if there's a metric
    (scorer) that assesses the quality of the transformation via
    score(X, actual=None, predicted=transformed_X).

If no
    scorer is applicable, an UNSUPERVISEDSCORER is used, which returns 0
    for any input. This value can be ignored, and signals Driverless AI
    that the experiment is converged after the first iteration. 5. No MLI support in 1.10.0, but is planned for future releases. 6. No ensembles and cross-validation for final models for unsupervised
    experiments (fixed_ensemble_level=0 is enforced). As a consequence,
    creation of training holdout predictions is not possible (all data
    is used for the final model). If predictions like cluster
    assignments are desired for the training data, please make
    predictions on the training data, with the usual caveats of
    overfitting (due to heavy tuning during AutoML) since fit() and
    predict() are performed with the same data. Isolation Forest Anomaly detection
Isolation forest isolates or identifies the anomalous entries by
randomly splitting the decision trees.

The algorithm assigns an anomaly score to each
observation based on its path length (from root node to terminal node)
in the forest. The lower the score, the more likely it is that the row
is an anomaly. Internally, Driverless AI runs sklearn's Isolation Forest
implementation. When building a model, the Accuracy and Time knobs of Driverless AI can
be toggled to adjust the effort spent on model tuning but presently as
there is no scorer being used for isolation forest, when doing
genetic algorithm <ga>, the model will converge immediately and use one
of the models from the tuning phase <full_pic> as the final model. The
Interpretability knob is ignored in the default set up. The number of
trees or n_estimators for the isolation forest model can be adjusted
with the isolation_forest_nestimators expert setting parameter. After building the model, the SCORES can be obtained by _predicting on
the same dataset_.

If you don't pass a test dataset, then you must go to MODEL ACTIONS >
PREDICT. The lower the scores of a row, the more likely it is an outlier
or anomaly by the model. The
Visualize Scoring Pipeline option <visualize_scoring_pipeline>
summarizes the features used and transformations applied in building the
model. To create LABELS from these scores, quantile value can be used as a
threshold. For example, if you know that 5% of the rows are anomalous in
your dataset, then this can be used to calculate the 95th quantile of
the scores. This quantile can act as a threshold to classify each row as
being an anomaly or not. The Python scoring pipeline <Python_Pipeline> can be used to deploy the
Isolation Forest model to production (currently no MOJO support). Use case idea: Given an anomaly detection experiment, you can create
predictions on the training dataset, including all original columns, and
re-upload into Driverless AI to run a supervised experiment.

NOTE: The following are some additional details on the transformers and
pretransformers that are relevant to IF. -   IsolationForestAnomalyAllNumericTransformer (transformer): This
    transformer trains an isolation forest model on all numeric columns
    as an unsupervised learning model. -   OrigFreqPreTransformer (pretransformer): Categoricals are frequency
    encoded with this pretransformer. Note that isolation forest itself
    only accepts numericals. KMeans Clustering
Clustering algorithms partition observations into clusters. Driverless
AI uses sklearn KMeans clustering algorithm to partition the
observations so that they belong to the cluster with the nearest mean
(centroid of the cluster). Driverless AI exposes the following unsupervised MODELS that run on
numeric and categorical columns to build a K-Means clustering model.

-   KMEANS : This does K-Means clustering only on numeric columns
  -   KMEANSFREQ : This does K-Means clustering on numeric and
      frequency transformed <cat_transformers> categorical (integer
      columns are treated only as numeric)
  -   KMEANSOHE : This does K-Means clustering on numeric and
      one-hot-encoding transformed categorical columns
Driverless AI provides the following SCORERS to enable automatic
unsupervised clustering:
  -   CALINSKI HARABASZ : The Calinski-Harabasz index also known as the
      Variance Ratio Criterion, is the ratio of the sum of
      between-clusters dispersion and of inter-cluster dispersion for
      all clusters. The HIGHER this score , the better the performances. -   DAVIES BOULDIN : The Davies-Bouldin Index signifies the average
      'similarity' between clusters, where similarity is a measure that
      compares distance between clusters with the size of the clusters
      themselves.

unsupervised_clustering_max_clusters`` parameters can be used in the
expert panel to set the upper and lower bound on the number of clusters
to build.

During model building, DRIVERLESS AI CREATES KMEANS CLUSTERING MODEL ON
A SUBSET OF FEATURES (between 2 to 5). The feature subset size, columns
to be used for clustering and the parameter tuning is decided during the
genetic algorithm <ga> process. User can set the feature subset size
(dimensionality of space to cluster) by

fixed_interaction_depthparameter of the expert settings. The value should lie between 2 to 5. Say,fixed_interaction_depth=4, then clustering will be performed in 4D. If say, more than 4 features are present in the dataset (or after accounting for the pre-transformations like one-hot-encoding), then when doing genetic algorithm, DAI will select input features and model parameters (based on internal train/valid split(s)) to decide the best possible subset of 4 features and their parameter set to build the model that optimizes the scores. The **scorer** takes the *full dataset* (pre transformed with all features) and *labels* for the rows as created by the (subset of features) clustering model to give the scores. It compares the output of the unsupervised transformer to its input. The **Insights** tab of the experiment gives a peek into the working of clustering transformer on the subset of features to build the best model.

The plot displays the cluster separation in 2 dimensions and is color coded for easy visualization. Aggregator algorithm is used to reduce the datasize for the plot. This is a preview of the custom visualization capability (using Vega) that is coming soon to DAI. After building the model, the :ref:`Visualize Scoring Pipeline option <visualize_scoring_pipeline>` can be used to inspect the **pre transformations** applied to the features, before building model (on subset of features) and scoring (on full set). It can also be used to inspect the features used to build the clustering model. The cluster **labels** can be created by predicting on the dataset. To get cluster label assignments for the training (or any) dataset, then the fitted model can be used to make predictions, just like any supervised model. Note that overfitting can occur anytime when fit and predict are performed on the same dataset.

.. figure:: images/clust_pipeline.png    :alt:   You can also write custom clustering recipes by defining your own pretransformer (i.e what columns with what encodings are fed in for clustering), clustering transformer, and scorer. To view an example, see the `KMeans Clustering Using RAPIDS.ai recipe <https://github.com/h2oai/driverlessai-recipes/blob/master/models/unsupervised/rapids_kmeans.py#L98>`__ in the official Driverless AI recipes repository. (For best results, use the release branch that corresponds with your version of Driverless AI.) .. _svd:  Truncated SVD (Dimensionality Reduction) ----------------------------------------  `Truncated SVD <https://en.wikipedia.org/wiki/Singular_value_decomposition#Truncated_SVD>`__ is a dimensionality reduction method and can be applied to a dataset to reduce the number of features before running say a supervised algorithm. It factorizes data matrix where the number of columns is equal to the specified truncation.

Internally Driverless AI runs `sklearn Truncated SVD <https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html>`__ implementation. .. raw:: html     <img src="_static/unsuper_svd.gif" alt="svd" data-linktype="relative_path">  Driverless AI exposes the TRUNCSVD transformer to reduce the number of features. Presently, none of the parameters can be toggled by the user. The n_components created by the TRUNCSVD transformer range between 1 and 5. (Note that these are considered random mutations.) After building the model, :ref:`Visualizing scoring pipeline <visualize_scoring_pipeline>` can be used to inspect the number of components created. Additionally, the dimensionality reduced dataset can be obtained by predicting on the dataset.

The Dimensionality Reduction model produces :ref:`MOJOs <mojo_scoring_pipelines>` and :ref:`Python <Python_Pipeline>` scoring pipelines to deploy to :ref:`production <deployment>`. .. _unsup_custom_recipes:  Unsupervised Custom Recipes ---------------------------  Driverless AI supports **custom Python recipes for unsupervised learning**. You can write custom unsupervised recipes by defining your own pretransformer, transformer, and scorer. To view examples, see the `official Driverless AI recipes repository <https://github.com/h2oai/driverlessai-recipes/tree/master/models/unsupervised>`__. (For best results, use the release branch that corresponds with your version of Driverless AI.) .. _unsup_expert_control:  Expert control over Unsupervised Experiments --------------------------------------------  You can control unsupervised experiments by selecting specific **pretransformers** and **transformers**.

To specify pretransformers and transformers, use the Expert Settings window of an experiment. For more information, see :ref:`understanding-configs`. The following steps describe how to control unsupervised experiments with the Expert Settings window. 1. On the **Experiment Setup** page, select **Unsupervised**. 2. Click **Unsupervised learning model** and select **Unsupervised**    from the list of options. The preview updates to display the    transformers that are used by default. 3. On the Experiment Setup page, click **Expert Settings**. The Expert    Settings window is displayed. a. **To select specific pretransformers:** In the **Training ->          Feature Engineering** tab, click the **Select values** button          for the **Include specific preprocessing transformers**          (included_pretransformers) setting. To confirm your          selection, click **Done**. b.

To confirm your selection, click          **Done**. **Note:** Selecting pretransformers isn't required. If no       pretransformers are selected, then the first layer is ignored. .. figure:: images/unsupervised-expert.png          :alt:   4. To confirm your overall selection and exit out of the Expert Settings    window, click the **Save** button. 5. In the **Training Settings** category on the Experiment Setup page,    specify the **Unsupervised** scorer. Alternatively, select a custom    scorer. .. figure:: images/unsup_expert.png    :alt:   Expert control example 1 ~~~~~~~~~~~~~~~~~~~~~~~~  The following list contains examples of how you can use expert control to configure unsupervised experiments. -  Input text through through **term frequency–inverse document    frequency (TFIDF)** by settingTextTransformeras a    pretransformer, and then through K-Means clustering by settingClusterIdAllNumTransformeras a transformer.

-  Setone_hot_encoding_cardinality_thresholdandone_hot_encoding_cardinality_threshold_default_useto a large    value like 10,000,000 to allow all possible categorical levels to be    included. Expert control example 2 ~~~~~~~~~~~~~~~~~~~~~~~~  The following example describes how you can use expert control to configure unsupervised experiments using a custom recipe for text handling. -  Upload    https://github.com/h2oai/driverlessai-recipes/blob/master/transformers/nlp/text_topic_modeling_transformer.py    (Or choose the version for your DAI release by selecting the correct    branch version.) -  Upload    https://github.com/h2oai/driverlessai-recipes/blob/master/models/unsupervised/TextKMeansIsolationForest.py    (Or choose the version for your DAI release by selecting the correct    branch version.) -  Upload a dataset. On the Experiment Setup page, select    **Unsupervised**, and then select KMeansFreqTextModel for the    unsupervised model.

Expert control example 3 ~~~~~~~~~~~~~~~~~~~~~~~~  The following example describes how you can use expert control to configure unsupervised experiments using a custom recipe for text handling. -  Upload    https://github.com/h2oai/driverlessai-recipes/blob/master/transformers/nlp/text_topic_modeling_transformer.py    (or choose the version for your DAI release) -  Upload a dataset. On the Experiment Setup page, select    **Unsupervised**, and then select **UnsupervisedModel** for the    unsupervised model. -  Click **Expert Settings**. The Expert Settings window is displayed. -  In the **Training -> Feature Engineering** tab, select          **Specific transformers to include** (TOMLincluded_transformers) and select only          ClusterIdAllNumTransformer. -  In the **Training -> Feature Engineering** tab, select          **Specific pretransformers to include** (TOMLincluded_pretransformers) and select only          TextLDATopicTransformer.

Expert control example 4 ~~~~~~~~~~~~~~~~~~~~~~~~  In many cases, you may only want a single output from an unsupervised model. For unsupervised models built into DAI, this is normal. However, by default, theUnsupservisedModelallows any number of outputs. In order to control the expert model to behave more like the internal unsupervised models, you can add a recipe like the following, which can be uploaded or added with **RECIPES->EDITOR**.

UNSUPERVISEDscorer to just do single model. Another way to achieve a similar result in Driverless AI version 1.10.5 and beyond is to make the recipe match the following:  .. code:: python     from h2oaicore.models_custom import CustomModel  # don't use CustomUnsupervisedModel    from h2oaicore.models_unsupervised import UnsupervisedModel    class MyUnsupervisedModel(UnsupervisedModel, CustomModel):        _ngenes_max = 1        _ngenes_max_by_layer = [1000, 1]  but then set expert optioncustom_unsupervised_expert_mode=true. This forces the experiment to use this custom unsupervised model as if it were likeUnsupervisedModelin terms of requiring you to go to the expert panel and select which scorers, transformers, and pretransformers to be used (like supervised experiments). However, by forcing this model to only havengenes_max=1, it ensures only a single instance of the transformer is produced. Note that in this case, onlyUnsupervisedScoreris available as an option.

FEATURE_ENGINEERING_EFFORT``

FEATURE ENGINEERING EFFORT
Specify a value from 0 to 10 for the Driverless AI feature engineering
effort. Higher values generally lead to more time (and memory) spent in
feature engineering. This value defaults to 5. -   0: Keep only numeric features. Only model tuning during evolution. -   1: Keep only numeric features and frequency-encoded categoricals. Only model tuning during evolution. -   2: Similar to 1 but instead just no Text features. Some feature
    tuning before evolution. -   3: Similar to 5 but only tuning during evolution. Mixed tuning of
    features and model parameters. -   4: Similar to 5 but slightly more focused on model tuning. -   5: Balanced feature-model tuning. (Default)
-   6-7: Similar to 5 but slightly more focused on feature engineering. -   8: Similar to 6-7 but even more focused on feature engineering with
    high feature generation rate and no feature dropping even if high
    interpretability.

CHECK_DISTRIBUTION_SHIFT``

DATA DISTRIBUTION SHIFT DETECTION

Specify whether Driverless AI should detect data distribution shifts
between train/valid/test datasets (if provided). When train and test
dataset differ (or train/valid or valid/test) in terms of distribution
of data, then a model can be built with high accuracy that tells for
each row, whether the row is in train or test. Currently, this
information is only presented to the user and not acted upon.

Shifted features should either be dropped. Or more meaningful aggregate
features be created by using them as labels or bins.

Also see
drop_features_distribution_shift_threshold_auc <drop_features_distribution_shift_threshold_auc>
and check_distribution_shift_drop <check_distribution_shift_drop>.

CHECK_DISTRIBUTION_SHIFT_DROP``

DATA DISTRIBUTION SHIFT DETECTION DROP OF FEATURES

Specify whether to drop high-shift features. This defaults to AUTO. Note
that Auto for time series experiments turns this feature off.

Also see
and check_distribution_shift <check_distribution_shift>.

DROP_FEATURES_DISTRIBUTION_SHIFT_THRESHOLD_AUC``

MAX ALLOWED FEATURE SHIFT (AUC) BEFORE DROPPING FEATURE

Specify the maximum allowed AUC value for a feature before dropping the
feature.

When train and test dataset differ (or train/valid or valid/test) in
terms of distribution of data, then a model can be built that tells for
each row, whether the row is in train or test. This model includes an
AUC value. If this AUC, GINI, or Spearman correlation of the model is
above the specified threshold, then Driverless AI will consider it a
strong enough shift to drop those features.

The default AUC threshold is 0.999.

check_leakage-----------------  .. container:: dropdown     **Data Leakage Detection**     Specify whether to check for data leakage for each feature. Some of    the features may contain over predictive power on the target column.    This may affect model generalization. Driverless AI runs a model to    determine the predictive power of each feature on the target    variable. Then, a simple model is built on each feature with    significant variable importance. The models with high AUC (for    classification) or R2 score (regression) are reported to the user as    potential leak.     Note that this option is always disabled if the experiment is a time    series experiment. This is set to **Auto** by default.     The equivalent config.toml parameter ischeck_leakage`.
Also see :ref:`drop_features_leakage_threshold_auc
<drop_features_leakage_threshold_auc>

drop_features_leakage_threshold_auc---------------------------------------  .. container:: dropdown     **Data Leakage Detection Dropping AUC/R2 Threshold**     If :ref:`Leakage Detection <check_leakage>` is enabled, specify the    threshold for dropping features. When the AUC (or R2 for regression),    GINI, or Spearman correlation is above this value, the feature is    dropped. This value defaults to 0.999.     The equivalent config.toml parameter isdrop_features_leakage_threshold_auc``.

LEAKAGE_MAX_DATA_SIZE``

MAX ROWS X COLUMNS FOR LEAKAGE

Specify the maximum number of (rows x columns) to trigger sampling for
leakage checks. This value defaults to 10,000,000.

MAX_FEATURES_IMPORTANCE``

MAX. NUM. FEATURES FOR VARIABLE IMPORTANCE

Specify the maximum number of features to use and show in importance
tables. For any interpretability higher than 1, transformed or original
features with low importance than top max_features_importance features
are always removed Feature importances of transformed or original
features correspondingly will be pruned. Higher values can lead to lower
performance and larger disk space used for datasets with more than 100k
columns.

enable_wide_rules---------------------  .. container:: dropdown     **Enable Wide Rules**     Enable various rules to handle wide datasets( i.e no. of columns >    no. of rows). The default value is "auto", that will automatically    enable the wide rules when detect that number of columns is greater    than number of rows. Setting "on" forces rules to be enabled regardless of any conditions. Enabling wide data rules sets allmax_cols,max_orig_col``, and ``fs_orig_tomls to large values, and enforces monotonicity to    be disabled unlessmonotonicity_constraints_dictis set or    default value ofmonotonicity_constraints_interpretability_switch` is changed. It also disables shift detection and data leakage checks. And enables :ref:`Xgboost Random Forest model <enable_xgboost_rf>
    for modeling. To disable wide rules, set enable_wide_rules to "off".

ORIG_FEATURES_FS_REPORT``

REPORT PERMUTATION IMPORTANCE ON ORIGINAL FEATURES

Specify whether Driverless AI reports permutation importance on original
features (represented as normalized change in the chosen metric) in logs
and the report file. This is disabled by default.

MAX_ROWS_FS``

MAXIMUM NUMBER OF ROWS TO PERFORM PERMUTATION-BASED FEATURE SELECTION

Specify the maximum number of rows when performing permutation feature
importance, reduced by (stratified) random sampling. This value defaults
to 500,000.

MAX_ORIG_COLS_SELECTED``

MAX NUMBER OF ORIGINAL FEATURES USED

Specify the maximum number of columns to be selected from an existing
set of columns using FEATURE SELECTION. This value defaults to
10,000000. For categorical columns, the selection is based upon how well
target encoding (or frequency encoding if not available) on categoricals
and numerics treated as categoricals helps. This is useful to reduce the
final model complexity. First the best [max_orig_cols_selected] are
found through feature selection methods and then these features are used
in feature evolution (to derive other features) and in modelling.

MAX_ORIG_NONNUMERIC_COLS_SELECTED``

MAX NUMBER OF ORIGINAL NON-NUMERIC FEATURES

Maximum number of non-numeric columns selected, above which will do
feature selection on all features and avoid treating numerical as
categorical same as above (max_orig_numeric_cols_selected) but for
categorical columns. Feature selection is performed on all features when
this value is exceeded. This value defaults to 300.

FS_ORIG_COLS_SELECTED``

MAX NUMBER OF ORIGINAL FEATURES USED FOR FS INDIVIDUAL

Specify the maximum number of features you want to be selected in an
experiment. This value defaults to 10,0000000. Additional columns above
the specified value add special individual with original columns
reduced.

FS_ORIG_NUMERIC_COLS_SELECTED``

NUMBER OF ORIGINAL NUMERIC FEATURES TO TRIGGER FEATURE SELECTION MODEL
TYPE

The maximum number of original numeric columns, above which Driverless
AI will do feature selection. Note that this is applicable only to
special individuals with original columns reduced. A separate individual
in the genetic algorithm <ga> is created by doing feature selection by
permutation importance on original features. This value defaults to
10,000000.

FS_ORIG_NONNUMERIC_COLS_SELECTED``

NUMBER OF ORIGINAL NON-NUMERIC FEATURES TO TRIGGER FEATURE SELECTION
MODEL TYPE

The maximum number of original non-numeric columns, above which
Driverless AI will do feature selection on all features. Note that this
is applicable only to special individuals with original columns reduced.
A separate individual in the genetic algorithm <ga> is created by doing
feature selection by permutation importance on original features. This
value defaults to 200.

MAX_RELATIVE_CARDINALITY``

MAX ALLOWED FRACTION OF UNIQUES FOR INTEGER AND CATEGORICAL COLUMNS

Specify the maximum fraction of unique values for integer and
categorical columns. If the column has a larger fraction of unique
values than that, it will be considered an ID column and ignored. This
value defaults to 0.95.

num_as_cat--------------  .. container:: dropdown     **Allow Treating Numerical as Categorical**     Specify whether to allow some numerical features to be treated as    categorical features. This is enabled by default.     The equivalent config.toml parameter isnum_as_cat``.

MAX_INT_AS_CAT_UNIQUES``

MAX NUMBER OF UNIQUE VALUES FOR INT/FLOAT TO BE CATEGORICALS

Specify the number of unique values for integer or real columns to be
treated as categoricals. This value defaults to 50.

MAX_FRACTION_INVALID_NUMERIC``

MAX. FRACTION OF NUMERIC VALUES TO BE NON-NUMERIC (AND NOT MISSING) FOR
A COLUMN TO STILL BE CONSIDERED NUMERIC

When the fraction of non-numeric (and non-missing) values is less or
equal than this value, consider the column numeric. Can help with minor
data quality issues for experimentation, not recommended for production,
since type inconsistencies can occur. NOTE: Replaces non-numeric values
with missing values at start of experiment, so some information is lost,
but column is now treated as numeric, which can help. Disabled if < 0.

nfeatures_max-----------------  .. container:: dropdown     **Max Number of Engineered Features**     Specify the maximum number of features to be included per model (and    in each model within the final model if an ensemble). After each    scoring, based on this parameter value, keeps top variable importance    features, and prunes away rest of the features. Final ensemble will    exclude any pruned-away features and only train on kept features, but    may contain a few new features due to fitting on different data view    (e.g. new clusters). Final scoring pipeline will exclude any    pruned-away features, but may contain a few new features due to    fitting on different data view (e.g. new clusters). The default value of **-1** means no restrictions are applied for    this parameter except internally-determined memory and    interpretability restrictions.

ngenes_max--------------  .. container:: dropdown     **Max Number of Genes**     Specify the maximum number of genes (transformer instances) kept per    model (and per each model within the final model for ensembles). This    controls the number of genes before features are scored, so    Driverless AI will just randomly samples genes if pruning occurs. If    restriction occurs after scoring features, then aggregated gene    importances are used for pruning genes. Instances includes all    possible transformers, including original transformer for numeric    features. A value of -1 means no restrictions except    internally-determined memory and interpretability restriction.     The equivalent config.toml parameter isngenes_max``.

features_allowed_by_interpretability----------------------------------------  .. container:: dropdown     **Limit Features by Interpretability**     Specify whether to limit feature counts with the **Interpretability**    training setting as specified by thefeatures_allowed_by_interpretability`
:ref:`config.toml <sample-configtoml> setting.

MONOTONICITY_CONSTRAINTS_INTERPRETABILITY_SWITCH``

THRESHOLD FOR INTERPRETABILITY ABOVE WHICH TO ENABLE AUTOMATIC
MONOTONICITY CONSTRAINTS FOR TREE MODELS

Specify an Interpretability setting value equal and above which to use
automatic monotonicity constraints in XGBoostGBM, LightGBM, or Decision
Tree models. This value defaults to 7.

Also see monotonic gbm recipe <pipeline-building-recipe> and
Monotonicity Constraints in Driverless AI <mc> for reference.

MONOTONICITY_CONSTRAINTS_CORRELATION_THRESHOLD``

CORRELATION BEYOND WHICH TO TRIGGER MONOTONICITY CONSTRAINTS (IF
ENABLED)

Specify the threshold of Pearson product-moment correlation coefficient
between numerical or encoded transformed feature and target above (below
negative for) which to use positive (negative) monotonicity for
XGBoostGBM, LightGBM and Decision Tree models. This value defaults to
0.1.

NOTE: This setting is only enabled when Interpretability is greater than
or equal to the value specified by the enable-constraints setting and
when the constraints-override setting is not specified.

MONOTONICITY_CONSTRAINTS_LOG_LEVEL``

CONTROL AMOUNT OF LOGGING WHEN CALCULATING AUTOMATIC MONOTONICITY
CONSTRAINTS (IF ENABLED)

For models that support monotonicity constraints, and if enabled, show
automatically determined monotonicity constraints for each feature going
into the model based on its correlation with the target. 'low' shows
only monotonicity constraint direction. 'medium' shows correlation of
positively and negatively constraint features. 'high' shows all
correlation values.

MONOTONICITY_CONSTRAINTS_DROP_LOW_CORRELATION_FEATURES``

WHETHER TO DROP FEATURES THAT HAVE NO MONOTONICITY CONSTRAINT APPLIED
(E.G., DUE TO LOW CORRELATION WITH TARGET)

If enabled, only monotonic features with +1/-1 constraints will be
passed to the model(s), and features without monotonicity constraints
(0) will be dropped. Otherwise all features will be in the model. Only
active when interpretability >=
monotonicity_constraints_interpretability_switch or
monotonicity_constraints_dict is provided.

MONOTONICITY_CONSTRAINTS_DICT``

MANUAL OVERRIDE FOR MONOTONICITY CONSTRAINTS

Specify a list of features for max_features_importance which
monotonicity constraints are applied. Original numeric features are
mapped to the desired constraint:

-   1: Positive constraint
-   -1: Negative constraint
-   0: Constraint disabled

Constraint is automatically disabled (set to 0) for features that are
not in this list.

The following is an example of how this list can be specified:

    "{'PAY_0': -1, 'PAY_2': -1, 'AGE': -1, 'BILL_AMT1': 1, 'PAY_AMT1': -1}"

NOTE: If a list is not provided, then the automatic correlation-based
method is used when monotonicity constraints are enabled at high enough
interpretability settings.

See Monotonicity Constraints in Driverless AI <mc> for reference.

max_feature_interaction_depth---------------------------------  .. container:: dropdown     **Max Feature Interaction Depth**     Specify the maximum number of features to use for interaction    features like grouping for target encoding, weight of evidence, and    other likelihood estimates. Exploring feature interactions can be important in gaining better    predictive performance. The interaction can take multiple forms (i.e. feature1 + feature2 or feature1 \* feature2 + … featureN). Although    certain machine learning algorithms (like tree-based methods) can do    well in capturing these interactions as part of their training    process, still generating them may help them (or other algorithms)    yield better performance. The depth of the interaction level (as in "up to" how many features    may be combined at once to create one single feature) can be    specified to control the complexity of the feature engineering    process.

FIXED_FEATURE_INTERACTION_DEPTH``

FIXED FEATURE INTERACTION DEPTH

Specify a fixed non-zero number of features to use for interaction
features like grouping for target encoding, weight of evidence, and
other likelihood estimates. To use all features for each transformer,
set this to be equal to the number of columns. To do a 50/50 sample and
a fixed feature interaction depth of n features, set this to -n.

ENABLE_TARGET_ENCODING``

ENABLE TARGET ENCODING

Specify whether to use Target Encoding when building the model. Target
encoding refers to several different feature transformations (primarily
focused on categorical data) that aim to represent the feature using
information of the actual target variable. A simple example can be to
use the mean of the target to replace each unique category of a
categorical feature. These type of features can be very predictive but
are prone to overfitting and require more memory as they need to store
mappings of the unique categories and the target values.

cvte_cv_in_cv-----------------  .. container:: dropdown     **Enable Outer CV for Target Encoding**     For target encoding, specify whether an outer level of cross-fold    validation is performed in cases where GINI is detected to flip sign    or have an inconsistent sign for weight of evidence betweenfit_transform(on training data) andtransform`` (on training

    and validation data). The degree to which GINI is inaccurate is also
    used to perform fold-averaging of look-up tables instead of using
    global look-up tables. This is enabled by default.

ENABLE_LEXILABEL_ENCODING``

ENABLE LEXICOGRAPHICAL LABEL ENCODING

Specify whether to enable lexicographical label encoding. This is

ENABLE_ISOLATION_FOREST``

ENABLE ISOLATION FOREST ANOMALY SCORE ENCODING

Isolation Forest is useful for identifying anomalies or outliers in
data. Isolation Forest isolates observations by randomly selecting a
feature and then randomly selecting a split value between the maximum
and minimum values of that selected feature. This split depends on how
long it takes to separate the points. Random partitioning produces
noticeably shorter paths for anomalies. When a forest of random trees
collectively produces shorter path lengths for particular samples, they
are highly likely to be anomalies.

This option lets you specify whether to return the anomaly score of each
sample. This is disabled by default.

ENABLE_ONE_HOT_ENCODING``

ENABLE ONE HOTENCODING

Specify whether one-hot encoding is enabled. The default AUTO setting is
only applicable for small datasets and GLMs.

ISOLATION_FOREST_NESTIMATORS``

NUMBER OF ESTIMATORS FOR ISOLATION FOREST ENCODING

Specify the number of estimators for Isolation Forest encoding. This

DROP_CONSTANT_COLUMNS``

DROP CONSTANT COLUMNS

Specify whether to drop columns with constant values. This is enabled by
default.

DROP_ID_COLUMNS``

DROP ID COLUMNS

Specify whether to drop columns that appear to be an ID. This is enabled

NO_DROP_FEATURES``

DON'T DROP ANY COLUMNS

Specify whether to avoid dropping any columns (original or derived).

COLS_TO_DROP``

FEATURES TO DROP

Specify which features to drop. This setting allows you to select many
features at once by copying and pasting a list of column names (in
quotes) separated by commas.

COLS_TO_FORCE_IN``

FEATURES TO ALWAYS KEEP OR FORCE IN, E.G. "G1", "G2", "G3"

Control over columns to force-in. Forced-in features are handled by the
most interpretable transformers allowed by the experiment options, and
they are never removed (even if the model assigns 0 importance to them).
Transformers used by default includes:

  -   OriginalTransformer for numeric,
  -   CatOriginalTransformer or FrequencyTransformer for categorical,
  -   TextOriginalTransformer for text,
  -   DateTimeOriginalTransformer for date-times,
  -   DateOriginalTransformer for dates,
  -   ImageOriginalTransformer or ImageVectorizerTransformer for images,
      etc

COLS_TO_GROUP_BY``

FEATURES TO GROUP BY

Specify which features to group columns by. When this field is left
empty (default), Driverless AI automatically searches all columns
(either at random or based on which columns have high variable
importance).

SAMPLE_COLS_TO_GROUP_BY``

SAMPLE FROM FEATURES TO GROUP BY

Specify whether to sample from given features to group by or to always
group all features. This is disabled by default.

AGG_FUNCS_FOR_GROUP_BY``

AGGREGATION FUNCTIONS (NON-TIME-SERIES) FOR GROUP BY OPERATIONS

Specify whether to enable aggregation functions to use for group by
operations. Choose from the following (all are selected by default):

-   mean
-   sd
-   min
-   max
-   count

FOLDS_FOR_GROUP_BY``

NUMBER OF FOLDS TO OBTAIN AGGREGATION WHEN GROUPING

Specify the number of folds to obtain aggregation when grouping.
Out-of-fold aggregations will result in less overfitting, but they
analyze less data in each fold. The default value is 5.

MUTATION_MODE``

TYPE OF MUTATION STRATEGY

Specify which strategy to apply when performing mutations on
transformers. Select from the following:

-   SAMPLE: Sample transformer parameters (Default)
-   BATCHED: Perform multiple types of the same transformation together
-   FULL: Perform more types of the same transformation together than
    the above strategy

DUMP_VARIMP_EVERY_SCORED_INDIV``

ENABLE DETAILED SCORED FEATURES INFO

Specify whether to dump every scored individual's variable importance
(both derived and original) to a csv/tabulated/json file. If enabled,
Driverless AI produces files such as
"individual_scored_id%d.iter%d*features*". This is disabled by default.

DUMP_TRANS_TIMINGS``

ENABLE DETAILED LOGS FOR TIMING AND TYPES OF FEATURES PRODUCED

Specify whether to dump every scored fold's timing and feature info to a
TIMINGS.TXT file. This is disabled by default.

COMPUTE_CORRELATION``

COMPUTE CORRELATION MATRIX

Specify whether to compute training, validation, and test correlation
matrixes. When enabled, this setting creates table and heatmap PDF files
that are saved to disk. Note that this setting is currently a single
threaded process that may be slow for experiments with many columns.

INTERACTION_FINDER_GINI_REL_IMPROVEMENT_THRESHOLD``

REQUIRED GINI RELATIVE IMPROVEMENT FOR INTERACTIONS

Specify the required GINI relative improvement value for the
InteractionTransformer. If the GINI coefficient is not better than the
specified relative improvement value in comparison to the original
features considered in the interaction, then the interaction is not
returned. If the data is noisy and there is no clear signal in
interactions, this value can be decreased to return interactions. This
value defaults to 0.5.

INTERACTION_FINDER_RETURN_LIMIT``

NUMBER OF TRANSFORMED INTERACTIONS TO MAKE

Specify the number of transformed interactions to make from generated
trial interactions. (The best transformed interactions are selected from
the group of generated trial interactions.) This value defaults to 5.

enable_rapids_transformers------------------------------  .. container:: dropdown     **Whether to enable RAPIDS cuML GPU transformers (no mojo)**     Specify whether to enable GPU-based `RAPIDS    cuML <https://docs.rapids.ai/api/cuml/nightly/>`__ transformers. Note    that **no MOJO** support for deployment is available for this    selection at this time, but python scoring is supported and this is    in beta testing status.     The equivalent config.toml parameter isenable_rapids_transformers``
and the default value is False.

VARIMP_THRESHOLD_AT_INTERPRETABILITY_10``

LOWEST ALLOWED VARIABLE IMPORTANCE AT INTERPRETABILITY 10

Specify the variable importance below which features are dropped (with
the possibility of a replacement being found that's better). This
setting also sets the overall scale for lower interpretability settings.
Set this to a lower value if you're content with having many weak
features despite choosing high interpretability, or if you see a drop in
performance due to the need for weak features.

STABILIZE_FS``

WHETHER TO TAKE MINIMUM (TRUE) OR MEAN (FALSE) OF DELTA IMPROVEMENT IN
SCORE WHEN AGGREGATING FEATURE SELECTION SCORES ACROSS MULTIPLE
FOLDS/DEPTHS
Whether to take minimum (True) or mean (False) of delta improvement in
score when aggregating feature selection scores across multiple
folds/depths. Delta improvement of score corresponds to original metric
minus metric of shuffled feature frame if maximizing metric, and
corresponds to negative of such a score difference if minimizing. Feature selection by permutation importance considers the change in
score after shuffling a feature, and using minimum operation ignores
optimistic scores in favor of pessimistic scores when aggregating over
folds. Note, if using tree methods, multiple depths may be fitted, in
which case regardless of this toml setting, only features that are kept
for all depths are kept by feature selection.

R CLIENT

This section describes how to install the Driverless AI R client.
also provided. For more information on the R client, see the Driverless
AI R client documentation.

r_install_client r_client_tutorial

SCORING PIPELINES

python-mojo-pipelines scoring_pipeline_visualize
scoring-pipeline-which-to-use scoring-standalone-python
scoring-mli-standalone-python scoring-mojo-pipelines

CHECKPOINTING, RERUNNING, AND RETRAINING EXPERIMENTS

The upper-right corner of the Driverless AI UI includes an EXPERIMENTS
link. []
Click this link to open the Experiments page. From this page, you can
rename an experiment, view previous experiments, begin a new experiment,
rerun an experiment, and delete an experiment. []

Checkpointing, Rerunning, and Retraining
In Driverless AI, you can retry an experiment from the last checkpoint,
you can run a new experiment using an existing experiment's settings,
and you can retrain an experiment's final pipeline. []
Checkpointing Experiments
In real-world scenarios, data can change. For example, you may have a
model currently in production that was built using 1 million records. At
a later date, you may receive several hundred thousand more records. Rather than building a new model from scratch, Driverless AI includes
H2O.ai Brain, which enables caching and smart re-use of prior models to
generate features for new models.

-   -1: Don’t use any brain cache
-   0: Don’t use any brain cache but still write to cache
-   1: Smart checkpoint if an old experiment_id is passed in (for
    example, via running “resume one like this” in the GUI)
-   2: Smart checkpoint if the experiment matches all column names,
-   3: Smart checkpoint like level #1, but for the entire population. Tune only if the brain population is of insufficient size. -   4: Smart checkpoint like level #2, but for the entire population. Tune only if the brain population is of insufficient size. -   5: Smart checkpoint like level #4, but will scan over the entire
    brain cache of populations (starting from resumed experiment if
    chosen) in order to get the best scored individuals. If you chooses Level 2 (default), then Level 1 is also done when
appropriate.

(This
    won't match if, e.g,. a column was all int and then had one string
    row.) -   The same target as the old experiment
-   The same target classes (if classification) as the old experiment
-   For time series, all choices for intervals and gaps must be the same
When the above conditions are met, then you can:
-   Start the same kind of experiment, just rerun for longer. -   Use a smaller or larger data set (i.e. fewer or more rows). -   Effectively do a final ensemble re-fit by varying the data rows and
    starting an experiment with a new accuracy, time=1, and
    interpretability. Check the experiment preview for what the ensemble
    will be. -   Restart/Resume a cancelled, aborted, or completed experiment
To run smart checkpointing on an existing experiment, click the right
side of the experiment that you want to retry, then select NEW /
CONTINUE -> FROM LAST CHECKPOINT. The experiment settings page opens. Specify the new dataset.

Click LAUNCH
EXPERIMENT to resume the experiment from the last checkpoint and build a
new experiment. The smart checkpointing continues by adding a prior model as another
model used during tuning. If that prior model is better (which is likely
if it was run for more iterations), then that smart checkpoint model
will be used during feature evolution iterations and final ensemble. NOTES:
-   Driverless AI does not guarantee exact continuation, only smart
    continuation from any last point. -   The directory where the H2O.ai Brain meta model files are stored is
    TMP/H2O.AI_BRAIN. In addition, the default maximum brain size is
    20GB. Both the directory and the maximum size can be changed in the
    config.toml file. Rerunning Experiments
To run a new experiment using an existing experiment's settings, click
the right side of the experiment that you want to use as the basis for
the new experiment, then select NEW EXPERIMENT WITH SAME SETTINGS.

From this page, you can rerun the
experiment using the original settings, or you can specify to use new
data and/or specify different experiment settings. Click LAUNCH
EXPERIMENT to create a new experiment with the same options. Retrain / Refit
To retrain an experiment's final pipeline, click on the group of square
icons next to the experiment that you want to use as the basis for the
new experiment and click RETRAIN / REFIT, then select FROM FINAL
CHECKPOINT. This opens the experiment settings page with the same
settings as the original experiment except that Time is set to 0. NOTE: When retraining an experiment's final pipeline, Driverless AI also
refits the experiment's final model(s). This may include the addition of
new features, the exclusion of previously used features, a change in the
hyperparameter search space, or finding new parameters for the existing
model architecture.

Note
For information on the equivalent Python client <python_client> calls
for Retrain / Refit options, refer to the following list. -   New / Continue - With Same Settings:
          retrain(...)
-   New / Continue - From Last Checkpoint:
          retrain(..., use_smart_checkpoint=True)
-   Retrain / Refit - From Final Checkpoint
          retrain(..., final_pipeline_only=True)
-   Retrain / Refit - From Best Models (1.10.1 client)
          retrain(..., final_models_only=True)

"Pausing" an Experiment
A trick for "pausing" an experiment is to:
1. Abort the experiment. 2. On the Experiments page, select RESTART FROM LAST CHECKPOINT for the
    aborted experiment. 3. On the Expert Settings page, specify 0 for the
    ensemble level for the final pipeline <fixed_ensemble_level> option. Deleting Experiments
To delete an experiment, click the right side of the experiment that you
want to remove, then select DELETE.

REDIS MULTINODE TRAINING

Redis Multinode training in Driverless AI can be used to run multiple
experiments at the same time. It is EFFECTIVE IN SITUATIONS where you
need TO RUN AND COMPLETE MANY EXPERIMENTS SIMULTANEOUSLY IN A SHORT
AMOUNT OF TIME without having to wait for each individual experiment to
finish.

Understanding Redis Multinode Training

Redis multinode training uses a load distribution technique in which a
set of machines (worker nodes) are used to help a main server node
process experiments. These machines can be CPU only or CPU + GPU, with
experiments being distributed accordingly.

[]

Jobs (experiments) within the multinode setup are organized into a
queue <dai-queuing>. Jobs remain in this queue when no processor is
available. When a worker's processor becomes available, it asks the job
queue service to assign it a new job. By default, each worker node
processes two jobs at a time (configured with the

worker_remote_processorsoption in the config.toml file). Each worker can process multiple jobs at the same time, but two workers cannot process the same experiment at the same time. Messaging and data exchange services are also implemented to allow the workers to effectively communicate with the main server node. **Notes**:  -  Redis multinode training in Driverless AI is currently in a preview    stage. If you are interested in using multinode configurations,    contact support@h2o.ai. -  Redis multinode training requires the transfer of data to several    different workers. For example, if an experiment is scheduled to be    on a remote worker node, the datasets it is using need to be copied    to the worker machine by using the MinIO service. The experiment can    take longer to initialize depending on the size of the transferred    objects. -  The number of jobs that each worker node processes is controlled by    theworker_remote_processors`
option in the config.toml file.

Workers consume tasks from the queue on a first in, first out
(FIFO) basis. - **A single experiment runs entirely on one machine (or
node)**. For this reason, using a large number of commodity-grade
hardware is not useful in the context of multinode. - For more
information on queuing in Driverless AI, see :ref:`dai-queuing. REQUIREMENTS

-   Redis

REDIS MULTINODE SETUP EXAMPLE

The following example configures a two-node Redis Multinode Driverless
AI cluster on AWS EC2 instances using bashtar distribution. This example
can be expanded to multiple worker nodes. This example assumes that you
have spun up two EC2 instances (Ubuntu 16.04) within the same VPC on
AWS. VPC Settings
In the VPC settings, enable inbound rules to listen to TCP connections
on port 6379 for Redis and 9000 for MinIO. Install Driverless AI Natively
Install Driverless AI on the server node. Refer to one of the following
documents for information on how to perform a native install on Linux
systems.

enable_dask_cluster = false`` would not be done.

START THE DRIVERLESS AI SERVER NODE

    cd |VERSION-dir|-linux-x86_64
    ./run-dai.sh

Install the Linux deb/rpm/tar package on the EC2 instance to create a
Driverless AI worker node. After the installation is complete, edit the
following in the config.toml.

    # Redis settings, point to the dai main server's redis server ip address
    redis_ip = "<dai_main_server_host_ip>"

    # Redis settings
    redis_port = 6379

    # Redis settings, point to the dai main server's redis server password
    main_server_redis_password = "<dai_main_server_host_redis_pwd>"

    # Location of the dai main server's minio server.
    main_server_minio_address = "<dai_main_server_host>:9000"

    enable_dask_cluster = false

To use the full multinode with both redis and dask support, see the
example multinode-example, in which case

START THE DRIVERLESS AI WORKER NODE

    ./run-dai.sh --worker

    # Note that when using rpm/deb you can run the following:
    sudo systemctl start dai-worker

Once the worker node starts, use the Driverless AI server IP to log into
Driverless AI. Click on RESOURCES > SYSTEM INFO to confirm that the
number of workers is "2" if only one worker is used. (By default, each
worker node processes two jobs at a time. This is configured with the

worker_remote_processorsoption in the config.toml file.) .. figure:: images/system_info_view.png    :alt:   .. _multinode-config-attributes:  Description of Configuration Attributes ---------------------------------------  -worker_mode: Specifies how the long-running tasks are scheduled. Available options include:     -multiprocessing: Forks the current process immediately. -singlenode: Shares the task through Redis and needs a worker       running. -multinode: Same assinglenode. Also shares the data       through MinIO and allows the worker to run on the different       machine. -redis_ip: Redis IP address. Defaults to 127.0.0.1 -redis_port: Redis port. Defaults to 6379. -redis_db: Redis database. Each DAI instance running on the Redis    server should have unique integer. Defaults to 0. -main_server_redis_password: Main Server Redis password. Defaults    to empty string. -local_minio_port: The port that MinIO will listen on.

-main_server_minio_address: The address of the main server's MinIO    server. Defaults to 127.0.0.1:9000. -main_server_minio_access_key_id: Access key of main server's    MinIO server. -main_server_minio_secret_access_key: The secret access key of    main server MinIO server. -main_server_minio_bucket: The name of MinIO bucket used for file    synchronization. -worker_local_processors: The maximum number of local tasks    processed at once. Defaults to 32. -worker_remote_processors: The maximum number of remote tasks    processed at once. Defaults to 2. -redis_result_queue_polling_interval: The frequency in    milliseconds that the server should extract results from the Redis    queue. Defaults to 100. -main_server_minio_bucket_ping_timeout: The number of seconds the    worker should wait for the main server MinIO bucket before it fails. Defaults to 30. -worker_healthy_response_period``: The number of seconds to wait for
    the worker to respond before being marked as unhealthy.

PYTHON CLIENT ADMIN API

The following sections describe Driverless AI's Admin API. Note
The Admin API is currently only available through the DAI Python client. Understanding the Admin API
The Driverless AI Admin API lets you manage entities created by other
users by providing options for listing, deleting, or transferring them. The primary component of the Admin API is the new user role called
ADMIN. Driverless AI currently supports only local Admin user
authorization, which is defined through the local_administrator_list
config parameter. For example, to promote UserA and UserB to
administrator, add the following config override to the config.toml
file:
    local_administrator_list = ['UserA', 'UserB']

Admin API methods
The following is a list of DAI Admin API methods. Note
The following examples assume that you have initialized the h2oai Python
client and are logged in with a user that has the Admin role.

SNOWFLAKE SETUP

Driverless AI lets you explore Snowflake data sources from within the
configuring Driverless AI to work with Snowflake. This setup requires
you to enable authentication. If you enable Snowflake connectors, those
file systems will be available in the UI, but you will not be able to
use those connectors without authentication.

docker run --runtime=nvidia(>= Docker 19.03) ornvidia-docker(< Docker 19.03) command when starting the Driverless AI Docker image. Usedocker versionto check which version of Docker you are using. Description of Configuration Attributes ---------------------------------------  -snowflake_account: The Snowflake account ID -snowflake_user: The username for accessing the Snowflake account -snowflake_password: The password for accessing the Snowflake    account -enabled_file_systems: The file systems you want to enable. This    must be configured in order for data connectors to function properly. Enable Snowflake with Authentication ------------------------------------  .. container:: tabs     .. group-tab:: Docker Image Installs     This example enables the Snowflake data connector with authentication    by passing theaccount,user, andpasswordvariables.

1. Configure the Driverless AI config.toml file. Set the following          configuration options. ..           -enabled_file_systems = "file, snow"-snowflake_account = "<account_id>"-snowflake_user = "<username>"-snowflake_password = "<password>"2. Mount the config.toml file into the Docker container.

1. Export the Driverless AI config.toml file or add it to          ~/.bashrc. For example:        ..           ::              # DEB and RPM             export DRIVERLESS_AI_CONFIG_FILE="/etc/dai/config.toml"              # TAR SH             export DRIVERLESS_AI_CONFIG_FILE="/path/to/your/unpacked/dai/directory/config.toml"         2. Specify the following configuration options in the config.toml          file.

(jdbc_app_configs)             # hive: Hive Connector, remember to configure Hive below. (hive_app_configs)             # recipe_url: load custom recipe from URL             # recipe_file: load custom recipe from local file system             enabled_file_systems = "file, snow"              # Snowflake Connector credentials             snowflake_account = "<account_id>"             snowflake_user = "<username>"             snowflake_password = "<password>"        3. Save the changes when you are done, then stop/restart          Driverless AI. Snowflake single sign-on (SSO) ------------------------------  The Snowflake connector features support for single sign-on (SSO). This means that with the proper setup, you can log into DAI, connect to Snowflake, and add datasets from Snowflake without having to enter any credentials in the UI or config.toml. To ensure that SSO works correctly, DAI and Snowflake must be configured with the same OpenID Connect (OIDC) authentication provider.

For Snowflake, refer to the `External OAuth Overview <https://docs.snowflake.com/en/user-guide/oauth-ext-overview>`__ in the official Snowflake documentation. Adding Datasets Using Snowflake -------------------------------  After the Snowflake connector is enabled, you can add datasets by selecting **Snowflake** from the **Add Dataset (or Drag and Drop)** drop-down menu. .. figure:: ../images/add_dataset_dropdown.png    :alt:     :width: 237px    :height: 338px  Specify the following information to add your dataset. 1. **Enter Database**: Specify the name of the Snowflake database that    you are querying. 2. **Enter Warehouse**: Specify the name of the Snowflake warehouse that    you are querying. 3. **Enter Schema**: Specify the schema of the dataset that you are    querying. 4. **Enter Name for Dataset to Be Saved As**: Specify a name for the    dataset to be saved as. Note that this can only be a CSV file (for    example, **myfile.csv**). 5.

This can be left blank ifsnowflake_userwas specified in the config.toml when starting Driverless AI;    otherwise, this field is required. 6. **Enter Password**: (Optional) Specify the password associated with    this Snowflake account. This can be left blank ifsnowflake_passwordwas specified in the config.toml when starting    Driverless AI; otherwise, this field is required. 7. **Enter Role**: (Optional) Specify your role as designated within    Snowflake. See    https://docs.snowflake.net/manuals/user-guide/security-access-control-overview.html    for more information. 8. **Enter Region**: (Optional) Specify the region of the warehouse that    you are querying. This can be found in the Snowflake-provided URL to    access your database (as in    **<optional-deployment-name>.<region>.<cloud-provider>.snowflakecomputing.com**). This is optional and can also be left blank ifsnowflake_urlwas    specified with a<region>in the config.toml when starting    Driverless AI. 9.

Available    parameters are listed in    https://docs.snowflake.com/en/sql-reference/sql/create-file-format.html#type-csv. (**Note**: Use only parameters forTYPE = CSV.) For example, if    your dataset includes a text column that contains commas, you can    specify a different delimiter usingFIELD_DELIMITER='character'. Multiple parameters must be separated with spaces:  ..     ::        FIELD_DELIMITER=',' FIELD_OPTIONALLY_ENCLOSED_BY="" SKIP_BLANK_LINES=TRUE     **Note**: Be sure that the specified delimiter is not also used as a    character within a cell; otherwise an error will occur.

SNOWFLAKE INTEGRATION

Overview
This document describes how to use the external function feature of
Snowflake to invoke Driverless AI models as HTTP REST API endpoints. Using the external function requires some setup and configuration in
Snowflake and Amazon. For more information, refer to the Snowflake
documentation on external functions. Note
DOWNLOADS:
-   Download the Driverless AI Snowflake Java UDF. -   Download the Driverless AI Snowflake external function
    (dai-snowflake-integration.tgz). The setup process for the Java UDF is typically easier than for the
external function. []

Requirements
1. Snowflake login credentials
2. Amazon EC2 login credentials
3. Driverless AI MOJO (pipelineSF.mojo)
    -   Included in the demo files
4. DAIMojoRestServer
5. Driverless AI license
    -   Provided through the partnership portal
    -   Copy the license to the Snowflake_H2Oai directory. Name the file
        license.sig. 6.

To use a different JVM, see the
        following steps:
          -   Execute “java -version” to verify the available JVM on
              your platform has JDK 1.8 installed. The output of the
              command should indicate JDK 1.8, for example:
          -   If the output does not show JDK 1.8, download a 1.8 JDK
              for your environment from one of the following sites:
                -   https://www.azul.com/downloads/zulu-community/
                -   https://openjdk.java.net/install/

Security
When using the external function, a call is made from Snowflake to the
AWS API Gateway. This requires the configuration of trust relationships
in AWS so that the call can be made. The H2O REST Server only accepts calls from the AWS Gateway endpoint. When the parameter
-DSecureModelAllowAgent=”AmazonAPIGateway._|snowflake._” is added to the
command line, it’s even possible to further limit this to a specific AWS
function.

Installation
DOWNLOADS
Download the Driverless AI Snowflake Java UDF. Download the Driverless AI Snowflake external function
(dai-snowflake-integration.tgz). Note that the setup process for the Java UDF is typically easier than
for the external function. The following installation includes steps in Snowflake, AWS, and an EC2
instance where the H2O REST server is installed. The following steps outline the REST server installation:
1. Create an EC2 Instance, a demo system should have the following
    minimum specification:
      -   Operating System: Linux
      -   CPU: 2
      -   Memory: 16GB
      -   Disk: 500MB
2. Copy the distribution to the EC2 instance and extract the file. 3. Create the database. 4. Populate the table with the sample data. 5. Verify that the data is available. Starting the REST Server
Use the following steps to start the H2O REST server on the EC2
instance. 1. Ensure the current working directory is Snowflake-H2Oai/Function. 2.

The log is written to
    nohup.log. 3. The REST server initiates after several seconds have passed. Check
    for a ready message similar to the following:
Verify REST Server Installation
To verify that the REST server and its model components were installed
successfully and that the server initialized correctly:
1. Ensure that the current working directory is
    Snowflake-H2Oai/Function
2.

-   The number of parameters matches the number of parameters that the
    model was created with. -   If the data in the table is a float and the function uses the
SQL Examples
Once the Snowflake and AWS Gateway has been configured, the following
example SQL statements return predictions:
      select H2OPredict('Modelname=pipelineSF.mojo’, LOAN_AMNT, TERM, INT_RATE, INSTALLMENT, EMP_LENGTH, ANNUAL_INC, VERIFICATION_STATUS, ADDR_STATE, DTI, DELINQ_2YRS, REVOL_BAL, REVOL_UTIL ) from LENDINGCLUB where ADDR_STATE='NJ' order by ID;
Passing Runtime Parameters
The following is a list of parameters used to pass specific values to
the REST server:
-   MODELNAME: The name of the Driverless AI MOJO file that exists in
    the REST server ModelDirectory. This is pipeline.mojo by default. -   PREDICTION: The numeric prediction to use. This is 0 by default.

In the following example, the HTTP header modelname is added to
the function and is defined as pipeline.mojo for all calls to this
function. create or replace external function H2OPredictHDR(v0 number, v1 varchar, v2 number, v3 number, v4 number, v5 number, v6 varchar, v7 varchar, v8 number, v9 number, v10 number, v11 number)
    returns variant
    HEADERS=('modelname' = 'pipelineSF.mojo')
    api_integration = demonstration_external_api_integration_01
    as 'https://aaaaaaaa.execute-api.us-east-1.amazonaws.com/production';     
This allows function calls to not require any parameters. A function by
itself is enough for each model:
    select id, H2OPredictHDR(LOAN_AMNT, TERM, INT_RATE, INSTALLMENT, EMP_LENGTH, ANNUAL_INC, VERIFICATION_STATUS, 
                  ADDR_STATE, DTI, DELINQ_2YRS, REVOL_BAL, REVOL_UTIL ) from LENDINGCLUB;
The prediction can also be passed if required. Otherwise, a probability
of 0 is returned.

When requesting Driverless AI to
build a model from a worksheet, the build status is updated in a table
called MODELBUILD so that the build can be monitored. []
This shows the different stages for the riskmodel being built, from
requesting the specific table in Snowflake to final deployment for
scoring in the REST Server. NOTE: When the build finishes, the build experiment UUID is reported for
auditability purposes.

Configuration
The script buildmodel.py must be copied from the SnowFlake Function
directory to the location where the REST server executes. By default,
this is /home/ec2-user/Snowflake-H2Oai/Function. NOTE: The script code must be updated based on the environment you are
using. Driverless AI Snowflake Configuration
The Driverless AI configuration uses the standard default settings
except for settings related to user security. Use the authentication
method that is best suited to the environment that you are using. For
more information, see config_file and dai_auth. authentication_method = "local"
    local_htpasswd_file = "/home/ec2-user/dai-1.8.5.1-linux-x86_64/.htpasswd"  
To create a username and password using local authentication:
    sudo htpasswd -B -c .htpasswd snowflake              
    Password yourpassword
The build functionality invokes a Python program that uses the
Driverless AI Python Client to create an experiment.

H2O DRIVE SETUP

H2O Drive is an object-store for H2O AI Cloud. This page describes how
to configure Driverless AI to work with H2O Drive. NOTE: For more information on the H2O Drive, refer to the official
documentation. Description of relevant configuration attributes
    enable the Feature Store data connector, h2o_drive must be added to
    this list of data sources. -   h2o_drive_endpoint_url: The H2O Drive server endpoint URL. -   h2o_drive_access_token_scopes: A space-separated list of OpenID
    scopes for the access token that are used by the H2O Drive
    connector. -   h2o_drive_session_duration: The maximum duration in seconds for a
    session with the H2O Drive. -   authentication_method: The authentication method used by DAI.

CHOOSING AN INSTALL METHOD

Consider the following when choosing between the AWS Marketplace and AWS
Community AMIs:

Driverless AI AWS Marketplace AMI

-   Native (Debian) install based
-   Certified by AWS
-   Will typically lag behind our standard releases, and may require
    updates to work with the latest versions of Driverless AI
-   Features several default configurations like default password and
    HTTPS configuration, which are required by AWS

Driverless AI AWS Community AMI

-   Docker based
-   Not certified by AWS
-   Will typically have an up-to-date version of Driverless AI for both
    LTS and latest stable releases
-   Base Driverless AI installation on Docker does not feature preset
    configurations

REPRODUCIBILITY IN DRIVERLESS AI

Reproducibility refers to the ability to reproduce or replicate the
results of a machine learning model or experiment. This concept is
essential in machine learning because it lets you validate the results
of your experiments (by ensuring that your findings are not the result
of chance or errors in the implementation) and lets others build on
them. Reproducibility also promotes transparency and accountability,
which are important for ensuring that machine learning models are safe,
ethical, and trustworthy. Driverless AI lets you build experiments with
a random seed to get reproducible results when running experiments. Enable reproducibility
When setting up an experiment, the REPRODUCIBLE toggle lets you specify
whether to enable reproducibility. By default, this toggle is disabled. When enabled, the Reproducible toggle works in tandem with the
reproducibility_level config option to enable a specific level of
reproducibility.

reproducibility_level=1`` by default. []
The following section describes the different levels of reproducibility
in more detail. REPRODUCIBILITY LEVELS

You can manually specify one of the four available levels of
reproducibility with the reproducibility_level config option. The
following list describes how these levels of reproducibility are
distinct from one another. -   1 (DEFAULT): Same experiment results for same operating system, same
    CPU(s), and same GPU(s). -   2: Same experiment results for same operating system, same CPU
    architecture, and same GPU architecture. -   3: Same experiment results for same operating system and same CPU
    architecture. Note that this reproducibility level excludes GPUs. -   4: Same experiment results for same operating system. This level is
    considered to be the best effort approximation.

OPENID CONNECT AUTHENTICATION EXAMPLES

This section describes how to enable OpenID Connect authentication in
Driverless AI. It provides two examples. The first describes how to
enable OpenID connect and log in to the Driverless AI UI. The second
describes additional token-based authentication settings, which allows
you to run the Driverless AI Python client. (Note that token-based
authentication is not yet supported on the Driverless AI R client.) This
section assumes that you have an understanding of OpenID Connect. The OpenID Connect Protocol
OpenID Connect follows a distinct protocol during the authentication
process:
1. A request is sent from the client (RP) to the OpenID provider (OP). 2. The OP authenticates the end user and obtains authorization. 3. The OP responds with an ID Token. (An Access Token is usually
    provided as well.) 4. The Relying Party (RP) can send a request with the Access Token to
    the UserInfo Endpoint. 5.

Refer to the OpenID Connect Basic Client Implementer's Guide for more
information: https://openid.net/specs/openid-connect-basic-1_0.html

Understanding the Well-Known Endpoint
In order to begin the process of configuring Driverless AI for
OpenID-based authentication, the end user must retrieve OpenID Connect
metadata about their authorization server by requesting information from
the WELL-KNOWN ENDPOINT. This information is subsequently used to
configure further interactions with the provider. The well-known endpoint is typically configured as follows:
    https://yourOpenIDProviderHostname/.well-known/openid-configuration

Configuration Options
OpenID Configuration Options
The following options in the config.toml file are used for enabling
OpenID-based authentication. Setting these options lets you log in to
the Driverless AI UI using OpenID. # The OpenID server URL.

Token-based authentication allows
clients to authenticate with the Driverless AI server by providing a
token with each request. This is targeted for (but not limited to) the
environments with OpenID Connect authentication. If these options are
not set, then clients are not able to authenticate with the server when
OpenID Connect is configured as the authentication method. # Sets token introspection URL for OpenID Connect authentication. (needs to be an absolute URL)
    auth_openid_token_introspection_url = ""
    # Enables option to use Bearer token for authentication with the RPC endpoint. api_token_introspection_enabled = false
    # Sets the method that is used to introspect the bearer token. # OAUTH2_TOKEN_INTROSPECTION: Uses  OAuth 2.0 Token Introspection (RPC 7662)
    # endpoint to introspect the bearer token. # This useful when 'openid' is used as the authentication method.

# 
    api_token_introspection_method = "OAUTH2_TOKEN_INTROSPECTION"
    # Sets the minimun of the scopes that the access token needs to have
    # in order to pass the introspection. Space separated./
    # This is passed to the introspection endpoint and also verified after response
    # for the servers that don't enforce scopes. # Keeping this empty turns any the verification off. # 
    api_token_oauth2_scopes = ""
    # Which field of the response returned by the token introspection endpoint should be used as a username. api_token_oauth2_username_field_name = "username"
    # Enables the option to initiate a PKCE flow from the UI in order to obtain tokens usable with Driverless clients
    oauth2_client_tokens_enabled = false
    # Sets up client id that will be used in the OAuth 2.0 Authorization Code Flow to obtain the tokens. Client needs to be public and be able to use PKCE with S256 code challenge.

oauth2_client_tokens_authorize_url = ""
    # Sets up the absolute url to the token endpoint. oauth2_client_tokens_token_url = ""
    # Sets up the absolute url to the token introspection endpoint.It's displayed in the UI so that clients can inspect the token expiration. oauth2_client_tokens_introspection_url = ""
    # Sets up the absolute to the redirect url where Driverless handles the redirect part of the Authorization Code Flow. this <Driverless base url>/oauth2/client_token
    oauth2_client_tokens_redirect_url = ""
    # Sets up the scope for the requested tokens. Space seprated list. oauth2_client_tokens_scope = "openid profile ai.h2o.storage"

Example 1: Enabling OpenID Connect
This example describes how to start Driverless AI in the Docker image
and with native installs after OpenID has been configured. Note that
this example does not enable tokens, so the Driverless AI Python client
will be incompatible with this installation. Docker Image Installs
1.

CLIENT CERTIFICATE AUTHENTICATION EXAMPLE

This section describes how to configure client certificate
authentication in Driverless AI. Client Certificate and SSL Configuration Options
The following options can be specified when configuring client
certificate authentication. SSL Configuration Options
Mutual TLS authentication (mTLS) must be enabled in order to enable
Client Certificate Authentication. Use the following configuration
options to configure mTLS. Refer to the mTLS Authentication topic for
more information on how to enable mTLS. -   ssl_client_verify_mode: Sets the client verification mode. Choose
    from the following verification modes:
-   ssl_ca_file: Specifies the path to the certification authority (CA)
    certificate file. This certificate will be used to verify the client
    certificate when client authentication is enabled. If this is not
    specified, clients are verified using the default system
    certificates.

Specifies the HTTPS
    settings path to the private key that Driverless AI uses to
    authenticate itself. -   ssl_client_crt_file: Required if
    ssl_client_verify_mode = "CERT_REQUIRED". Specifies the HTTPS
    settings path to the client certificate that Driverless AI will use
    to authenticate itself. Client Certificate Options
-   auth_tls_crl_file: The path to the certificate revocation list (CRL)
    file that is used to verify the client certificate. -   auth_tls_user_lookup: Specifies how a user's identity is obtained. Choose from the following:
      -   REGEXP_ONLY: Uses auth_tls_subject_field and
          auth_tls_field_parse_regexp to extract the username from the
          client certificate. -   LDAP_LOOKUP: Uses the LDAP server to obtain the username. (Refer to the ldap_authentication section for information
          about additional LDAP Authentication configuration options.)

-   auth_tls_ldap_port: Specifies the LDAP server port number. This is
    389 by default. -   auth_tls_ldap_use_ssl: Specifies whether to enable (True) or disable
    (False) SSL when connecting to the LDAP server. -   auth_tls_ldap_tls_file: Specifies the path to the SSL certificate. -   auth_tls_ldap_bind_dn: Specifies the complete DN of the LDAP bind
    user. -   auth_tls_ldap_bind_password: Specifies the password for the LDAP
    bind. -   auth_tls_subject_field: The subject field that is used as a source
    for a username or other values that provide further validation. -   auth_tls_field_parse_regexp: The regular expression that is used to
    parse the subject field in order to obtain the username or other
    values that provide further validation. -   auth_tls_ldap_search_base: Specifies the location in the Directory
    Information Tree (DIT) where the search will start.

This can be dynamically built
    by using the named capturing groups from auth_tls_field_parse_regexp
    for substitution:
          auth_tls_field_parse_regexp = "\w+ (?P<id>\d+)"
          auth_tls_ldap_search_filter = "(&(objectClass=person)(id={{id}}))
-   auth_tls_ldap_username_attribute: Specifies the LDAP record
    attribute that is used as a username. -   auth_tls_ldap_authorization_lookup_filter: (Optional) Specifies an
    additional search filter that is performed after the user is found. This is useful for checking whether a user is a member of a specific
    group in LDAP schemas where group membership is defined within group
    entries as opposed to individual user entries. (Refer to the
    lookup-filter-example section that follows to see an example of how
    this option can be used.) -   auth_tls_ldap_authorization_search_base: Specifies the base
    distinguished name (DN) to start the authorization lookup from.

auth_tls_ldap_authorization_lookup_filteroption to determine whether individual users are members of thechemistsgroup in an LDAP schema where group (organizational unit) membership is defined within group entries.

Each    variable must be prepended with D\ `RIVERLESS_AI <>`__. The following    example enables Client Certification authentication and usesLDAP_LOOKUPfor the TLS user lookup method.

KDB+ SETUP

Driverless AI lets you explore kdb+ data sources from within the
configuring Driverless AI to work with kdb+.

docker run
versionto check which version of Docker you are using. Description of Configuration Attributes ---------------------------------------  -kdb_user: (Optional) User name -kdb_password: (Optional) User's password -kdb_hostname: IP address or host of the KDB server -kdb_port: Port on which the kdb+ server is listening -kdb_app_jvm_args: (Optional) JVM args for kdb+ distributions (for    example,-Dlog4j.configuration). Separate each argument with    spaces. -kdb_app_classpath: (Optional) The kdb+ classpath (or other if the    jar file is stored elsewhere). -enabled_file_systems: The file systems you want to enable. This    must be configured in order for data connectors to function properly.

The    only required flags are the hostname and the port. .. code:: bash        nvidia-docker run \         --pid=host \         --init \         --rm \         --shm-size=2g --cap-add=SYS_NICE --ulimit nofile=131071:131071 --ulimit nproc=16384:16384 \         --add-host name.node:172.16.2.186 \         -e DRIVERLESS_AI_ENABLED_FILE_SYSTEMS="file,kdb" \         -e DRIVERLESS_AI_KDB_HOSTNAME="<ip_or_host_of_kdb_server>" \         -e DRIVERLESS_AI_KDB_PORT="<kdb_server_port>" \         -p 12345:12345 \         -v /tmp/dtmp/:/tmp \         -v /tmp/dlog/:/log \         -v /tmp/dlicense/:/license \         -v /tmp/ddata/:/data \         -u $(id -u):$(id -g) \         h2oai/dai-ubi8-x86_64:|tag|  ..     .. container:: group-tab        Docker Image with the config.toml        This example shows how to configure kdb+ options in the       config.toml file, and then specify that file when starting       Driverless AI in Docker.

1. Configure the Driverless AI config.toml file. Set the following          configuration options. ..           -enabled_file_systems
= "file, upload, kdb"-kdb_hostname =
<ip_or_host_of_kdb_server>"-kdb_port =
"<kdb_server_port>"2. Mount the config.toml file into the Docker container.

The only required flags are the hostname and the port. 1. Export the Driverless AI config.toml file or add it to          ~/.bashrc. For example:        ..           ::              # DEB and RPM             export DRIVERLESS_AI_CONFIG_FILE="/etc/dai/config.toml"              # TAR SH             export DRIVERLESS_AI_CONFIG_FILE="/path/to/your/unpacked/dai/directory/config.toml"         2. Specify the following configuration options in the config.toml          file.

(jdbc_app_configs)             # hive: Hive Connector, remember to configure Hive below. (hive_app_configs)             # recipe_url: load custom recipe from URL             # recipe_file: load custom recipe from local file system             enabled_file_systems = "file, kdb"              # KDB Connector credentials             kdb_hostname = <ip_or_host_of_kdb_server>"             kdb_port = "<kdb_server_port>"        3. Save the changes when you are done, then stop/restart          Driverless AI. Example 2: Enable kdb+ with Authentication ------------------------------------------  .. container:: tabs     .. group-tab:: Docker Image Installs     This example provides users credentials for accessing a kdb+ server    from Driverless AI.

Note that this example enables kdb+ with       no authentication. 1. Configure the Driverless AI config.toml file. Set the following          configuration options. ..           -enabled_file_systems
= "file, upload, kdb"-kdb_user = "<username>"-kdb_password =
"<password>"-kdb_hostname = <ip_or_host_of_kdb_server>"-kdb_port =
"<kdb_server_port>"-kdb_app_classpath = ""-kdb_app_jvm_args =
""2. Mount the config.toml file into the Docker container.

(jdbc_app_configs)             # hive: Hive Connector, remember to configure Hive below. (hive_app_configs)             # recipe_url: load custom recipe from URL             # recipe_file: load custom recipe from local file system             enabled_file_systems = "file, kdb"              # kdb+ Connector credentials             kdb_user = "<username>"             kdb_password = "<password>"             kdb_hostname = <ip_or_host_of_kdb_server>"             kdb_port = "<kdb_server_port>"             kdb_app_classpath = ""             kdb_app_jvm_args = ""        3. Save the changes when you are done, then stop/restart          Driverless AI. Adding Datasets Using kdb+ --------------------------  After the kdb+ connector is enabled, you can add datasets by selecting **kdb+** from the **Add Dataset (or Drag and Drop)** drop-down menu. .. figure:: ../images/add_dataset_dropdown.png    :alt:     :width: 237px    :height: 338px  Specify the following information to add your dataset.

DATASET JOIN WIZARD

The Driverless AI Dataset Join Wizard makes it simple for you to join
two datasets together. This wizard performs a left (outer) join. Note
that the join key column name(s) must match between both datasets. To
rename columns, or to prepare datasets more generally, go to Dataset
Details and select MODIFY BY RECIPE -> LIVE CODE, or use data recipes. If a model is trained on the resulting dataset, make sure to also
perform the same join on testing or production data. To access the Dataset Join Wizard, navigate to the DATASETS page and
click on the name of the dataset you want to join with another dataset. A list of dataset-specific options is displayed. Select JOIN WIZARD to
open the wizard. []
When using the Join Datasets wizard, you can either specify a dataset to
join, or first specify the join key column(s) to use. NOTES:
-   This feature is currently in an experimental state.

DRIVERLESS AI LOGS

Driverless AI provides several logs that can be viewed and/or retrieved
when performing different tasks. All content in the logs are labeled
with INFO, DATA, WARNING and ERROR tags. Driverless AI Modeling and MLI
experiments also provide access to anonymized logs that do not contain
contents from the DATA tag. -   logs-available
-   logs-sending
-   Obtaining System Log Files <logs-system>

Available Log Files
The following is a list of available Driverless AI log files. -   dai_log
  -   exp_log
  -   mli_log
  -   auto_viz_log
  -   h2oai_server_log
  -   audit_log
dai.log
DAI.LOG are part of Driverless AI System Logs <logs-system>. They are
generated as part of stderr/stdout and are useful for debugging or
detailed support in case of issues. If needed, the verbosity or logging
level of this log file can be toggled using config.toml settings. ADMIN ACCESS to Driverless AI installation location is required to
obtain these logs.

Experiment Log
This log store information about an experiment's parameters, model runs,
performance and more. It helps with understanding the run details and
debugging experiment related issues. The log file naming convention is
H2OAI_EXPERIMENT_{EXPERIMENT_ID}.LOG and the content is labeled with
INFO, DATA, WARNING and ERROR tags. USERS can download these log directly from the experiment page of the
Driverless AI GUI. For an EXPERIMENT IN PROGRESS, logs can be accessed
from under the _Log tab_ to the right. For COMPLETED EXPERIMENTS, the
logs reside with the SUMMARY ZIP FILE. []
The zip also contains an ANONYMIZED VERSION OF EXPERIMENT LOGS that does
not report any information relating to the data used in the experiment
(i.e no DATA label), such as column names and individual data points. And a DETAILS folder that comprises of error stack traces that may help
with debugging.

MLI SURROGATE MODEL RUN LOGS can be downloaded from the Action button on
the MLI GUI page. This folder naming convention is
MLI_EXPERIMENT_{EXPERIMENT_ID}.LOG. It contains three files, the
stdout/stderr log for full MLI process run, an ANONYMIZED copy (i.e no
DATA label) of the same log file and surrogate model run logs. []
The EXPLAINER OR RECIPE LOGS are accessible from the task run button. []
MLI uses H2O_3 (Java backend) to build surrogate models. ADMINS can
access the H2O_3 SERVER LOGS using System Logs <logs-system> commands in
case of issues with starting the MLI server. The /tmp folder of DAI
contains H2O_MLI.LOG, that keeps track of rolling mli logs and are also
admin accessible. Auto Visualization Logs
This log store run information for automatic data visualization in
Driverless AI. USERS can obtain them from the Autoviz page of DAI GUI. []
ADMINS can access the VIZ-SERVER LOGS using System Logs <logs-system>
commands in case of issues with starting of Viz server.

h2oai_server Log
These logs register ALL ISSUES RELATING TO DATASETS like Adding Datasets
or viewing Dataset Details or Auto Visualization of datasets. They are
stored under the Driverless AI ./TMP folder. An ANONYMIZED copy (i.e no
DATA label) of this log file is also available in the same folder. Accessing h2oai_server log requires ADMIN access to Driverless AI. Audit logs
Audit logs register ALL USER INTERACTIONS WITH THE DRIVERLESS AI SYSTEM
like login/logout, downloads/uploads, experiment creation/deletion etc. ADMINS can access them from /tmp folder of Driverless AI. Sending logs to support@H2O.ai
This section describes what logs to send in the event of failures when
running Driverless AI. All content in the logs are labeled with INFO,
DATA, WARNING and ERROR tags. Driverless AI Modeling and MLI experiments
also provides access to anonymized logs that do not contain contents
from the DATA tag.

-   DATASET FAILURES: A simple error stack trace is displayed on the GUI
    in case of datasets failures like ADDING DATASETS or viewing DATASET
    DETAILS and detailed logs are registered as
    h2oai_server logs <h2oai_server_log> that requires admin access. -   AUTO VIZ FAILURES: auto_viz_log on datasets can be accessed from the
    GUI. A full detailed stacktrace is also available in the
    h2oai_server.log <h2oai_server_log> file in ./TMP folder of DAI that
    requires admin access. -   EXPERIMENT FAILURES: User needs to send the
    experiment logs <exp_log>. In some cases, for in depth analysis,
    support@h2o.ai may request dai.logs <dai_log> that requires admin
    access to retrieve. -   MLI FAILURES: See MLI Logs <mli_log> for details. -   CUSTOM RECIPES FAILURES: If a Custom Recipe is producing errors, the
    entire zip file obtained by clicking on the DOWNLOAD SUMMARY & LOGS
    button on the experiment <exp_log> page, can be sent for
    troubleshooting.

System logs
System logs include useful information about Driverless AI. Driverless
AI solution needs following set of services to work-
-   DRIVERLESS AI SERVER: This is a python code, that internally starts
    a local worker to start a web server for UI pages (DAI GUI) and runs
    the actual experiment work. -   H2O3 SERVER: This is needed for MLI and recipes that use h2o3. -   PROCSY: This handles the communication between the DAI server
    (python code) and other binaries or java jar files, like data
    connectors or the vis-sever. -   VIS-SERVER: This is needed for Auto visualization of Datasets, DAI
    sends a request to procsy, which in turn will query the vis-server
    to make the computations necessary for autoviz. -   REDIS-SERVER: It is used as a communication bus between the backend
    (DAI) server and the local worker or remote workers (in case of DAI
    multinode set up).

AZURE BLOB STORE SETUP

Driverless AI lets you explore Azure Blob Store data sources from within
the Driverless AI application.

docker run
versionto check which version of Docker you are using. Supported Data Sources Using the Azure Blob Store Connector -----------------------------------------------------------  The following data sources can be used with the Azure Blob Store connector. -  :ref:`Azure Blob Storage (general purpose v1)<example1>` -  Blob Storage -  :ref:`Azure Files (File Storage)<example2>` -  :ref:`Azure Data Lake Storage Gen 2 (Storage V2)<example4>`  The following data sources can be used with the Azure Blob Store connector when also using the HDFS connector. -  :ref:`Azure Data Lake Gen 1 (HDFS connector required)<example3>` -  :ref:`Azure Data Lake Gen 2 (HDFS connector optional)<example4>`  Description of Configuration Attributes ---------------------------------------  The following configuration attributes are specific to enabling Azure Blob Storage.

This should be the dns prefix created when the account was    created (for example, "mystorage"). -azure_blob_account_key: Specify the account key that maps to your    account name. -azure_connection_string: Optionally specify a new connection    string. With this option, you can include an override for a host,    port, and/or account name. For example,     .. code:: bash        azure_connection_string = "DefaultEndpointsProtocol=http;AccountName=<account_name>;AccountKey=<account_key>;BlobEndpoint=http://<host>:<port>/<account_name>;"  -azure_blob_init_path: Specifies the starting Azure Blob store    path displayed in the UI of the Azure Blob store browser. -enabled_file_systems: The file systems you want to enable. This    must be configured in order for data connectors to function properly. The following additional configuration attributes can be used for enabling an HDFS Connector to connect to Azure Data Lake Gen 1 (and optionally with Azure Data Lake Gen 2).

This    folder can contain multiple config files. -hdfs_app_classpath: The HDFS classpath. -hdfs_app_supported_schemes: Supported schemas list is used as an    initial check to ensure valid input to connector. .. _example1:  Example 1: Enabling the Azure Blob Store Data Connector -------------------------------------------------------  .. container:: tabs     .. group-tab:: Docker Image Installs     This example enables the Azure Blob Store data connector by    specifying environment variables when starting the Driverless AI    Docker image. This lets users reference data stored on your Azure    storage account using the account name, for example:https://mystorage.blob.core.windows.net.

1. Configure the Driverless AI config.toml file. Set the following          configuration options:        ..           -enabled_file_systems
= "file, upload, azrbs"-azure_blob_account_name =
"mystorage"-azure_blob_account_key =
"<account_key>"2. Mount the config.toml file into the Docker container.

This lets users reference data stored on your       Azure storage account using the account name, for example:https://mystorage.blob.core.windows.net. 1. Export the Driverless AI config.toml file or add it to          ~/.bashrc. For example:        ..           ::              # DEB and RPM             export DRIVERLESS_AI_CONFIG_FILE="/etc/dai/config.toml"              # TAR SH             export DRIVERLESS_AI_CONFIG_FILE="/path/to/your/unpacked/dai/directory/config.toml"         2. Specify the following configuration options in the config.toml          file.

(jdbc_app_configs)             # hive: Hive Connector, remember to configure Hive below. (hive_app_configs)             # recipe_url: load custom recipe from URL             # recipe_file: load custom recipe from local file system             enabled_file_systems = "file, azrbs"              # Azure Blob Store Connector credentials             azure_blob_account_name = "mystorage"             azure_blob_account_key = "<account_key>"        3. Save the changes when you are done, then stop/restart          Driverless AI. .. _example2:  Example 2: Mount Azure File Shares to the Local File System -----------------------------------------------------------  Supported Data Sources Using the Local File System ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  -  Azure Files (File Storage)  Mounting Azure File Shares ~~~~~~~~~~~~~~~~~~~~~~~~~~  Azure file shares can be mounted into the Local File system of Driverless AI.

.. _example3:  Example 3: Enable HDFS Connector to Connect to Azure Data Lake Gen 1 --------------------------------------------------------------------  This example enables the HDFS Connector to connect to Azure Data Lake Gen1. This lets users reference data stored on your Azure Data Lake using the adl uri, for example:adl://myadl.azuredatalakestore.net. .. container:: tabs     .. group-tab:: Docker Image with the config.toml     1. Create an Azure AD web application for service-to-service       authentication:       https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-service-to-service-authenticate-using-active-directory    2.

Take note of the Hadoop Classpath and add theazure-datalake-store.jarfile. This file can found on any       Hadoop version in:$HADOOP_HOME/share/hadoop/tools/lib/*. ..        .. code:: bash           echo "$HADOOP_CLASSPATH:$HADOOP_HOME/share/hadoop/tools/lib/*"     4. Configure the Driverless AI config.toml file. Set the following       configuration options:     ..        .. code:: bash           enabled_file_systems = "upload, file, hdfs, azrbs, recipe_file, recipe_url"          hdfs_config_path = "/path/to/hadoop/conf"          hdfs_app_classpath = "/hadoop/classpath/"          hdfs_app_supported_schemes = "['adl://']"     5. Mount the config.toml file into the Docker container.

Create an Azure AD web application for service-to-service          authentication. https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-service-to-service-authenticate-using-active-directory       2.

abfs[s]://file_system@account_name.dfs.core.windows.net/<path>/<path>/<file_name>. .. container:: tabs     .. group-tab:: Docker Image with the config.toml     1. Create an Azure Service Principal:       https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal    2. Grant permissions to the Service Principal created on step 1 to       access blobs:       https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad    3.

Take note of the Hadoop Classpath and add the required jar files. These files can found on any Hadoop version 3.2 or higher at:$HADOOP_HOME/share/hadoop/tools/lib/*..        .. code:: bash           echo "$HADOOP_CLASSPATH:$HADOOP_HOME/share/hadoop/tools/lib/*"        **Note**: ABFS is only supported for Hadoop version 3.2 or higher. 5. Configure the Driverless AI config.toml file. Set the following       configuration options:     ..        .. code:: bash           enabled_file_systems = "upload, file, hdfs, azrbs, recipe_file, recipe_url"          hdfs_config_path = "/path/to/hadoop/conf"          hdfs_app_classpath = "/hadoop/classpath/"          hdfs_app_supported_schemes = "['abfs://']"     6. Mount the config.toml file into the Docker container.

Create an Azure Service Principal. https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal       2. Grant permissions to the Service Principal created on step 1 to          access blobs:          https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad       3.

Take note of the Hadoop Classpath and add the required jar          files. These files can found on any hadoop version 3.2 or          higher at:$HADOOP_HOME/share/hadoop/tools/lib/*..           .. code:: bash              echo "$HADOOP_CLASSPATH:$HADOOP_HOME/share/hadoop/tools/lib/*"           **Note**: ABFS is only supported for hadoop version 3.2 or          higher        5. Configure the Driverless AI config.toml file. Set the following          configuration options:        ..           .. code:: bash              enabled_file_systems = "upload, file, hdfs, azrbs, recipe_file, recipe_url"             hdfs_config_path = "/path/to/hadoop/conf"             hdfs_app_classpath = "/hadoop/classpath/"             hdfs_app_supported_schemes = "['abfs://']"        6. Save the changes when you are done, then stop/restart          Driverless AI.

You can enable support for the SAS token by setting the following variables in theconfig.tomlfile:  1.enable_artifacts_upload=true2.artifacts_store="azure"3.artifacts_azure_sas_token="token"``
For instructions on exporting artifacts, see export_artifacts. FAQ

CAN I CONNECT TO MY STORAGE ACCOUNT USING PRIVATE ENDPOINTS? Yes. Driverless AI can use private endpoints if Driverless AI is
  located in the allowed VNET. DOES DRIVERLESS AI SUPPORT SECURE TRANSFER? Yes. The Azure Blob Store Connector make all connections over HTTPS. DOES DRIVERLESS AI SUPPORT HIERARCHICAL NAMESPACES? Yes. CAN I USE AZURE MANAGED IDENTITIES (MSI) TO ACCESS THE DATALAKE? Yes. If Driverless AI is running on an Azure VM with managed
  identities.

EXPERIMENT SETUP COMPARISON

This page describes the experiment comparison feature of Driverless AI. Overview
Driverless AI lets you review the differences in how multiple
experiments have been set up, including the selected experiment
parameters and Expert Settings <understanding-expert-settings> for each
experiment. This includes information like the selected target column,
the datasets used for the experiment, and the selected values for the
Accuracy, Time, and Interpretability knobs. The experiment setup comparison page lets you view the following
information:
-   The names of the experiments being compared. -   Differences in the selected experiment parameters between each
    experiment being compared. -   Differences in the selected Expert Settings between each experiment
    being compared. -   Full lists of the selected experiment parameters and Expert Settings
    for each experiment being compared. To view these lists, click the
    FULL SETTINGS tab.

DRIVERLESS AI MOJO SCORING PIPELINE - C++ RUNTIME WITH PYTHON (SUPPORTS SHAPLEY) AND R WRAPPERS

The C++ Scoring Pipeline is provided as R and Python packages for the
protobuf-based MOJO2 protocol. Use your preferred method once the MOJO
Scoring Pipeline has been built. NOTES:
  -   These scoring pipelines are currently not available for RuleFit
      models. -   Unlike the Java Runtime, TensorFlow/Bert _are supported_ by C++
      Runtime MOJO. -   You can have Driverless AI attempt to reduce the size of the MOJO
      scoring pipeline when the experiment is being built by enabling
      the Reduce MOJO Size <reduce_mojo_size> expert setting also
      see <mojo-size>. -   Shapley contributions come with the downloaded experiment MOJO
      scoring pipeline. See cpp_scoring_shapley for scoring example.

libopenblas-dev, run the following command:  ::     sudo apt install libopenblas-dev  .. _cpp-mojo-downloads:  Downloads ---------  This section contains download links for the C++ MOJO runtime and its Python and R wrappers. **Python:**  -  :mojo-runtime38:C++ MOJO runtime (Python 3.8)    <https://s3.amazonaws.com/artifacts.h2o.ai/releases/ai/h2o/daimojo/> -  :mojo-runtime37:C++ MOJO runtime (Python 3.7)    <https://s3.amazonaws.com/artifacts.h2o.ai/releases/ai/h2o/daimojo/> -  :mojo-runtime36:C++ MOJO runtime (Python 3.6)    <https://s3.amazonaws.com/artifacts.h2o.ai/releases/ai/h2o/daimojo/>  **R**:  -  :daimojo-r:`C++ MOJO runtime <https://s3.amazonaws.com/artifacts.h2o.ai/releases/ai/h2o/daimojo-r/>`  .. note::     The Python and R packages can also be downloaded from within the    Driverless AI application by clicking **Resources**, and then    clicking **MOJO Py Runtime** or **MOJO R Runtime** from the drop-down    menu.

For download links, see          :ref:`cpp-mojo-downloads`.) -datatable``. Run the following to install:
      # Install on Linux PPC or Linux x86
      pip install datatable
  -   Non-binary version of protobuf:
          pip install --no-binary=protobuf protobuf
  -   Python MOJO runtime.

--with-prediction-interval.. code:: bash     java -Xmx5g -Dai.h2o.mojos.runtime.license.file=license.file -jar mojo2-runtime.jar --with-prediction-interval pipeline.mojo example.csv  .. _cpp_scoring_shapley:  C++ MOJO runtime Shapley values support ---------------------------------------  The C++ MOJO runtime and its Python wrapper support Shapley contributions for transformed features and original features.

THE INTERPRETED MODELS PAGE

Click the MLI link in the upper-right corner of the UI to view a list of
interpreted models.

You can sort this page by Name, Target, Model, Dataset, N-Folds, Feature
Set, Cluster Col, LIME Method, Status, or ETA/Runtime. You can also use
the search bar to locate a specific interpreted model. To specify which
columns are visible on this page, click the top right-most column, then
select VISIBLE COLUMNS.

Click the right-most column of an interpreted model to view an
additional menu. This menu allows you to open, rename, or delete the

NOTE: Driverless AI version 1.9 features a redesigned MLI page for
interpreted models. To view the legacy version of an interpreted model's
MLI page, select OPEN LEGACY from the menu.

Click on an interpreted model to view the MLI page for that
interpretation. The MLI page that displays will vary depending on
whether the experiment was a regular experiment or a time series

This section describes the settings that are available when running an
experiment. Display Name
Optional: Specify a display name for the new experiment. There are no
character or length restrictions for naming. If this field is left
blank, Driverless AI will automatically generate a name for the
experiment. Dropped Columns
Dropped columns are columns that you do not want to be used as
predictors in the experiment. Note that Driverless AI will automatically
drop ID columns and columns that contain a significant number of unique
values (above max_relative_cardinality in the config.toml file or MAX. ALLOWED FRACTION OF UNIQUES FOR INTEGER AND CATEGORICAL COLS in Expert
settings). Validation Dataset
The validation dataset is used for tuning the modeling pipeline. If
provided, the entire training data will be used for training, and
validation of the modeling pipeline is performed with only this
validation dataset.

experiments and
multiple rolling window validation splits for time series experiments. For this reason it is not generally recommended to include a validation
dataset as you are then validating on only a single dataset. Note that
time series experiments cannot be used with a validation dataset:
including a validation dataset will disable the ability to select a time
column and vice versa. This dataset must have the same number of columns (and column types) as
the training dataset. Also note that if provided, the validation set is
not sampled down, so it can lead to large memory usage, even if
accuracy=1 (which reduces the train size). Test Dataset
The test dataset is used for testing the modeling pipeline and creating
test predictions. The test set is never used during training of the
modeling pipeline. (Results are the same whether a test set is provided
or not.) If a test dataset is provided, then test set predictions will
be available at the end of the experiment.

sample or
row weight), if applicable. This column must be numeric with values >=
0. Rows with higher weights have higher importance. The weight affects
model training through a weighted loss function and affects model
scoring through weighted metrics. The weight column is not used when
making test set predictions, but a weight column (if specified) is used
when computing the test score. NOTE: The weight column is not used as a feature in modeling. Fold Column
Optional: Rows with the same value in the fold column represent groups
that should be kept together in the training, validation, or
cross-validation datasets. This can prevent data leakage and improve
generalization for data that is naturally grouped and not i.i.d. (identically and independently distributed). This column must be an
integer or categorical variable, and it cannot be specified if a
validation set is used or if a Time Column is specified. By default, Driverless AI assumes that the dataset is i.i.d.

The fold column is used to create the training and validation datasets
so that all rows with the same Fold value will be in the same dataset. This can prevent data leakage and improve generalization. For example,
when viewing data for a pneumonia dataset, person_id would be a good
Fold Column. This is because the data may include multiple diagnostic
snapshots per person, and we want to ensure that the same person’s
characteristics show up only in either the training or validation
frames, but not in both to avoid data leakage. This column must be an integer or categorical variable and cannot be
specified if a validation set is used or if a Time Column is specified. NOTE: The fold column is not used as a feature in modeling. Time Column
Optional: Specify a column that provides a time order (time stamps for
observations), if applicable.

The values in this column must be a datetime format understood by
pandas.to_datetime(), like "2017-11-29 00:30:35" or "2017/11/29", or
integer values. If [AUTO] is selected, all string columns are tested for
potential date/datetime content and considered as potential time
columns. If a time column is found, feature engineering and model
validation will respect the causality of time. If [OFF] is selected, no
time order is used for modeling and data may be shuffled randomly (any
potential temporal causality will be ignored). When your data has a date column, then in most cases, specifying [AUTO]
for the Time Column will be sufficient. However, if you select a
specific date column, then Driverless AI will provide you with an
additional side menu. From this side menu, you can specify Time Group
columns or specify [Auto] to let Driverless AI determine the best time
group columns.

Refer to time-series-in-dai for more information about time series
experiments in Driverless AI and to see a time series example. []
NOTES:
-   Engineered features will be used for MLI when a time series
    experiment is built. This is because munged time series features are
    more useful features for MLI compared to raw time series features. -   A Time Column cannot be specified if a Fold Column is specified. This is because both fold and time columns are only used to split
    training datasets into training/validation, so once you split by
    time, you cannot also split with the fold column. If a Time Column
    is specified, then the time group columns play the role of the fold
    column for time series. -   A Time Column cannot be specified if a validation dataset is used. -   A column that is specified as being unavailable at prediction time
    will only have lag-related features created for (or with) it.

Accuracy, Time, and Interpretability Knobs
The experiment preview describes what the Accuracy, Time, and
Interpretability settings mean for your specific experiment. This
preview automatically updates when any of the experiment's settings
change (including the knobs). -   The Accuracy knob stands for relative accuracy: Higher values
      should lead to higher confidence in model performance. Usually
      achieved through the use of larger data (less sampling), more
      modeling effort (more tuning, higher accuracy settings), more
      statistical calculations (cross-validation, bootstrapping). Doesn't always mean that the final model is better, but generally
      means that the final estimate is more accurate. If in doubt, trust
      the results of the experiment with higher accuracy settings. -   The Time knob stands for relative time tolerance: Higher values
      generally lead to longer run times.

Larger values mean higher
      chance of getting a better model. If it takes too long, just click
      on 'Finish' button and it will finish the experiment as if
      convergence was achieved. -   The Interpretability knob stands for relative interpretability:
      Higher values favor more interpretable models (e.g. linear models,
      decision trees, single models) with less complex feature
      engineering (fewer features, simple features). Lower values lead
      to more complex models (e.g. neural networks, GBMs, ensembles) and
      more complex feature pipelines (more features, higher-order
      interaction features). Note
- You can manually select individual features to force into an
experiment—regardless of Accuracy, Time, and Interpretability
levels—with the Features to Force In <cols_to_force_in> expert setting.

[_Accuracy, Time, and Interpretability Knobs_]
[_Experiment Preview_]
Accuracy
As accuracy increases, Driverless AI gradually adjusts the method for
performing the evolution and ensemble. At low accuracy, Driverless AI
varies features and models, but they all compete evenly against each
other. At higher accuracy, each independent main model will evolve
independently and be part of the final ensemble as an ensemble over
different main models. At higher accuracies, Driverless AI will
evolve+ensemble feature types like Target Encoding on and off that
evolve independently. Finally, at highest accuracies, Driverless AI
performs both model and feature tracking and ensembles all those
variations. Changing this value affects the feature evolution and final pipeline. NOTE: A check for a shift in the distribution between train and test is
done for accuracy >= 5. TRAINING DATA SIZE: Displays the number of rows and columns in the
training data.

If a test set is provided without a validation set, then
Driverless AI will perform a 1/3 validation split during the experiment. If a validation set is provided, then the experiment will perform
external validation. FINAL PIPELINE: This represents the number of models and the validation
method used in the final pipeline. For ensemble modeling, information
about how models are combined is also shown here. Time
This specifies the relative time for completing the experiment (that is,
higher settings take longer). FEATURE BRAIN LEVEL: Displays the feature brain level for the
experiment. For more information, see feature_brain1. FEATURE EVOLUTION: Displays the number of individuals and maximum number
of iterations that will be run in this experiment. EARLY STOPPING: Early stopping will take place if the experiment doesn't
improve the score for the specified amount of iterations. Interpretability
Specify the relative interpretability for this experiment.

Changing the interpretability level
affects the feature pre-pruning strategy, monotonicity constraints, and
the feature engineering search space. FEATURE PRE-PRUNING STRATEGY: This represents the feature selection
strategy (to prune-away features that do not clearly give improvement to
model score). Strategy = “Permutation Importance FS” if interpretability
>= 6; otherwise strategy is None. MONOTONICITY CONSTRAINTS: If MONOTONICITY CONSTRAINTS are enabled, the
model will satisfy knowledge about monotonicity in the data and monotone
relationships between the predictors and the target variable. For
example, in house price prediction, the house price should increase with
lot size and number of rooms, and should decrease with crime rate in the
area. If enabled, Driverless AI will automatically determine if
monotonicity is present and enforce it in its modeling pipelines. Depending on the correlation, Driverless AI will assign positive,
negative, or no monotonicity constraints.

All other predictors will
not have monotonicity enforced. For more information, see mc. NOTE: Monotonicity constraints are used in XGBoost GBM, XGBoost Dart,
  LightGBM, and Decision Tree models. FEATURE ENGINEERING SEARCH SPACE: This represents the transformers that
will be used during the experiment. [...] Models to Train
For the listed models:
  MODEL AND FEATURE TUNING: Represents the number of validation splits
  multiplied by the tuning population size. FEATURE EVOLUTION: Represents the number of models trained in order to
  evaluate engineered features. FINAL PIPELINE: Represents the number of final models. PER-MODEL HYPERPARAMETER OPTIMIZATION TRIALS:
    -   EVOLUTION - Represents the number of trials performed for
        hyperparameter optimization for tuning models. -   FINAL - Represents the number of trials performed for
        hyperparameter optimization for final models.

Though not recommended, you can override this setting
by clicking this button. Reproducible
The REPRODUCIBLE toggle lets you build an experiment with a random seed
and get reproducible results. If this is disabled (default), then
results vary between runs, which can give a good sense of variance among
experiment results. When enabling this option, keep the following notes in mind:
-   Experiments are only reproducible when run on the same hardware
    (that is, using the same number and type of GPUs/CPUs and the same
    architecture). For example, you will not get the same results if you
    try an experiment on a GPU machine, and then attempt to reproduce
    the results on a CPU-only machine or on a machine with a different
    number and type of GPUs. -   This option should be used with the reproducibility_level expert
    setting option, which ensures different degrees of reproducibility
    based on the OS and environment architecture.

-   Experiments run using TensorFlow with multiple cores cannot be
    reproduced. -   LightGBM is more reproducible with 64-bit floats, and Driverless AI
    will switch to 64-bit floats for LightGBM. (Refer to
    https://lightgbm.readthedocs.io/en/latest/Parameters.html#gpu_use_dp
    for more information.) -   Enabling this option automatically disables all of the Feature Brain
    expert settings options; specifically:
    -   feature_brain1
    -   feature_brain2
    -   feature_brain3
    -   feature_brain4
    -   feature_brain5
For more information on reproducibility in Driverless AI, see
reproducibility. Enable GPUs
To enable or disable GPUs, click the GPUS ENABLED button. Note that even
in cases where a GPU is available, the system may still recommend using
CPUs only. In these cases, the toggle is automatically switched to OFF. When the GPUs Enabled button is switched manually, the system uses the
selected value from that moment on.

ENABLE_TENSORFLOW_IMAGE``

ENABLE IMAGE TRANSFORMER FOR PROCESSING OF IMAGE DATA

Specify whether to use pretrained deep learning models for processing of
image data as part of the feature engineering pipeline. When this is
enabled, a column of Uniform Resource Identifiers (URIs) to images is
converted to a numeric representation using ImageNet-pretrained deep
learning models. This is enabled by default.

tensorflow_image_pretrained_models--------------------------------------  .. container:: dropdown     **Supported ImageNet Pretrained Architectures for Image Transformer**     Specify the supported    `ImageNet <https://imagenet.stanford.edu/about.php>`__ pretrained    architectures for image transformer. Select from the following:     -  densenet121    -  efficientnetb0    -  efficientnetb2    -  inception_v3    -  mobilenetv2    -  resnet34    -  resnet50    -  seresnet50    -  seresnext50    -  xception (Selected by default)     **Notes**:     -  If an internet connection is available, non-default models are       downloaded automatically. If an internet connection is not       available, non-default models must be downloaded from       http://s3.amazonaws.com/artifacts.h2o.ai/releases/ai/h2o/pretrained/dai_image_models_1_10.zip       and extracted intotensorflow_image_pretrained_models_dir``.

TENSORFLOW_IMAGE_VECTORIZATION_OUTPUT_DIMENSION``

DIMENSIONALITY OF FEATURE SPACE CREATED BY IMAGE TRANSFORMER

Specify the dimensionality of the feature (embedding) space created by
Image Transformer. Select from the following:

-   10
-   25
-   50
-   100 (Default)
-   200
-   300

NOTE: Multiple transformers can be activated at the same time to allow
the selection of multiple options.

TENSORFLOW_IMAGE_FINE_TUNE``

ENABLE FINE-TUNING OF THE PRETRAINED MODELS USED FOR THE IMAGE
TRANSFORMER

Specify whether to enable fine-tuning of the ImageNet pretrained models
used for the Image Transformer. This is disabled by default.

TENSORFLOW_IMAGE_FINE_TUNING_NUM_EPOCHS``

NUMBER OF EPOCHS FOR FINE-TUNING USED FOR THE IMAGE TRANSFORMER

Specify the number of epochs for fine-tuning ImageNet pretrained models
used for the Image Transformer. This value defaults to 2.

TENSORFLOW_IMAGE_AUGMENTATIONS``

LIST OF AUGMENTATIONS FOR FINE-TUNING USED FOR THE IMAGE TRANSFORMER

Specify the list of possible image augmentations to apply while
fine-tuning the ImageNet pretrained models used for the Image
Transformer. Select from the following:

-   Blur
-   CLAHE
-   Downscale
-   GaussNoise
-   GridDropout
-   HorizontalFlip (Default)
-   HueSaturationValue
-   ImageCompression
-   OpticalDistortion
-   RandomBrightnessContrast
-   RandomRotate90
-   ShiftScaleRotate
-   VerticalFlip

NOTE: For more information on individual augmentations, see
https://albumentations.ai/docs/.

TENSORFLOW_IMAGE_BATCH_SIZE``

BATCH SIZE FOR THE IMAGE TRANSFORMER

Specify the batch size for the Image Transformer. By default, the batch
size is set to -1 (selected automatically).

NOTE: Larger architectures and batch sizes use more memory.

IMAGE_DOWNLOAD_TIMEOUT``

IMAGE DOWNLOAD TIMEOUT IN SECONDS

When providing images through URLs, specify the maximum number of
seconds to wait for an image to download. This value defaults to 60 sec.

STRING_COL_AS_IMAGE_MAX_MISSING_FRACTION``

MAXIMUM ALLOWED FRACTION OF MISSING VALUES FOR IMAGE COLUMN

Specify the maximum allowed fraction of missing elements in a string
column for it to be considered as a potential image path. This value

string_col_as_image_min_valid_types_fraction------------------------------------------------  .. container:: dropdown     **Minimum Fraction of Images That Need to Be of Valid Types for Image    Column to Be Used**     Specify the fraction of unique image URIs that need to have valid    endings (as defined bystring_col_as_image_valid_types``) for a

    string column to be considered as image data. This value defaults to
    0.8.

TENSORFLOW_IMAGE_USE_GPU``

ENABLE GPU(S) FOR FASTER TRANSFORMATIONS WITH THE IMAGE TRANSFORMER

Specify whether to use any available GPUs to transform images into
embeddings with the Image Transformer. Enabling this setting can lead to
significantly faster transformation speeds. This is enabled by default.

NOTE: This setting only applies when scoring inside Driverless AI or
with Py Scoring.

ADDING LICENSES FOR THE FIRST TIME

Specifying a License File for the Driverless AI Application
A license file to run Driverless AI can be added in one of three ways
when starting Driverless AI. -   Specifying the license.sig file during launch in native installs
-   Using the DRIVERLESS_AI_LICENSE_FILE and DRIVERLESS_AI_LICENSE_KEY
    environment variables when starting the Driverless AI Docker image
-   Uploading your license in the Web UI
Specifying the license.sig File During Launch
By default, Driverless AI looks for a license key in
/OPT/H2OAI/DAI/HOME/.DRIVERLESSAI/LICENSE.SIG. If you are installing
Driverless AI programmatically, you can copy a license key file to that
location. If no license key is found, the application will prompt you to
add one via the Web UI. Specifying Environment Variables
You can use the DRIVERLESS_AI_LICENSE_FILE or DRIVERLESS_AI_LICENSE_KEY
environment variable when starting the Driverless AI Docker image.

[]
Click the ENTER LICENSE button, and then paste the entire license into
the provided LICENSE KEY entry field. Click SAVE when you are done. Upon
successful completion, you will be able to begin using Driverless AI. []

Specifying a License for Scoring Pipelines
When deploying models to production, Driverless AI requires a license to
be specified in order to run both the Python and MOJO Scoring Pipelines.

The license can be specified in one of the
following ways:
-   Via an environment variable:
          file, or
-   Via a system property of JVM (-D option):
-   Via an application classpath:
      -   The license is loaded from a resource called /license.sig. -   The default resource name can be changed via the JVM system
          property ai.h2o.mojos.runtime.license.filename.

QUICK-START TABLES BY ENVIRONMENT

Use the following tables for Cloud, Server, and Desktop to find the
right setup instructions for your environment.

|      |          |                       |
|                 | 8       | ---- |     Seri |                       |
|                 |         | ---- | o        |                       |
|                 | xlarge  | --+  |          |                       |
|                 |         |      | us       |                       |
|                 | ------- |      |          |                       |
|                 | ------- | 1    |     use  |                       |
|                 | -+      |      |          |                       |
|                 |         | 6    | -------- |                       |
|                 |     p2.

| ---- |          |                       |
|                 | 2       | --+  | -------- |                       |
|                 |         |      | -------- |                       |
|                 | xlarge  |      | -+       |                       |
|                 |         | 4    |          |                       |
|                 | ------- |      |     Expe |                       |
|                 | ------- | ---- | r        |                       |
|                 | -+      | ---- |          |                       |
|                 |         | --+  | imentati |                       |
|                 |     p3.

| 1    | us       |                       |
|                 | 1       |      |          |                       |
|                 |         | ---- |     use  |                       |
|                 | 6xlarge | ---- |          |                       |
|                 |         | --+  | -------- |                       |
|                 | ------- |      | -------- |                       |
|                 | ------- |      | -+       |                       |
|                 | -+      | 2    |          |                       |
|                 |         |      |     Seri |                       |
|                 |     g3.

|      |          |                       |
|                 | 8       |      |     Expe |                       |
|                 |         |      | r        |                       |
|                 | xlarge  |      |          |                       |
|                 |         |      | imentati |                       |
|                 | ------- |      | on       |                       |
|                 | -+      |      | -------- |                       |
|                 |     g3.

--------------------- ----- ------- --------------------------------
  NVIDIA DGX-1          Yes   128 GB  install-on-nvidia-dgx
  Ubuntu with GPUs      Yes   64 GB   :ref:install-on-ubuntu-with-gp
                                      us
  Ubuntu with CPUs      No    64 GB   :ref:install-on-ubuntu-cpus-on
                                      ly
  RHEL with GPUs        Yes   64 GB   :ref:install-on-rhel-with-gpus
  RHEL with CPUs        No    64 GB   :ref:install-on-rhel-cpus-only
  IBM Power (Minsky)    Yes   64 GB   Contact sales@h2o.ai

Desktop
  Operating System GPU        Min    Suitable for Refer to Section
                   Support?

FREE UP SPACE ON A DAI INSTANCE

The following sections describe how to free up disk space on an instance
of Driverless AI. Python API guide
This section describes how to free up disk space on an instance of
Driverless AI (DAI) with the Python API. Note
- The method described in this section is only available for H2O AI
Cloud customers. The following code sample lets you perform the following tasks:
1. Link any of your experiments to a Project. Once an experiment is
    linked to a Project, it is automatically pushed to an external
    remote storage. 2. Delete the experiment from the DAI instance. Doing so frees up disk
    space on your DAI instance, and you can always import any experiment
    back into the DAI instance as needed.

DRIVERLESS AI WITH H2O-3 ALGORITHMS

Driverless AI already supports a variety of
algorithms <supported_algorithms>. This example shows how you can use
our H2O-3-MODELS-PY recipe to include H2O-3 supervised learning
algorithms in your experiment. The available H2O-3 algorithms in the
recipe include:

-   Naive Bayes
-   GBM
-   Random Forest
-   Deep Learning
-   GLM
-   AutoML

CAUTION: Because AutoML is treated as a regular ML algorithm here, the
runtime requirements can be large. We recommend that you adjust the

max_runtime_secs`` parameters as suggested here:
https://github.com/h2oai/driverlessai-recipes/blob/rel-1.9.0/models/algorithms/h2o-3-models.py#L45
1. Start an experiment in Driverless AI by selecting your training
    then specifying a Target Column. Notice the list of algorithms that
    will be used in the FEATURE EVOLUTION section of the experiment
    summary. In the example below, the experiment will use LightGBM and
    XGBoostGBM. 2. Click on EXPERT SETTINGS. 3. Specify the custom recipe using one of the following methods:
4. In the Expert Settings page, specify any additional settings and
    then click SAVE. This returns you to the experiment summary. 5. To include each of the new models in your experiment, return to the
    Expert Settings option. Click the RECIPES > INCLUDE SPECIFIC MODELS
    option. Select the algorithm(s) that you want to include.

ABOUT LICENSES

Driverless AI is licensed per a single named user. Therefore, in order,
to have different users run experiments simultaneously, they would each
need a license. Driverless AI manages the GPU(s) that it is given and
ensures that different experiments from different users can run safely
simultaneously and don’t interfere with each other. So when two licensed
users log in with different credentials, neither of them will see the
other’s experiment. Similarly, if a licensed user logs in using a
different set of credentials, that user will not see any previously run
experiments.

INTERPRET A MODEL

Model interpretations can be run on a Driverless AI experiment or on the
predictions created by an external model (that is, a model not created
by Driverless AI). Use the INTERPRET THIS MODEL button on a completed experiment page to
interpret a Driverless AI model on original and transformed features. You can also click the MLI link from the top navigation menu to
interpret either a Driverless AI model or an external model. -   Interpreting a Driverless AI Model <interpret-dai-model>
-   Interpreting Predictions From an External Model <interpret-external-model>

Interpret a Driverless AI model
A completed Driverless AI model can be interpreted from either the
INTERPRETED MODELS page or the completed_experiment. -   from-mli-page
-   from-exp-page
Note
- This release deprecates experiments run in 1.8.9 and earlier. MLI
migration is not supported for experiments from versions <= 1.8.9.

- MLI is not supported for
unsupervised learning models. - MLI is not supported for Image or
multiclass Time Series experiments. - MLI does not require an Internet
connection to run on current models. - To specify a port of a specific
H2O instance for use by MLI, use the h2o_port
config.toml <sample-configtoml> setting. You can also specify an IP
address for use by MLI with the h2o_ip setting. Run interpretations from the Interpreted Models page
The following steps describe how to run an interpretation from the
INTERPRETED MODELS page. 1. Click the MLI link in the upper-right corner of the UI to view a
      list of interpreted models. 2. Click the NEW INTERPRETATION button. The INTERPRETATION SETTINGS
      page is displayed. 3. Select a dataset to use for the interpretation. The selected
      dataset must contain the same columns as the training dataset used
      for the experiment. 4. Specify the Driverless AI model that you want to use for the
      interpretation.

5. Optionally specify which MLI recipes <mli_default_recipes> (or
      Explainers) to run. You can also change
      Explainer (recipe) specific settings <mli_default_recipes> when
      selecting which recipes to use for the interpretation. 6. Optionally specify any additional
      Interpretation Expert Settings <mli_expert_settings> to use when
      running this interpretation. 7. Optionally specify a weight column. 8. Optionally specify one or more dropped columns. Columns that were
      dropped when the model was created are automatically dropped for
      the interpretation. 9. Click the LAUNCH MLI button. Filter input data to MLI
Driverless AI provides a built-in option to filter input data to MLI
when creating a new interpretation. The following steps describe how to
access and use the built-in input data filtering feature. 1. On the INTERPRETED MODELS page, click NEW INTERPRETATION. 2.

[]
3. The rows and columns of the specified dataset are displayed. To
    filter the data, specify a row, operator, and value to filter by,
    then click the FILTER button. To undo your most recent change, click
    the UNDO button. You can also reset the data to its original
    unfiltered state by clicking the RESET button. 4. To confirm your changes, click the SAVE button. Specify a name for
    the filtered dataset, and then click SAVE to proceed. The newly
    created filtered dataset is automatically selected for the
    interpretation. Run interpretation from the Completed Experiment page
completed_experiment. 1. On the Completed Experiment page, click the INTERPRET THIS MODEL
    button. 2. Select a dataset to use for the interpretation. The selected dataset
    must contain the same columns as the training dataset used for the
    experiment. 3.

Note that this
          option is only displayed if you have already run an
          interpretation on the experiment. -   WITH DEFAULT SETTINGS: Run an interpretation using the default
          settings. -   WITH CUSTOM SETTINGS: Run an interpretation using custom
          settings. Selecting this option opens the INTERPRETATION
          SETTINGS page, where you can specify which
          MLI recipes (explainers) <mli_default_recipes> to use for the
          interpretation and change
          explainer-specific settings <mli_default_recipes> and
          interpretation expert settings <mli_expert_settings>. To run
          an interpretation with your specified custom settings, click
          the LAUNCH MLI button. The interpretation includes a summary of the interpretation,
interpretations using the built Driverless AI model, and interpretations
using surrogate models that are built on the predictions from the

The plots are interactive, and the logs / artifacts can be downloaded by
clicking on the ACTIONS button. For non-time-series experiments, this page provides several visual
explanations and reason codes for the trained Driverless AI model and
its results. More information about this page is available in the
Understanding the Model Interpretation Page <interpret-regular-understand-model>
section later in this chapter. []
Interpreting Predictions From an External Model
Model Interpretation does not need to be run on a Driverless AI
experiment. You can train an external model and run Model
Interpretability on the predictions from the model. This can be done
from the MLI page. 1. Click the MLI link in the upper-right corner of the UI to view a
      list of interpreted models. 2. Click the NEW INTERPRETATION button. 3. Leave the SELECT MODEL option to none
  4. Select the dataset that you want to use for the model

If the dataset does not have
      predictions, then you can join the external predictions. An
      example showing how to do this in Python is available in the Run
      Model Interpretation on External Model Predictions section of the
      Credit Card Demo. 5. Specify a Target Column (actuals) and the Prediction Column
      (scores from the external model). 6. Optionally specify any additional MLI
      Expert Settings <mli_expert_settings> to use when running this
      interpretation. 7. Optionally specify a weight column. 8. Optionally specify one or more dropped columns. Columns that were
      the interpretation. 9. Click the LAUNCH MLI button. NOTE: When running interpretations on an external model, leave the
  SELECT MODEL option empty. That option is for selecting a Driverless
  AI model.

interpret-regular-understand-model. Explainer Recipes
Driverless AI Machine Learning Interpretability comes with a number of
out-of-the-box explainer recipes for model interpretation that can be
enabled when
running a new interpretation from the MLI page <from-mli-page>. Details
about the interpretations generated by these recipes can be found
here <interpret-regular-understand-model>. And a list of explainer
specific expert settings can be found here <explainer-expert-settings>. The following is a list of explainers that are run by default for MLI.

For more information, see mli-byor. []
LEGACY EXPLAINERS
The following is a list of legacy explainers that are not run by default
for MLI. You can run legacy explainers by using config variables. -   Absolute Permutation Feature Importance
-   AutoDoc
-   Interpretability Data ZIP (Surrogate and Shapley Techniques)
-   NLP Tokenizer
-   NLP Vectorizer + Linear Model (VLM) Text Feature Importance
-   Original Feature Importance
-   Relative Permutation Feature Importance
-   Shapley Values for Original Features (Kernel SHAP Method)
-   Surrogate Random Forest Importance
-   Surrogate Random Forest Leave-one-covariate-out (LOCO)
-   Surrogate Random Forest Partial Dependence Plot
-   Transformed Feature Importance
-   k-LIME / LIME-SUP
Interpretation Expert Settings
When interpreting from the MLI page <from-mli-page>, a variety of
configuration options are available in the INTERPRETATION EXPERT
SETTINGS panel that let you customize interpretations.

Use the search bar to
refine the list of settings or locate a specific setting. For more information on each of these settings, see
interpretation-expert-settings. Also see <explainer-expert-settings> for
explainer (recipe) specific expert settings. NOTES:
  -   The selection of available expert settings is determined by the
      type of model you want to interpret and the specified LIME method. -   Expert settings are not available for time-series models. Expert Settings from Recipes (Explainers)
For some recipes <mli_default_recipes> like
Driverless AI Partial dependence <partial-dependence-plot>,
Disparate Impact Analysis <dai-dia> (DIA) explainer and
DT (Decision Tree) Surrogate explainer <decision-tree>, some of the
settings can be toggled from the recipe page. Also, enabling some of the
recipes like Original Kernel SHAP explainer <dai-shapley> will add new
options to the expert settings.

SPLIT DATASETS

Driverless AI lets you split a dataset into two subsets that can be used
as training and validation/test datasets during modeling. When splitting
datasets for modeling, each split should have a similar distribution to
avoid over fitting on the training set. Depending on the use case, you
can either split the dataset RANDOMLY, perform a STRATIFIED SAMPLING
based on the TARGET COLUMN, perform a FOLD COLUMN-based split to keep
rows belonging to the same group together, or perform a TIME
COLUMN-based split to train on past data and validate/test on future
data. Perform the following steps to split a dataset:
1. Click the dataset or select the [CLICK FOR ACTIONS] button next to
    the dataset that you want to split and select DATA PREP > SPLIT from
    the submenu that appears. 2. The Dataset Splitter form displays. Optionally edit the default
    output names for each segment of the split.

AUTODOC CUSTOM TEMPLATE PLACEHOLDERS

The following sections describe placeholders for AutoDoc's custom
template feature. Using placeholders
You can customize the content that appears in an AutoDoc report by using
placeholders. When you insert a placeholder into a template, the content
unique to that specific placeholder appears in the generated report in
the location where you inserted it.

This
                                      placeholder is used in the standard
                                      AutoDoc. The content is similar to
                                      Data Overview.DAI Training Data
                                      Detailed but has less descriptive
                                      text and does not include
                                      information about missing values
  -----------------------------------------------------------------------
Methodology
Placeholders related to METHODOLOGY:
  Name                                Description
  ----------------------------------- -----------------------------------
  Methodology.Assumptions             A high-level overview of DAI's
                                      assumptions and limitations.

This
                                      section includes the full feature
                                      importance dataset and is not
                                      limited to a user-specified number
                                      of features
  Final Model.DAI Final Model         A section describing the components
  Architecture                        of the final model
  Final Model.DAI Final Model         A table with the final model's
  Components Table                    summary components like the model
                                      type, model weight, number of
                                      folds, etc.

This placeholder
                                      includes a short paragraph about
                                      final model selection and a
                                      performance table
  Final Model.DAI Model and Component This section includes the model
  Table                               component table (i.e., this
                                      placeholder calls the Final
                                      Model.DAI Final Model Components
                                      Table), which shows information
                                      like the model type, model weight,
                                      number of folds, etc. Final Model.DAI Model Objective     A table with the final models
  Table                               optimized loss function - this is
                                      different than the scorer a user
                                      selected.

This
                                      placeholder is called by the Final
                                      Model.DAI Model Components
                                      placeholder
  Final Model.DAI Models Evaluated    A table with the algorithms
  Table                               available in DAI and the reason an
                                      algorithm was or wasn't selected
                                      for the final model. This
  Final Model.Pipeline Overview       This placeholder is called by the
                                      Final Model.Pipeline placeholder
                                      and shows a table of the final
                                      model components.

Note the local
                                      interpretation based plots and
                                      table require that the user
                                      specifies individual records of
                                      interest with the Python client's
                                      individual_rows parameter
  MLI.KLIME Plot                      A description of kLIME with the
                                      kLIME plot
  MLI.KLIME Reason Code Text          A documentation-type section that
                                      describes kLIME reason codes
  MLI.Local Interpretability Row      This placeholder is only available
  Information                         if the user-specified
                                      individual_rows are provided.

Note this section requires an mli
                                      key or if running through the MLI
                                      UI make sure to run the surrogate
                                      decision tree explainer and the
                                      AutoDoc
Model Tuning
Placeholders related to MODEL TUNING:
  Model Tuning.template               Similar to the Model Tuning.DAI
                                      Hyperparameters placeholder.

This sub
                                      template includes additional
                                      explanations about sensitivity
                                      analysis in general and shows a
                                      records original feature values
                                      along with the ICE overlaid PDP. This template expects a user to
                                      pass in the individual_rows
                                      parameter to the Python client with
                                      records of interest
  Partial Dependence Plots.template   A section describing how partial
                                      dependence plots work and showing
                                      the partial dependence plots.

The reference will appear
                                      for models that use a "bagged
                                      ensemble" (pasting)
  Appendix.NLP Appendix               The full NLP architecture
                                      information. Appendix.PSI_Appendix               The table used to calculate PSI
  Appendix.Response_Rates_Appendix    The quantile-base plots calculation
                                      table. Appendix.template                   This template points to the
                                      Appendix.PSI,
                                      Appendix.Response_Rates_Appendix,
                                      and the Appendix.NLP Appendix.

INSTALL ON UBUNTU

Ubuntu. The installation steps vary depending on whether your system has
  Ubuntu with GPUs          Yes     64 GB
  Ubuntu with CPUs          No      64 GB

Install on Ubuntu with GPUs
NOTE: Driverless AI is supported on Ubuntu 16.04 or later. Open a Terminal and ssh to the machine that will run Driverless AI. Once
    https://www.h2o.ai/download/. (Note that the contents of this Docker
    image include a CentOS kernel and CentOS packages.) 2. Install and run Docker on Ubuntu (if not already installed):
3. Install nvidia-docker2 (if not already installed).

4. Verify that the NVIDIA driver is up and running. If the driver is
    latest NVIDIA Tesla V/P/K series driver:
5. Set up a directory for the version of Driverless AI on the host
    machine:
6. Change directories to the new folder, then load the Driverless AI
    Docker image inside the new directory:
7. Enable persistence of the GPU. Note that this needs to be run once
    http://docs.nvidia.com/deploy/driver-persistence/index.html. 8. Set up the data, log, and license directories on the host machine:
    image tag.

nvidia-docker. GPU support will not be available. **Watch the installation video** `here <https://www.youtube.com/watch?v=ZQRlvLVHQ3s&index=3&list=PLNtMya54qvOE9fs3ylzaR_McnoUsuMV7X>`__. Note that some of the images in this video may change between releases, but the installation steps remain the same. Open a Terminal and ssh to the machine that will run Driverless AI. Once you are logged in, perform the following steps. 1. Retrieve the Driverless AI Docker image from    https://www.h2o.ai/download/. 2. Install and run Docker on Ubuntu (if not already installed):  ..     .. code:: bash        # Install and run Docker on Ubuntu       curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -       sudo apt-key fingerprint 0EBFCD88 sudo add-apt-repository \         "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"       sudo apt-get update       sudo apt-get install docker-ce       sudo systemctl start docker  3.

Change directories to the new folder, then load the Driverless AI    Docker image inside the new directory:  ..     .. code:: bash        # cd into the new directory       cd |VERSION-dir|        # Load the Driverless AI docker image       docker load < dai-docker-ubi8-x86_64-|VERSION-long|.tar.gz  5. Set up the data, log, license, and tmp directories on the host    machine (within the new directory):  ..     .. code:: bash        # Set up the data, log, license, and tmp directories       mkdir data       mkdir log       mkdir license       mkdir tmp  6. At this point, you can copy data into the data directory on the host    machine. The data will be visible inside the Docker container. 7. Rundocker
imagesto find the new image tag. 8. Start the Driverless AI Docker image. Note that GPU support will not    be available.

..     We recommend--shm-size=2g
--cap-add=SYS_NICE --ulimit nofile=131071:131071 --ulimit
nproc=16384:16384in docker launch command. But if user plans to build    :ref:`image auto model <image-model>` extensively, then--shm-size=4gis recommended for Driverless AI docker command.

Connect to Driverless AI with your browser:  ..     .. code:: bash        http://Your-Driverless-AI-Host-Machine:12345  Stopping the Docker Image -------------------------  To stop the Driverless AI Docker image, type **Ctrl + C** in the Terminal (Mac OS X) or PowerShell (Windows 10) window that is running the Driverless AI Docker image. Upgrading the Docker Image --------------------------  This section provides instructions for upgrading Driverless AI versions that were installed in a Docker container. These steps ensure that existing experiments are saved. **WARNING**: Experiments, MLIs, and MOJOs reside in the Driverless AI tmp directory and are not automatically upgraded when Driverless AI is upgraded. -  Build MLI models before upgrading. -  Build MOJO pipelines before upgrading. -  Stop Driverless AI and make a backup of your Driverless AI tmp       directory before upgrading.

Before upgrading, be sure to run MLI jobs on models that you want to    continue to interpret in future releases. If that MLI job appears in    the list of Interpreted Models in your current version, then it will    be retained after upgrading. If you did not build a MOJO pipeline on a model before upgrading    Driverless AI, then you will not be able to build a MOJO pipeline on    that model after upgrading. Before upgrading, be sure to build MOJO    pipelines on all desired models and then back up your Driverless AI    tmp directory. **Note**: Stop Driverless AI if it is still running. Requirements ~~~~~~~~~~~~  We recommend to have NVIDIA driver >= installed (GPU only) in your host environment for a seamless experience on all architectures, including Ampere. Driverless AI ships with CUDA 11.2.2 for GPUs, but the driver must exist in the host environment.

For reference on CUDA Toolkit and Minimum Required Driver Versions and CUDA Toolkit and Corresponding Driver Versions, see `here <https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html>`__ . .. note::  .. If you are using K80 GPUs, the minimum required NVIDIA driver version    is 450.80.02. Upgrade Steps ~~~~~~~~~~~~~  1. SSH into the IP address of the machine that is running Driverless AI. 2. Set up a directory for the version of Driverless AI on the host    machine:  ..     .. code:: bash        # Set up directory with the version name       mkdir |VERSION-dir|        # cd into the new directory       cd |VERSION-dir|  3. Retrieve the Driverless AI package from https://www.h2o.ai/download/    and add it to the new directory. 4. Load the Driverless AI Docker image inside the new directory:  ..     .. code:: bash        # Load the Driverless AI docker image       docker load < dai-docker-ubi8-x86_64-|VERSION-long|.tar.gz  5.

DATA RECIPE FILE SETUP

Driverless AI lets you explore data recipe file data sources from within
the Driverless AI application. This section provides instructions for
configuring Driverless AI to work with local data recipe files. When
enabled (default), you will be able to modify datasets that have been
added to Driverless AI. (Refer to modify_by_recipe for more
information.) NOTES:
-   This connector is enabled by default. These steps are provided in
    case this connector was previously disabled and you want to
    re-enable it. -   Depending on your Docker install version, use either the
    Docker 19.03) command when starting the Driverless AI Docker image. Use docker version to check which version of Docker you are using. Enable Data Recipe File
This example enables the data recipe file data connector.

MODIFYING DATASETS

Viewing dataset details
To view a summary of a dataset or to preview the dataset, click on the
dataset or select the [CLICK FOR ACTIONS] button next to the dataset
that you want to view and select DETAILS from the submenu that appears. This opens the Dataset Details page, which provides a summary of the
dataset that lists each of the dataset's columns and displays
accompanying rows for column name, feature engineering type
(categorical, date, datetime, ID, numerical, text, or image), storage
type (integer, string, real, boolean, or time), count, number of missing
values, mean, minimum, maximum, standard deviation, frequency, and
number of unique values. Hover over the top of a column to view a summary of the first 20 rows of
that column. To view information for a specific column, type the column
name in the field above the graph. To switch the view and preview the dataset, click the DATASET ROWS
button in the top right portion of the UI.

[]
A number of dataset-specific actions are available from this page by
clicking DATASET ACTIONS. These are the same options that are available
from the DATASETS page. For more information on these options, see
datasets_options. []

Change column type
Driverless AI also lets you change a column's type. If a column's data
type or distribution does not match the manner in which you want the
column to be handled during an experiment, changing the LOGICAL TYPE can
help to make the column fit better. For example, an integer zip code can
be changed into a categorical so that it is only used with
categorical-related feature engineering. For Date and Datetime columns,
use the FORMAT option. To change the Logical Type or Format of a column,
click on the group of square icons located to the right of the words
AUTO-DETECT. (The squares light up when you hover over them with your
cursor.) Then select the new column type for that column.

Scoring pipelines can
be created on the new dataset by building an experiment. This feature is
useful when you want to make changes to the training data that you would
not need to make on the new data you are predicting on. For example, you
can change the target column from regression to classification, add a
weight column to mark specific training rows as being more important, or
remove outliers that you do not want to model on. Refer to the
custom_recipes_data_recipes section for more information. Click the MODIFY BY RECIPE drop-down menu in the top right portion of
the UI and select from the following options:
-   DATA RECIPE URL: Load a custom recipe from a URL to use to modify
    the dataset. The URL must point to either an HTML or raw version of
    the file, a GitHub repository or tree, or a local file. Sample
    custom data recipes are available in the
    driverlessai-recipes repository <https://github.com/h2oai/driverlessai-recipes/tree/>.

-   LIVE CODE: Manually enter custom recipe code that is used to modify
    the dataset. Click the GET PREVIEW button to preview the code's
    effect on the dataset, then click APPLY to create a new dataset. To
    download the entered code script as a Python file, click the
    DOWNLOAD button. -   APPLY EXISTING DATA RECIPE: Apply an existing data recipe to the
    dataset. For more information on adding recipes, see custom-recipes. NOTES:
-   These options are enabled by default. You can disable them by
    removing recipe_file and recipe_url from the enabled_file_systems
    configuration option. -   Modifying a dataset with a recipe does not overwrite the original
    dataset. The dataset that is selected for modification remains in
    the list of available datasets in its original form, and the
    modified dataset appears in this list as a new dataset. -   Changes made to the original dataset through this feature are not
    applied to any new data that is scored.

MISSING AND UNSEEN LEVELS HANDLING

This section describes how missing and unseen levels are handled by each
algorithm during training and scoring. How Does the Algorithm Handle Missing Values During Training? LightGBM, XGBoost, RuleFit
Driverless AI treats missing values natively. (I.e., a missing value is
treated as a special value.) Experiments rarely benefit from imputation
techniques, unless the user has a strong understanding of the data. GLM
Driverless AI automatically performs mean value imputation (equivalent
to setting the value to zero after standardization). TensorFlow
Driverless AI provides an imputation setting for TensorFlow in the
config.toml file: tf_nan_impute_value (post-normalization). If you set
this option to 0, then missing values will be imputed by the mean. Setting it to (for example) +5 will specify 5 standard deviations above
the mean of the distribution.

Specify 0 if you prefer mean
imputation. FTRL
In FTRL, missing values have their own representation for each datable
column type. These representations are used to hash the missing value,
with their column's name, to an integer. This means FTRL replaces
missing values with special constants that are the same for each column
type, and then treats these special constants like a normal data value. Unsupervised Algorithms
For unsupervised algorithms <unsupervised_algos>, standardization in the
pre-transformation layer (where it is decided which columns and column
encodings are fed in for clustering) is performed by ignoring any
missing values. Scikit-learn’s StandardScaler is used internally during
the standardization process. Missing values are then replaced with 0 for
further calculations or clustering. How Does the Algorithm Handle Missing Values During Scoring (Production)?

This optimal direction is then used for missing values during
scoring. If no missing data is present during training (for a particular
feature), then the majority path is followed if the value is missing. GLM
Missing values are replaced by the mean value (from training), same as
in training. TensorFlow
Missing values are replaced by the same value as specified during
training (parameterized by tf_nan_impute_value). FTRL
To ensure consistency, FTRL treats missing values during scoring in
exactly the same way as during training. Clustering in Transformers
Missing values are replaced with the mean along each column. This is
used only on numeric columns. Isolation Forest Anomaly Score Transformer
Isolation Forest uses out-of-range imputation that fills missing values
with the values beyond the maximum. What Happens When You Try to Predict on a Categorical Level Not Seen During Training?

BUILDING MODELS IN DRIVERLESS AI

launching ga modeling_before_you_begin running-experiment time-series
nlp image-processing unsupervised

EXPERIMENTS

experiment-settings expert-settings scorers experiment-new
experiment-sharing experiment-comparison experiment-completed
experiment-insights experiment-scores experiment-graphs
experiment-summary experiment-performance

diagnosing view-experiments retrain-using-recipes leaderboard projects

GOOGLE BIGQUERY SETUP

Driverless AI lets you explore Google BigQuery (GBQ) data sources from
within the Driverless AI application. This page provides instructions
for configuring Driverless AI to work with GBQ. Note
The setup described on this page requires you to enable authentication. Enabling the GCS and/or GBQ connectors causes those file systems to be
displayed in the UI, but the GCS and GBQ connectors cannot be used
without first enabling authentication. Before enabling the GBQ data connector with authentication, the
following steps must be performed:
1. In the Google Cloud Platform (GCP), create a private key for your
    service account. To create a private key, click SERVICE ACCOUNTS >
    KEYS, and then click the ADD KEY button. When the CREATE PRIVATE KEY
    dialog appears, select JSON as the key type. To finish creating the
    JSON private key and download it to your local file system, click
    CREATE. 2. Mount the downloaded JSON file to the Docker instance.

Specify the path to the downloaded and mounted auth-key.json file
    with the gcs_path_to_service_account_json config option. Note
Depending on your Docker install version, use either the
docker run --runtime=nvidia (>= Docker 19.03) or nvidia-docker (< Docker
19.03) command when starting the Driverless AI Docker image. Use
docker version to check which version of Docker you are using. The following sections describe how to enable the GBQ data connector:
-   gbq-config-toml
-   gbq-environment-variable
-   gbq-workload-identity

Enabling GBQ with the config.toml file
This example enables the GBQ data connector with authentication by
contains Google BigQuery authentications.

GOOGLE_APPLICATION_CREDENTIALSenvironment variable as follows:  ::     export GOOGLE_APPLICATION_CREDENTIALS="SERVICE_ACCOUNT_KEY_PATH"  In the preceding example, replaceSERVICE_ACCOUNT_KEY_PATHwith the path of the JSON file that contains your service account key.

.. _gbq-workload-identity:  Enabling GBQ by enabling Workload Identity for your GKE cluster ---------------------------------------------------------------  The GBQ data connector can be configured by enabling Workload Identity for your Google Kubernetes Engine (GKE) cluster. For information on how to enable Workload Identity, refer to the `official documentation on enabling Workload Identity on a GKE cluster <https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#enable_on_cluster>`__. .. note::     If Workload Identity is enabled, then theGOOGLE_APPLICATION_CREDENTIALSenvironment variable does not need    to be set. Adding Datasets Using GBQ -------------------------  After Google BigQuery is enabled, you can add datasets by selecting **Google Big Query** from the **Add Dataset (or Drag and Drop)** drop-down menu.

.. figure:: ../images/add_dataset_dropdown.png    :alt:   Specify the following information to add your dataset:  1. **Enter BQ Dataset ID with write access to create temporary table**:    Enter a dataset ID in Google BigQuery that this user has read/write    access to. BigQuery uses this dataset as the location for the new    table generated by the query. ..     **Note**: Driverless AI's connection to GBQ will inherit the    top-level directory from the service JSON file. So if a dataset named    "my-dataset" is in a top-level directory named "dai-gbq", then the    value for the dataset ID input field would be "my-dataset" and not    "dai-gbq:my-dataset". 2. **Enter Google Storage destination bucket**: Specify the name of    Google Cloud Storage destination bucket. Note that the user must have    write access to this bucket. 3. **Enter Name for Dataset to be saved as**: Specify a name for the    dataset, for example,my_file. 4.

AUTOMATIC FEATURE ENGINEERING

Driverless AI performs automatic feature engineering as part of an
experiment's model building process. New features are created by
performing transformations <Transformations> and/or
interactions <max-feature-interaction-depth> on the dataset columns. The
default transformers picked up by Driverless depends on interpretability
settings of an experiment. For more interpretable models, simpler
transformations are applied. This can be seen in the preview of the
experiment. Feature engineering expert settings like _include/exclude
transformers_ can be used to control the applied transformations. Transformers like binning, target encoding, weight of evidence,
clustering, dimensionality reduction, autoencoders, TensorFlow, NLP BERT
models, lags, aggregates, can be used to create Feature interactions.

The details about features created (transformations applied) and used by
the experiment can be obtained from the Autodoc <autodoc> of an
experiment. The feature engineering effort and evolution can be controlled from the
features-settings on the expert panel of an experiment. []
You can also upload your own CUSTOM TRANSFORMERS to be included in
addition to the Driverless AI built in transformers. Some open-source
custom transformers can be obtained from Driverless AI open-source
custom recipes <custom-recipes>. The inclusion of the transformers can be controlled from the Recipe tab
of the expert panel of an experiment. []
Preprocessing transformers <included_pretransformers> can be used to
control the features fed into the evolutionary transformer layer. You can obtain the dataset with (engineered) features by clicking
Transform Dataset <Transform_Another_Dataset> from MODEL ACTIONS of a
finished experiment.

DRIVERLESS AI TRANSFORMATIONS

Transformations in Driverless AI are applied to columns in the data. The
transformers create the engineered features <feature_engineering> in
experiments. Driverless AI provides a number of transformers. The downloaded
experiment logs include the transformations that were applied to your
experiment. NOTES:
-   You can include or exclude specific transformers in your Driverless
    AI environment using the included_transformers or
    excluded_transformers config options. -   You can control which transformers to use in individual experiments
    with the included_transformers Expert Setting in Recipe panel. -   You can set transformers to be used as pre-processing transformers
    with the included_pretransformers Expert Setting in Recipe panel. Additional layers can be added with the num_pipeline_layers Expert
    Setting in Recipe panel.

Available Transformers
The following transformers are available for regression and
classification (multiclass and binary) experiments. -   Numeric Transformers <numeric_transformers>
-   Categorical Transformers <cat_transformers>
-   Time and Date Transformers <time_transformers>
-   Time Series Transformers <ts_transformers>
-   NLP (text) Transformers <text_transformers>
-   Image Transformers <image_transformers>
-   Autoviz Recommendation Transformer <autoviz_transformer>

Transformed Feature Naming Convention
Transformed feature names are encoded as follows:
  <Transformation_indexORgene_details_id>_<Transformation_name>:<original_feature_name>:<...>:<original_feature_name>.<extra>
For example in 32_NumToCatTE:BILL_AMT1:EDUCATION:MARRIAGE:SEX.0 :
  -   32_ is the transformation index for specific transformation
      parameters. -   NumToCatTE is the transformer name. -   BILL_AMT1:EDUCATION:MARRIAGE:SEX represents original features
      used.

For
      multiclass experiments, this value is > 0. For binary experiments,
      this value is always 0. Numeric Transformers (Integer, Real, Binary)
-   ClusterDist Transformer
      The Cluster Distance Transformer clusters selected numeric columns
      and uses the distance to a specific cluster as a new feature. -   ClusterDist cuML Transformer
      The Cluster Distance cuML Transformer runs on GPUs to train cuML
      accelerated k-means clustering to create clusters on selected
      numeric columns and uses the distance to a specific cluster as a
      new feature. -   ClusterTE Transformer
      The Cluster Target Encoding Transformer clusters selected numeric
      columns and calculates the mean of the response column for each
      cluster. The mean of the response is used as a new feature. Cross
      Validation is used to calculate mean response to prevent
      overfitting.

-   Interactions Transformer
      The Interactions Transformer adds, divides, multiplies, and
      subtracts two numeric columns in the data to create a new feature. This transformation uses a smart search to identify which feature
      pairs to transform. Only interactions that improve the baseline
      model score are kept. -   InteractionsSimple Transformer
      The InteractionsSimple Transformer adds, divides, multiplies, and
      subtracts two numeric columns in the data to create a new feature. This transformation randomly selects pairs of features to
      transform. -   NumCatTE Transformer
      The Numeric Categorical Target Encoding Transformer calculates the
      mean of the response column for several selected columns. If one
      of the selected columns is numeric, it is first converted to
      categorical by binning. The mean of the response column is used as
      a new feature.

-   NumToCatTE Transformer
      The Numeric to Categorical Target Encoding Transformer converts
      numeric columns to categoricals by binning and then calculates the
      mean of the response column for each group. The mean of the
      response for the bin is used as a new feature. Cross Validation is
      used to calculate mean response to prevent overfitting. -   NumToCatWoEMonotonic Transformer
      The Numeric to Categorical Weight of Evidence Monotonic
      Transformer converts a numeric column to categorical by binning
      and then calculates Weight of Evidence for each bin. The monotonic
      constraint ensures the bins of values are monotonically related to
      the Weight of Evidence value. The Weight of Evidence is used as a
      new feature. Weight of Evidence measures the “strength” of a
      grouping for separating good and bad risk and is calculated by
      taking the log of the ratio of distributions for a binary response
      column.

The Weight of Evidence is used as
      a new feature. Weight of Evidence measures the “strength” of a
      column. -   Original Transformer
      The Original Transformer applies an identity transformation to a
      numeric column. -   TruncSVDNum Transformer
      Truncated SVD Transformer trains a Truncated SVD model on selected
      numeric columns and uses the components of the truncated SVD
      matrix as new features. -   TruncSVDNum cuML Transformer
      The Truncated SVD cuML Transformer runs on GPUs to train cuML
      accelerates Truncated SVD model on selected numeric columns and
      uses the components of the truncated SVD matrix as new features.

-   DateTimeOriginal Transformer
      The Date Time Original Transformer retrieves date _and_ time
      values such as year, quarter, month, day, day of the year, week,
      weekday, hour, minute, and second values. -   EwmaLags Transformer
      The Exponentially Weighted Moving Average (EWMA) Transformer
      calculates the exponentially weighted moving average of target or
      feature lags. -   LagsAggregates Transformer
      The Lags Aggregates Transformer calculates aggregations of
      target/feature lags like mean(lag7, lag14, lag21) with support for
      mean, min, max, median, sum, skew, kurtosis, std. The aggregation
      is used as a new feature. -   LagsInteraction Transformer
      The Lags Interaction Transformer creates target/feature lags and
      calculates interactions between the lags (lag2 - lag1, for
      instance). The interaction is used as a new feature.

Each lag is used as a new feature. Lag transformers may
      apply to categorical (strings) features or binary/multiclass
      string valued targets after they have been internally numerically
      encoded. -   LinearLagsRegression Transformer
      The Linear Lags Regression transformer trains a linear model on
      the target or feature lags to predict the current target or
      feature value. The linear model prediction is used as a new
      feature. Categorical Transformers (String)
-   Cat Transformer
      The Cat Transformer sorts a categorical column in lexicographical
      order and uses the order index created as a new feature. This
      transformer works with models that cannot handle categorical
      features. -   CatOriginal Transformer
      The Categorical Original Transformer applies an identity
      transformation that leaves categorical features as they are. This
      transformer works with models that can handle non-numeric feature
      values.

-   CVTargetEncode Transformer
      The Cross Validation Target Encoding Transformer calculates the
      mean of the response column for each value in a categorical column
      and uses this as a new feature. Cross Validation is used to
      calculate mean response to prevent overfitting. -   Frequent Transformer
      The Frequent Transformer calculates the frequency for each value
      in categorical column(s) and uses this as a new feature. This
      count can be either the raw count or the normalized count. -   LexiLabelEncoder Transformer
      The Lexi Label Encoder sorts a categorical column in
      lexicographical order and uses the order index created as a new
      feature. -   NumCatTE Transformer
      categorical by binning.

Cross Validation is used to calculate mean response
      to prevent overfitting. -   OneHotEncoding Transformer
      The One-hot Encoding transformer converts a categorical column to
      a series of Boolean features by performing one-hot encoding. The
      Boolean features are used as new features. If there are more than
      a specific number of unique values in the column, then they will
      be binned to the max number (10 by default) in lexicographical
      order. This value can be changed with the ohe_bin_list config.toml
      configuration option. -   SortedLE Transformer
      The Sorted Label Encoding Transformer sorts a categorical column
      by the response column and uses the order index created as a new
      feature. -   WeightOfEvidence Transformer
      The Weight of Evidence Transformer calculates Weight of Evidence
      for each value in categorical column(s). The Weight of Evidence is
      used as a new feature.

[]
      This only works with a binary target variable. The likelihood
      needs to be created within a stratified k-fold if a fit_transform
      method is used. More information can be found here:
      http://ucanalytics.com/blogs/information-value-and-weight-of-evidencebanking-case/. Text Transformers (String)
-   BERT Transformer
      The Bidirectional Encoder Representations from Transformers (BERT)
      Transformer creates new features for each text column based on the
      pre-trained model embeddings and is ideally suited for datasets
      that contain additional important non-text features. -   TextBiGRU Transformer
      The Text Bidirectional GRU Transformer trains a bi-directional GRU
      TensorFlow model on word embeddings created from a text feature to
      predict the response column. The GRU prediction is used as a new
      feature. Cross Validation is used when training the GRU model to
      prevent overfitting.

The CNN prediction is used as a new feature. Cross Validation is used when training the CNN model to prevent
      overfitting. -   TextCNN Transformer
      The Text CNN Transformer trains a CNN TensorFlow model on word
      embeddings created from a text feature to predict the response
      column. The CNN prediction is used as a new a feature. Cross
      Validation is used when training the CNN model to prevent
      overfitting. -   TextLinModel Transformer
      The Text Linear Model Transformer trains a linear model on a
      TF-IDF matrix created from a text feature to predict the response
      column. The linear model prediction is used as a new feature. Cross Validation is used when training the linear model to prevent
      overfitting. -   Text Transformer
      The Text Transformer tokenizes a text column and creates a TFIDF
      matrix (term frequency-inverse document frequency) or count (count
      of the word) matrix.

Selected components of the TF-IDF/Count matrix are used as new
      features. -   TextOriginal Transformer
      The TextOriginal Transformer performs no feature engineering on
      the text column. Note that this transformer is only available for
      models that have text feature support. Models that have text
      feature support are ImageAutoModel, FTRL, BERT, and unsupervised
      models, in addition to custom model recipes where _can_handle_text
      is set to True. Time Transformers (Date, Time)
-   Dates Transformer
      The Dates Transformer retrieves any date values, including:
      -   Year
      -   Quarter
      -   Month
      -   Day
      -   Day of year
      -   Week
      -   Week day
      -   Hour
      -   Minute
      -   Second
-   IsHoliday Transformer
      The Is Holiday Transformer determines if a date column is a
      holiday. A Boolean column indicating if the date is a holiday is
      added as a new feature.

Other countries available in the python
      Holiday package can be added via the configuration file. Image Transformers
-   ImageOriginal Transformer
      The Image Original Transformer passes image paths to the model
      without performing any feature engineering. -   ImageVectorizer Transformer
      The Image Vectorizer Transformer uses pre-trained ImageNet models
      to convert a column with an image path or URI to an embeddings
      (vector) representation that is derived from the last global
      average pooling layer of the model. NOTE: Fine-tuning of the pre-trained image models can be enabled
      with the image-model-fine-tune expert setting. Autoviz Recommendation Transformer
The Autoviz recommendation transformer applies the recommended
transformations obtained by
visualizing the dataset in Driverless AI <autoviz_reco>.

The
autoviz_recommended_transformation <autoviz_recommended_transformation>
in the expert experiment settings list/control the transformation
applied. The syntax is a dict of transformations from Autoviz
{column_name: transformation} like
{"DIS":"log","INDUS":"log","RAD":"inverse","ZN":"square_root"}. The
Autoviz recommendation transformer itself can be enabled or disabled
from the expert panel by included_transformers <included_transformers>
config setting. This transformer is supported in
python scoring pipelines <Python_Pipeline> and
mojo scoring pipelines with Java Runtime <Mojo_Pipeline> (no C++ support
at the moment). Example Transformations
In this section, we will describe some of the available transformations
using the example of predicting house prices on the example dataset.

Bulk Interactions Transformer
-   add, divide, multiply, and subtract two columns in the data
  Date    Square     Num    Num     St   Pric e Interaction_NumBeds#
  Built   Footage    Beds   Baths   at e        subtract#NumBaths
  ------- ---------- ------ ------- ---- ------ ----------------------
  01/01   1700       3      2       NY   700,   1
  /1920                                  000    
There is one more bedroom than there are number of bathrooms for this
property.

Dates Transformer
-   get year, get quarter, get month, get day, get day of year, get
    week, get week day, get hour, get minute, get second
  Date      Square       Num Beds Num      Sta   Price   DateBuilt_M
  Built     Footage               Baths    te            onth
  --------- ------------ -------- -------- ----- ------- -------------
  01/01/1   1700         3        2        NY    700,0   1
  920                                            00      
The home was built in the month January.

*In order to prevent overfitting, Driverless AI calculates this average
on out-of-fold data using cross validation. Numeric to Categorical Target Encoding Transformer
-   numeric column converted to categorical by binning
-   cross validation target encoding done on the binned numeric column
  Date     Square      Num     Num      Sta   Pric e CV_TE_SquareFo
  Built    Footage     Beds    Baths    te           otage
  -------- ----------- ------- -------- ----- ------ ----------------
  01/01/   1700        3       2        NY    700,   345,000
  1920                                        000    
The column Square Footage has been bucketed into 10 equally populated
bins. This property lies in the Square Footage bucket 1,572 to 1,749. The average price of properties with this range of square footage is
$345,000*.

Cluster Target Encoding Transformer
-   selected columns in the data are clustered
-   target encoding is done on the cluster ID
  Date    Square    Num    Num    St   Pri   ClusterTE_4_NumBeds_Nu
  Built   Footage   Beds   Bath s at e ce    mBaths_SquareFootage
  ------- --------- ------ ------ ---- ----- ------------------------
  01/01   1700      3      2      NY   700   450,000
  /1920                                ,00 0 
The columns: Num Beds, Num Baths, Square Footage have been segmented
into 4 clusters. The average price of properties in the same cluster as
the selected property is $450,000*. *In order to prevent overfitting, Driverless AI calculates this average
on out-of-fold data using cross validation.

VISUALIZING DATASETS

Perform one of the following steps to visualize a dataset:
-   On the Datasets page, select the [CLICK FOR ACTIONS] button beside
    the dataset that you want to view, and then click VISUALIZE from the
    submenu that appears. -   Click the AUTOVIZ top menu link to go to the Visualizations list
    page, click the NEW VISUALIZATION button, then select or import the
    dataset that you want to visualize. The Visualization page shows all available graphs for the selected
dataset. Note that the graphs on the Visualization page can vary based
on the information in your dataset. You can also view and download logs
that were generated during the visualization. Autoviz Recommendations
For some cases, Autoviz suggests certain recommended transformations to
the columns of the dataset. These recommendations can be directly applied to the experiment. This is
done internally by using the
autoviz recommendation transformer <autoviz_transformer>.

Correlated Scatterplots
Correlated scatterplots are 2D plots with large values of the squared
Pearson correlation coefficient. All possible scatterplots based on
pairs of features (variables) are examined for correlations. The
displayed plots are ranked according to the correlation. Some of these
plots may not look like textbook examples of correlation. The only
criterion is that they have a large value of squared Pearson's r
(greater than .95). When modeling with these variables, you may want to
leave out variables that are perfectly correlated with others. Note that points in the scatterplot can have different sizes. Because
  Driverless AI aggregates the data and does not display all points, the
  bigger the point is, the bigger number of exemplars (aggregated
  points) the plot covers. Spikey Histograms
Spikey histograms are histograms with huge spikes. This often indicates
an inordinate number of single values (usually zeros) or highly similar
values.

You should be careful when
modeling (particularly regression models) with spikey variables. Skewed Histograms
Skewed histograms are ones with especially large skewness (asymmetry). The robust measure of skewness is derived from Groeneveld, R.A. and
Meeden, G. (1984), "Measuring Skewness and Kurtosis." The Statistician,
33, 391-399. Highly skewed variables are often candidates for a
transformation (e.g., logging) before use in modeling. The histograms in
the output are sorted in descending order of skewness. Varying Boxplots
Varying boxplots reveal unusual variability in a feature across the
categories of a categorical variable. The measure of variability is
computed from a robust one-way analysis of variance (ANOVA). Sufficiently diverse variables are flagged in the ANOVA. A boxplot is a
graphical display of the fractiles of a distribution.

Sometimes outliers occur, in which case the adjacent whisker is
shortened to the next lower or upper value. For variables (features)
having only a few values, the boxes can be compressed, sometimes into a
single horizontal line at the median. Heteroscedastic Boxplots
Heteroscedastic boxplots reveal unusual variability in a feature across
the categories of a categorical variable. Heteroscedasticity is
calculated with a Brown-Forsythe test: Brown, M. B. and Forsythe, A. B. (1974), "Robust tests for equality of variances. Journal of the American
Statistical Association, 69, 364-367. Plots are ranked according to
their heteroscedasticity values. A boxplot is a graphical display of the
fractiles of a distribution. The center of the box denotes the median,
the edges of a box denote the lower and upper quartiles, and the ends of
the "whiskers" denote that range of values. Sometimes outliers occur, in
which case the adjacent whisker is shortened to the next lower or upper
value.

Biplots
A Biplot is an enhanced scatterplot that uses both points and vectors to
represent structure simultaneously for rows and columns of a data
matrix. Rows are represented as points (scores), and columns are
represented as vectors (loadings). The plot is computed from the first
two principal components of the correlation matrix of the variables
(features). You should look for unusual (non-elliptical) shapes in the
points that might reveal outliers or non-normal distributions. And you
should look for purple vectors that are well-separated. Overlapping
vectors can indicate a high degree of correlation between variables. Outliers
Variables with anomalous or outlying values are displayed as red points
in a dot plot. Dot plots are constructed using an algorithm in
Wilkinson, L. (1999). "Dot plots." The American Statistician, 53,
276–281. Not all anomalous points are outliers.

You should inspect outliers to see if they are
miscodings or if they are due to some other mistake. Outliers should
ordinarily be eliminated from models only when there is a reasonable
explanation for their occurrence. Correlation Graph
The correlation network graph is constructed from all pairwise squared
correlations between variables (features). For continuous-continuous
variable pairs, the statistic used is the squared Pearson correlation. For continuous-categorical variable pairs, the statistic is based on the
squared intraclass correlation (ICC). This statistic is computed from
the mean squares from a one-way analysis of variance (ANOVA). The
formula is (MSbetween - MSwithin)/(MSbetween + (k - 1)MSwithin), where k
is the number of categories in the categorical variable. For
categorical-categorical pairs, the statistic is computed from Cramer's V
squared.

From this table, we compute a chi-square
statistic. Cramer's V squared statistic is then (chi-square / n) /
min(k1,k2), where n is the total of the joint frequencies in the table. Variables with large values of these respective statistics appear near
each other in the network diagram. The color scale used for the
connecting edges runs from low (blue) to high (red). Variables connected
by short red edges tend to be highly correlated. Parallel Coordinates Plot
A Parallel Coordinates Plot is a graph used for comparing multiple
variables. Each variable has its own vertical axis in the plot. Each
profile connects the values on the axes for a single observation. If the
data contain clusters, these profiles will be colored by their cluster
number. Radar Plot
A Radar Plot is a two-dimensional graph that is used for comparing
multiple variables. Each variable has its own axis that starts from the
center of the graph.

Each profile,
which usually appears in the form of a star, connects the values on the
axes for a single observation. Multivariate outliers are represented by
red profiles. The Radar Plot is the polar version of the popular
Parallel Coordinates plot. The polar layout enables us to represent more
variables in a single plot. Data Heatmap
The heatmap graphic is constructed from the transposed data matrix. Rows
of the heatmap represent variables, and columns represent cases
(instances). The data are standardized before display so that small
values are yellow and large values are red. The rows and columns are
permuted via a singular value decomposition (SVD) of the data matrix so
that similar rows and similar columns are near each other. Recommendations
The recommendations graphic implements the Tukey ladder of powers
collection of log, square root, and inverse data transformations
described in _Exploratory Data Analysis_ (Tukey, 1977).

Yeo and R.A. Johnson, “A new family of power
transformations to improve normality or symmetry.” Biometrika, 87(4),
(2000). For each transformer, transformations are selected by comparing
the robust skewness of the transformed column with the robust skewness
of the original raw column. When a transformation leads to a relatively
low value of skewness, it is recommended. Missing Values Heatmap
The missing values heatmap graphic is constructed from the transposed
data matrix. Rows of the heatmap represent variables and columns
represent cases (instances). The data are coded into the values 0
(missing) and 1 (nonmissing). Missing values are colored red and
nonmissing values are left blank (white). The rows and columns are
that similar rows and similar columns are near each other. Gaps Histogram
The gaps index is computed using an algorithm of Wainer and Schacht
based on work by John Tukey.

MULTINODE TRAINING (ALPHA)

Driverless AI can be configured to run in a multinode worker mode. This
document describes the multinode training process and how to configure
it.

NOTES: For more information on queuing in Driverless AI, see
dai-queuing.

redis_multinode dask_multinode multinode_example health_api

LDAP AUTHENTICATION EXAMPLE

This section describes how to enable Lightweight Directory Access
Protocol in Driverless AI. The available parameters can be specified as
environment variables when starting the Driverless AI Docker image, or
they can be set via the config.toml file for native installs. Upon
completion, all the users in the configured LDAP should be able to log
in to Driverless AI and run experiments, visualize datasets, interpret
models, etc. NOTE: Driverless AI does not support LDAP client auth. If you have LDAP
client auth enabled, then the Driverless AI LDAP connector will not
work. Description of Configuration Attributes
The following options can be specified when enabling LDAP
authentication. -   ldap_server: The LDAP server domain or IP. -   ldap_port: The LDAP server port. -   ldap_bind_dn: The complete distinguished name (DN) of the LDAP bind
    user. -   ldap_bind_password: The password for the LDAP bind.

-   ldap_use_ssl: Whether to enable (TRUE) or disable (FALSE) SSL. -   ldap_search_base: The location in the Directory Information Tree
    (DIT) where the search will start. -   ldap_search_filter: A string that describes what you are searching
    for. You can use Python substitution to have this constructed
    dynamically. (Only {{DAI_USERNAME}} is supported. For example,
    "(&(objectClass=person)(cn:dn:={{DAI_USERNAME}}))".) -   ldap_search_attributes: LDAP attributes to return from search. -   ldap_user_name_attribute="uid": Specify the key to find user name. LDAP without SSL
The following examples describe how to enable LDAP without SSL when
running Driverless AI in the Docker image or through native installs. If
the configuration and authentication authentication are successful, the
user can access Driverless AI and run experiments, visualize datasets,
interpret models, etc.

Upon completion, all the users in
the configured LDAP should be able to log in to Driverless AI and run
experiments, visualize datasets, interpret models, etc. Docker Image Installs
Specify the following LDAP environment variables when starting the
Driverless AI Docker image. This example enables LDAP authentication and
shows how to specify additional options enabling SSL.

SCORING PIPELINES OVERVIEW

Driverless AI provides Scoring Pipelines that can be deployed to
production for experiments <main-build-models> and/or
interpreted <interpret-regular-model> models. -   A standalone PYTHON SCORING PIPELINE is available for experiments
    and interpreted models. -   A low-latency, standalone MOJO SCORING PIPELINE is available for
    experiments, with both Java and C++ backends. The Python Scoring Pipeline is implemented as a Python WHL file. While
this allows for a single process scoring engine, the scoring service is
generally implemented as a client/server architecture and supports
interfaces for TCP and HTTP. The MOJO (Model Objects, Optimized) Scoring Pipeline provides a
standalone scoring pipeline that converts experiments to MOJOs, which
can be scored in real time. The MOJO Scoring Pipeline is available as
either a Java runtime <Mojo_Pipeline> or a
C++ runtime <cpp_scoring_pipeline>.

HDFS SETUP

Driverless AI lets you explore HDFS data sources from within the
configuring Driverless AI to work with HDFS.

docker run
versionto check which version of Docker you are using. Description of Configuration Attributes ---------------------------------------  -hdfs_config_path(Required): The location the HDFS config folder    path. This folder can contain multiple config files. -hdfs_auth_type(Required): Specifies the HDFS authentication. Available values are:        -principal: Authenticate with HDFS with a principal user. -keytab: Authenticate with a keytab (recommended). If          running DAI as a service, then the Kerberos keytab needs to be          owned by the DAI user. -keytabimpersonation: Login with impersonation using a          keytab. -noauth: No authentication needed. -key_tab_path: The path of the principal key tab file. This is    required whenhdfs_auth_type='principal'. -hdfs_app_principal_user: The Kerberos application principal user.

-hdfs_app_jvm_args: JVM args for HDFS distributions. Separate each    argument with spaces. --Djava.security.krb5.conf--Dsun.security.krb5.debug--Dlog4j.configuration-hdfs_app_classpath: The HDFS classpath. -hdfs_app_supported_schemes: The list of DFS schemas that is used    to check whether a valid input to the connector has been established. For example:     ::        hdfs_app_supported_schemes = ['hdfs://', 'maprfs://', 'custom://']     The following are the default values for this option. Additional    schemas can be supported by adding values that are not selected by    default to the list. -hdfs://-maprfs://-swift://-hdfs_max_files_listed: Specifies the maximum number of files that    are viewable in the connector UI. Defaults to 100 files. To view more    files, increase the default value. -hdfs_init_path: Specifies the starting HDFS path displayed in the    UI of the HDFS browser. -enabled_file_systems: The file systems you want to enable.

Example 1: Enable HDFS with No Authentication ---------------------------------------------  .. container:: tabs     .. group-tab:: Docker Image Installs     This example enables the HDFS data connector and disables HDFS    authentication. It does not pass any HDFS configuration file; however    it configures Docker DNS by passing the name and IP of the HDFS name    node. This lets you reference data stored in HDFS directly using name    node address, for example:hdfs://name.node/datasets/iris.csv.

Note that this example enables HDFS with       no authentication. 1. Configure the Driverless AI config.toml file. Set the following          configuration options. Note that the procsy port, which          defaults to 12347, also has to be changed. ..           -enabled_file_systems
= "file, upload, hdfs"-procsy_ip = "127.0.0.1"-procsy_port =
80802. Mount the config.toml file into the Docker container.

This allows users to       reference data stored in HDFS directly using the name node       address, for example:hdfs://name.node/datasets/iris.csv. 1. Export the Driverless AI config.toml file or add it to          ~/.bashrc. For example:        ..           ::              # DEB and RPM             export DRIVERLESS_AI_CONFIG_FILE="/etc/dai/config.toml"              # TAR SH             export DRIVERLESS_AI_CONFIG_FILE="/path/to/your/unpacked/dai/directory/config.toml"         2. Specify the following configuration options in the config.toml          file. Note that the procsy port, which defaults to 12347, also          has to be changed. ..           ::              # IP address and port of procsy process.

(jdbc_app_configs)             # hive: Hive Connector, remember to configure Hive below. (hive_app_configs)             # recipe_url: load custom recipe from URL             # recipe_file: load custom recipe from local file system             enabled_file_systems = "file, hdfs"        3. Save the changes when you are done, then stop/restart          Driverless AI. Example 2: Enable HDFS with Keytab-Based Authentication -------------------------------------------------------  **Notes**:  -  If using Kerberos Authentication, then the time on the Driverless AI    server must be in sync with Kerberos server. If the time difference    between clients and DCs are 5 minutes or higher, there will be    Kerberos failures. -  If running Driverless AI as a service, then the Kerberos keytab needs    to be owned by the Driverless AI user; otherwise Driverless AI will    not be able to read/access the Keytab and will result in a fallback    to simple authentication and, hence, fail.

-  Configures the environment variableDRIVERLESS_AI_HDFS_APP_PRINCIPAL_USERto reference a user for       whom the keytab was created (usually in the form of user@realm).

-  Configures the optionhdfs_app_prinicpal_userto reference          a user for whom the keytab was created (usually in the form of          user@realm). 1. Configure the Driverless AI config.toml file. Set the following          configuration options. Note that the procsy port, which          defaults to 12347, also has to be changed. ..           -enabled_file_systems
8080-hdfs_auth_type = "keytab"-key_tab_path =
"/tmp/<keytabname>"-hdfs_app_principal_user =
"<user@kerberosrealm>"2. Mount the config.toml file into the Docker container.

-  Configures the optionhdfs_app_prinicpal_userto reference          a user for whom the keytab was created (usually in the form of          user@realm). 1. Export the Driverless AI config.toml file or add it to          ~/.bashrc. For example:        ..           ::              # DEB and RPM             export DRIVERLESS_AI_CONFIG_FILE="/etc/dai/config.toml"              # TAR SH             export DRIVERLESS_AI_CONFIG_FILE="/path/to/your/unpacked/dai/directory/config.toml"         2. Specify the following configuration options in the config.toml          file. ..           ::              # IP address and port of procsy process.

(jdbc_app_configs)             # hive: Hive Connector, remember to configure Hive below. (hive_app_configs)             # recipe_url: load custom recipe from URL             # recipe_file: load custom recipe from local file system             enabled_file_systems = "file, hdfs"              # HDFS connector             # Auth type can be Principal/keytab/keytabPrincipal             # Specify HDFS Auth Type, allowed options are:             #   noauth : No authentication needed             #   principal : Authenticate with HDFS with a principal user             #   keytab : Authenticate with a Key tab (recommended)             #   keytabimpersonation : Login with impersonation using a keytab             hdfs_auth_type = "keytab"              # Path of the principal key tab file             key_tab_path = "/tmp/<keytabname>"              # Kerberos app principal user (recommended)             hdfs_app_principal_user = "<user@kerberosrealm>"        3.

Example 3: Enable HDFS with Keytab-Based Impersonation ------------------------------------------------------  **Notes**:  -  If using Kerberos, be sure that the Driverless AI time is synched    with the Kerberos server. -  If running Driverless AI as a service, then the Kerberos keytab needs    to be owned by the Driverless AI user. -  Logins are case sensitive when keytab-based impersonation is    configured. .. container:: tabs     .. group-tab:: Docker Image Installs     The example:     -  Sets the authentication type tokeytabimpersonation. -  Places keytabs in the/tmp/dtmpfolder on your machine and       provides the file path as described below. -  Configures theDRIVERLESS_AI_HDFS_APP_PRINCIPAL_USERvariable,       which references a user for whom the keytab was created (usually       in the form of user@realm).

-  Places keytabs in the/tmp/dtmpfolder on your machine and          provides the file path as described below. -  Configures thehdfs_app_principal_uservariable, which          references a user for whom the keytab was created (usually in          the form of user@realm). 1. Configure the Driverless AI config.toml file. Set the following          configuration options. Note that the procsy port, which          defaults to 12347, also has to be changed. ..           -enabled_file_systems
8080-hdfs_auth_type = "keytabimpersonation"-key_tab_path =

-  Places keytabs in the/tmp/dtmpfolder on your machine and          provides the file path as described below. -  Configures thehdfs_app_principal_uservariable, which          references a user for whom the keytab was created (usually in          the form of user@realm). 1. Export the Driverless AI config.toml file or add it to          ~/.bashrc. For example:        ..           ::              # DEB and RPM             export DRIVERLESS_AI_CONFIG_FILE="/etc/dai/config.toml"              # TAR SH             export DRIVERLESS_AI_CONFIG_FILE="/path/to/your/unpacked/dai/directory/config.toml"         2. Specify the following configuration options in the config.toml          file. ..           ::              # IP address and port of procsy process.

(jdbc_app_configs)             # hive: Hive Connector, remember to configure Hive below. (hive_app_configs)             # recipe_url: load custom recipe from URL             # recipe_file: load custom recipe from local file system             enabled_file_systems = "file, hdfs"              # HDFS connector             # Auth type can be Principal/keytab/keytabPrincipal             # Specify HDFS Auth Type, allowed options are:             #   noauth : No authentication needed             #   principal : Authenticate with HDFS with a principal user             #   keytab : Authenticate with a Key tab (recommended)             #   keytabimpersonation : Login with impersonation using a keytab             hdfs_auth_type = "keytabimpersonation"              # Path of the principal key tab file             key_tab_path = "/tmp/<keytabname>"              # Kerberos app principal user (recommended)             hdfs_app_principal_user = "<user@kerberosrealm>"        3.

Specifying a Hadoop Platform ----------------------------  The following example shows how to build an H2O-3 Hadoop image and run Driverless AI. This example uses CDH 6.0. Change theH2O_TARGETto specify a different platform. 1. Clone and then build H2O-3 for CDH 6.0. ..     .. code:: bash        git clone https://github.com/h2oai/h2o-3.git       cd h2o-3       ./gradlew clean build -x test       export H2O_TARGET=cdh6.0       export BUILD_HADOOP=true       ./gradlew clean build -x test  2. Start H2O.

Driverless AI can run on machines with only CPUs or machines with CPUs
and GPUs. For the best (and intended-as-designed) experience, install
Driverless AI on modern data center hardware with GPUs and CUDA support. Feature engineering and model building are primarily performed on CPU
and GPU respectively. For this reason, Driverless AI benefits from
multi-core CPUs with sufficient system memory and GPUs with sufficient
RAM. For best results, we recommend GPUs that use the Pascal or Volta
architectures. The older K80 and M60 GPUs available in EC2 are supported
and very convenient, but not as fast. Ampere-based NVIDIA GPUs are also
supported on x86, as Driverless AI ships with NVIDIA CUDA 11.2.2
toolkit. Image processing and NLP use cases in particular, benefit
significantly from GPU usage. For details, see gpu_in_dai. Driverless AI supports local, LDAP, and PAM authentication.

Refer to the dai_auth section for more information. Note that the default authentication method is "unvalidated." Driverless AI also supports HDFS, S3, Google Cloud Storage, Google Big
Query, KDB, MinIO, and Snowflake access. Support for these data sources
can be configured by setting environment variables for the data
connectors or via a config.toml file. Refer to the Data Connectors
section for more information. Sizing Requirements
Sizing Requirements for Native Installs
Driverless AI requires a minimum of 5 GB of system memory in order to
start experiments and a minimum of 5 GB of disk space in order to run a
small experiment. Note that these limits can changed in the config.toml
file. We recommend that you have sufficient system CPU memory (64 GB or
more) and 1 TB of free disk space available. Sizing Requirements for Docker Installs
For Docker installs, we recommend 1 TB of free disk space. Driverless AI
uses approximately 38 GB.

Once Driverless
AI runs, the mounts from the Docker container can point to other file
system mount points. GPU Sizing Requirements
If you are running Driverless AI with GPUs, ensure that your GPU has
compute capability >=3.5 and at least 4GB of RAM. If these requirements
are not met, then Driverless AI switches to CPU-only mode. Sizing Requirements for Storing Experiments
We recommend that your Driverless tmp directory has at least 500 GB to 1
TB of space. The (Driverless) tmp directory holds all experiments and
all datasets. We also recommend that you use SSDs (preferably NVMe). Virtual Memory Settings in Linux
If you are running Driverless AI on a Linux machine, we recommend
setting the overcommit memory to 0. The setting can be changed with the
    sudo sh -c "/bin/echo 0 > /proc/sys/vm/overcommit_memory"
This is the default value that indicates that the Linux kernel is free
to overcommit memory.

In the latter case, the memory requirements
of Driverless AI may surpass the memory allocation limit and prevent the
experiment from completing. If Redis runs out of memory, then background save may fail under low
memory condition with "0" set. The value "0" is not required to run DAI, but DAI might erroneously seem
to run out of memory even if plenty of memory is available. Large Pages
If Transparent Huge Pages (THP) support is enabled in your kernel,
avoiding huge page support is highly recommended. sudo sh -c "/bin/echo never > /sys/kernel/mm/transparent_hugepage/enabled"
Note that this isn't required to run DAI. However, if the preceding
command isn't used, older systems may run much slower when allocating
memory, and issues relating to latency and memory usage may arise with
Redis. Per-Experiment GPU memory and usage
If you're using Docker, you need to use --pid=host for NVIDIA to be able
to access per-process GPU usage for logging.

--shm-size=2g``

Without this option, those packages will fail. Triton inference server
also requires this option be set, and if under heavy load, may require
even larger values than 2g.

DOCKER RESOURCE LIMITS

DAI controls various resources and needs more resources than what
systems typically set by default. You can use the following option to
ensure that DAI is given enough resources:

--ulimit nofile=131071:131071 --ulimit nproc=16384:16384``

Without this option, DAI crashes under load.

DOCKER NICE

As stated in the official Docker documentation, the

--cap-add=SYS_NICEoption grants the container theCAP_SYS_NICEcapability, which lets the container raise processnicevalues, set real-time scheduling policies, set CPU affinity, and other operations. If this flag isn't passed when starting the container, DAI isn't able to control resources and can end up with all processes only using a single core. This is also required to use the built-in NVIDIA Triton Inference Server and its use of non-uniform memory access (NUMA) control. Memory Requirements per Experiment ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  As a rule of thumb, the memory requirement per experiment is approximately 5 to 10 times the size of the dataset. Dataset size can be estimated as the number of rows x columns x 4 bytes; if text is present in the data, then more bytes per element are needed. Backup Strategy ---------------  The **Driverless AI tmp** directory is used to store all experiment artifacts such as deployment artifacts and MLIs.

Note that no files should be added or deleted in the tmp folder outside of what Driverless AI adds automatically. We recommend periodically stopping Driverless AI and backing up the **Driverless AI** **tmp** directory to ensure that a copy of the Driverless AI state is available for instances where you may need to revert to a prior state. Upgrade Strategy ----------------  When upgrading Driverless AI, note that:  -  Image models from version 1.9.x aren't supported in 1.10.x. All other    models from 1.9.x are supported in 1.10.x. -  (**MLI**) Interpretations made in version 1.9.0 are supported in    1.9.x and later. -  (**MLI**) Interpretations made in version 1.8.x aren't supported in    1.9.x and later. However, interpretations made in 1.8.x can still be    viewed and rerun.

If an MLI job appears in the list of Interpreted Models       in your current version, then it is retained after upgrading. -  *Build MOJO pipelines*: Before upgrading, build MOJO pipelines on       all desired models. -  Stop Driverless AI and make a backup (copy) of the **Driverless       AI** **tmp** directory. The upgrade process inherits the service user and group from /etc/dai/User.conf and /etc/dai/Group.conf. You do not need to manually specify the DAI_USER or DAI_GROUP environment variables during an upgrade. **Note**: Driverless AI does not support data migration from a newer version to an older version. If you rollback to an older version of Driverless AI after upgrading, newer versions of the **master.db** file will not work with the older Driverless AI version. For this reason, we recommend saving a copy of the older 'tmp' directory to fully restore the older Driverless AI version's state.

ulimitoptions by using the--ulimitargument todocker
run. The following is an example of how to configure these options:  ::     --ulimit nproc=65535:65535 \    --ulimit nofile=4096:8192 \  Refer to https://docs.docker.com/engine/reference/commandline/run/#set-ulimits-in-container---ulimit for more information on these options. Note about nvidia-docker 1.0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~  If you have nvidia-docker 1.0 installed, you need to remove it and all existing GPU containers. Refer to https://github.com/NVIDIA/nvidia-docker/blob/master/README.md for more information. Deprecation ofnvidia-smi~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  Thenvidia-smi``
command has been deprecated by NVIDIA. Refer to
https://github.com/nvidia/nvidia-docker#upgrading-with-nvidia-docker2-deprecated
for more information. The installation steps have been updated for
enabling persistence mode for GPUs.

NVIDIA driver >=
471.68installed in your environment, for a seamless experience on all NVIDIA architectures, including Ampere. Go to `NVIDIA download driver <https://www.nvidia.com/Download/index.aspx>`__ to get the latest NVIDIA Tesla A/T/V/P/K series driver. For reference on CUDA Toolkit and Minimum Required Driver Versions and CUDA Toolkit and Corresponding Driver Versions, see `here <https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html>`__ . .. note::     If you are using K80 GPUs, the minimum required NVIDIA driver version    is 450.80.02. Note About Authentication ~~~~~~~~~~~~~~~~~~~~~~~~~  The default authentication setting in Driverless AI is "unvalidated." In this case, Driverless AI will accept any login and password combination, it will not validate whether the password is correct for the specified login ID, and it will connect to the system as the user specified in the login ID.

We recommend that you configure authentication. Driverless AI provides a number of authentication options, including LDAP, PAM, Local, and None. Refer to :ref:`dai_auth` for information on how to enable a different authentication method. **Note**: Driverless AI is also integrated with IBM Spectrum Conductor and supports authentication from Conductor. Contact sales@h2o.ai for more information about using IBM Spectrum Conductor authentication. Note About Shared File Systems ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  If your environment uses a shared file system, then you must set the following configuration option:  ::     datatable_strategy='write'  The above can be specified in the `config.toml file <config_toml.html#sample-config-toml-file>`__ (for native installs) or specified as an `environment variable <setting-environment-variables.html#setting-environment-variables-in-docker-images>`__ (Docker image installs).

MINIO SETUP

with MinIO. Note that unlike S3, authentication must also be configured
when the MinIO data connector is specified.

docker run
versionto check which version of Docker you are using. Description of Configuration Attributes ---------------------------------------  -minio_endpoint_url: The endpoint URL that will be used to access    MinIO. -minio_access_key_id: The MinIO access key. -minio_secret_access_key: The MinIO secret access key. -minio_skip_cert_verification: If this is set to true, then MinIO    connector will skip certificate verification. This is set to false by    default. -enabled_file_systems: The file systems you want to enable. This    must be configured in order for data connectors to function properly.

It also    configures Docker DNS by passing the name and IP of the name node. This lets you reference data stored in MinIO directly using the    endpoint URL, for example:    http://\ <endpoint_url>/<bucket>/datasets/iris.csv.

= "file, upload, minio"-minio_endpoint_url =
"<endpoint_url>"-minio_access_key_id =
"<access_key_id>"-minio_secret_access_key =
"<access_key>"-minio_skip_cert_verification = "false"``
  2. Mount the config.toml file into the Docker container. Native Installs
  This example enables the MinIO data connector with authentication by
  passing an endpoint URL, access key ID, and an access key. It also
  configures Docker DNS by passing the name and IP of the MinIO
  endpoint. This allows users to reference data stored in MinIO directly
  using the endpoint URL, for example: http://\
  <endpoint_url>/<bucket>/datasets/iris.csv. 1. Export the Driverless AI config.toml file or add it to ~/.bashrc. For example:
  2. Specify the following configuration options in the config.toml
      file. 3.

USING A CUSTOM TRANSFORMER

Driverless AI supports a number of feature transformers as described in
transformations. This example shows how you can include a custom
transformer in your experiment. Specifically, this example will show how
to add the ExpandingMean transformer. 1. Start an experiment in Driverless AI by selecting your training
    then specifying a Target Column. Notice the list of transformers
    that will be used in the FEATURE ENGINEERING SEARCH SPACE (WHERE
    APPLICABLE) section of the experiment summary. Driverless AI
    determines this list based on the dataset and experiment. 2. Click on EXPERT SETTINGS. 3. Specify the custom recipe using one of the following methods:
4. Navigate to the EXPERT SETTINGS > RECIPES tab and click the INCLUDE
    SPECIFIC TRANSFORMERS button.

DATA CONNECTORS

Driverless AI provides a number of data connectors for accessing
external data sources. The following data connection types are enabled
by default:
-   upload: The standard upload feature of Driverless AI. -   file: Local file system or server file system. -   hdfs: Hadoop file system. Remember to configure the HDFS config
    folder path and keytab. -   s3: Amazon S3. Optionally configure secret and access key. -   recipe_file: Custom recipe file upload. -   recipe_url: Custom recipe upload via URL.

-   To reference data on S3, use s3://. -   To reference data on HDFS, use the prefix hdfs://. -   To reference data on Azure Blob Store, use
    https://<storage_name>.blob.core.windows.net. -   To reference data on BlueData Datatap, use dtap://. -   To reference data on Google BigQuery, make sure you know the Google
    BigQuery dataset and the table that you want to query. Use a
    standard SQL query to ingest data. -   To reference data on Google Cloud Storage, use gs://
-   To reference data on kdb+, use the hostname and the port
    http://<kdb_server>:<port>
-   To reference data on MinIO, use http://<endpoint_url>. -   To reference data on Snowflake, use a standard SQL query to ingest
    data. -   To access a SQL database via JDBC, use a SQL query with the syntax
    associated with your database.

WINDOWS 10

This section describes how to install, start, stop, and upgrade
Driverless AI on a Windows 10 machine. The installation steps assume
that you have a license key for Driverless AI. For information on how to
obtain a license key for Driverless AI, visit
https://h2o.ai/o/try-driverless-ai/. Once obtained, you will be prompted
to paste the license key into the Driverless AI UI when you first log
in, or you can save it as a .sig file and place it in the license folder
that you will create during the installation process. Overview of Installation on Windows
To install Driverless AI on Windows, use a Driverless AI Docker image. NOTES:
-   GPU support is not available on Windows. -   Scoring is not available on Windows. CAUTION: Installing Driverless AI on Windows 10 is not recommended for
serious use. Environment
  Operating System        GPU Support?

Refer to
https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/hyper-v-requirements
for more information. Docker Image Installation
NOTES:
-   Be aware that there are known issues with Docker for Windows. More
    information is available here:
    https://github.com/docker/for-win/issues/188. -   Consult with your Windows System Admin if
    -   Your corporate environment does not allow third-part software
        installs
    -   You are running Windows Defender
    -   You your machine is not running with
        Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux. WATCH THE INSTALLATION VIDEO here. Note that some of the images in this
same.

    https://www.h2o.ai/download/. 2. Download, install, and run Docker for Windows from
    https://docs.docker.com/docker-for-windows/install/. You can verify
    that Docker is running by typing docker version in a terminal (such
    as Windows PowerShell). Note that you may have to reboot after
    installation. 3. Before running Driverless AI, you must:
4. Open a PowerShell terminal and set up a directory for the version of
    Driverless AI on the host machine:
5. With Docker running, navigate to the location of your downloaded
    Driverless AI image. Move the downloaded Driverless AI image to your
    new directory. 6. Change directories to the new directory, then load the image using
7. Set up the data, log, license, and tmp directories (within the new
    directory). 8. Copy data into the /DATA directory. The data will be visible inside
    the Docker container at /DATA.

Run docker images to find the image tag. 10. Start the Driverless AI Docker image. Be sure to replace path_to_
    below with the entire path to the location of the folders that you
    created (for example,
    "c:/Users/user-name/driverlessai_folder/data"). Note that this is
    regular Docker, not NVIDIA Docker. GPU support will not be
    available. Note that from version 1.10 DAI docker image runs with
    both are enabled in the launch command, tini prints a (harmless)
    warning message. 11. Connect to Driverless AI with your browser at
that were installed in a Docker container.

WARNING: Experiments, MLIs, and MOJOs reside in the Driverless AI tmp

MLI FOR REGULAR (NON-TIME-SERIES) EXPERIMENTS

This section describes MLI functionality and features for regular
experiments. Refer to interpret-ts for MLI information with time-series

interpret-a-model interpret-expert-settings
interpret-explainer-expert-settings interpret-understanding
viewing-explanations interpret-general-considerations

INTERPRETATION EXPERT SETTINGS

The following is a list of the Interpretation expert settings that are
available when setting up a new interpretation from the
MLI page <from-mli-page>. The name of each setting is preceded by its
config.toml <config_file> label. For info on explainer-specific expert
settings, see explainer-expert-settings.

-   interpretation-expert-settings-mli
-   interpretation-expert-settings-nlp
-   interpretation-expert-settings-surrogate

MLI Tab

mli_sample~~~~~~~~~~~~~~  .. container:: dropdown     **Sample All Explainers**     Specify whether to perform the interpretation on a sample of the    training data. By default, MLI will sample the training dataset if it    is greater than 100k rows. (The equivalent config.toml setting ismli_sample_size``.) This is enabled by default. Turn this toggle

    off to run MLI on the entire dataset.

MLI_ENABLE_MOJO_SCORER``

ALLOW USE OF MOJO SCORING PIPELINE

Use this option to disable MOJO scoring pipeline. Scoring pipeline is
chosen automatically (from MOJO and Python pipelines) by default. In
case of certain models, MOJO vs. Python choice can impact pipeline
performance and robustness.

MLI_FAST_APPROX``

Specify whether to speed up predictions with a fast approximation. When
enabled, this setting can reduce the number of trees or cross-validation
folds and ultimately reduce the time needed to complete interpretations.
This setting is enabled by default.

MLI_CUSTOM``

ADD TO CONFIG.TOML VIA TOML STRING

Use this input field to add to the Driverless AI server config.toml
configuration file with TOML string.

MLI NLP Tab

MLI_NLP_TOP_N``

NUMBER OF TOKENS USED FOR MLI NLP EXPLANATIONS

Specify the number of tokens used for MLI NLP explanations. To use all
available tokens, set this value to -1. By default, this value is set to
20.

MLI_NLP_SAMPLE_LIMIT``

SAMPLE SIZE FOR NLP SURROGATE MODELS

Specify the maximum number of records used by MLI NLP explainers. The
default value is 10000.

MLI_NLP_MIN_DF``

MINIMUM NUMBER OF DOCUMENTS IN WHICH TOKEN HAS TO APPEAR

Specify the minimum number of documents in which token has to appear.
Use integer values to denote absolute counts and floating-point values
to denote percentages. By default, this value is set to 3.

MLI_NLP_MAX_DF``

MAXIMUM NUMBER OF DOCUMENTS IN WHICH TOKEN HAS TO APPEAR

Specify the maximum number of documents in which token has to appear.

mli_nlp_min_ngram~~~~~~~~~~~~~~~~~~~~~  .. container:: dropdown     **Minimum Value in n-gram Range**     Specify the minimum value in the n-gram range. The tokenizer    generates all possible tokens in the range specified bymli_nlp_min_ngramandmli_nlp_max_ngram``. By default, this

    value is set to 1.

mli_nlp_max_ngram~~~~~~~~~~~~~~~~~~~~~  .. container:: dropdown     **Maximum Value in n-gram Range**     Specify the maximum value in the n-gram range. The tokenizer    generates all possible tokens in the range specified bymli_nlp_min_ngramandmli_nlp_max_ngram``. By default, this

MLI_NLP_MIN_TOKEN_MODE``

MODE USED TO CHOOSE N TOKENS FOR MLI NLP

Specify the mode used to choose N tokens. Select from the following:

-   TOP - Chooses N top tokens
-   BOTTOM - Chooses N bottom tokens
-   TOP-BOTTOM - Chooses math.floor (N/2) top and math.ceil (N/2) bottom
    tokens
-   LINSPACE - Chooses N evenly spaced tokens

MLI_NLP_TOKENIZER_MAX_FEATURES``

NUMBER OF TOP TOKENS TO USE AS FEATURES (TOKEN-BASED FEATURE IMPORTANCE)

Specify the number of top tokens to use as features when building
token-based feature importance. By default, this value is set to -1.

MLI_NLP_LOCO_MAX_FEATURES``

NUMBER OF TOP TOKENS TO USE AS FEATURES (LOCO)

Specify the number of top tokens to use as features when computing text
LOCO. By default, this value is set to -1.

MLI_NLP_SURROGATE_TOKENS``

NUMBER OF TOP TOKENS TO USE AS FEATURES (SURROGATE MODEL)

surrogate models. Note that this setting only applies to NLP models. By
default, this value is set to 100.

MLI_NLP_USE_STOP_WORDS``

STOP WORDS FOR MLI NLP

Specify whether to use stop words for MLI NLP. This setting is enabled

mli_nlp_stop_words~~~~~~~~~~~~~~~~~~~~~~  .. container:: dropdown     **List of Words to Filter Before Generating Text Tokens**     Specify a list of words to filter out before generating text tokens,    which are passed to MLI NLP LOCO and surrogate models (if enabled).    Enter a custom list of stop words. For example, you can enter['great',
'good']`` to filter out the words _great_ and _good_.

mli_nlp_append_to_english_stop_words~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  .. container:: dropdown     **Append List of Custom Stop Words to Default Stop Words**     Specify whether to append the list of stop words specified bymli_nlp_stop_words`` to the default list of stop words. This

    setting is disabled by default.

MLI SURROGATE MODELS TAB

mli_lime_method~~~~~~~~~~~~~~~~~~~  .. container:: dropdown     **LIME Method**     Select a LIME method of either K-LIME (default) or LIME-SUP. -  **K-LIME** (default): creates one global surrogate GLM on the       entire training data and also creates numerous local surrogate       GLMs on samples formed from *k*-means clusters in the training       data. The features used for *k*-means are selected from the Random       Forest surrogate model's variable importance. The number of       features used for *k*-means is the minimum of the top 25% of       variables from the Random Forest surrogate model's variable       importance and the max number of variables that can be used for       *k*-means, which is set by the user in the config.toml setting formli_max_number_cluster_vars. (Note, if the number of features       in the dataset are less than or equal to 6, then all features are       used for *k*-means clustering.)

All penalized GLM surrogates are trained to model the predictions of the Driverless AI model. The number of clusters for local explanations is chosen by a grid search in which the :math:`R2 between the Driverless AI model predictions and all of
    the local K-LIME model predictions is maximized. The global and
    local linear model’s intercepts, coefficients, R2 values, accuracy,
    and predictions can all be used to debug and develop explanations
    for the Driverless AI model’s behavior. - LIME-SUP: explains local regions of the trained Driverless AI
    model in terms of the original variables. Local regions are defined
    by each leaf node path of the decision tree surrogate model instead
    of simulated, perturbed observation samples - as in the original
    LIME. For each local region, a local GLM model is trained on the
    original inputs and the predictions of the Driverless AI model.

MLI_USE_RAW_FEATURES``

USE ORIGINAL FEATURES FOR SURROGATE MODELS

Specify whether to use original features or transformed features in the
surrogate model for the new interpretation. This is enabled by default.

MLI_VARS_TO_PDP``

NUMBER OF FEATURES FOR PARTIAL DEPENDENCE PLOT

Specify the maximum number of features to use when building the Partial
Dependence Plot. Use -1 to calculate Partial Dependence Plot for all
features. By default, this value is set to 10.

MLI_NFOLDS``

CROSS-VALIDATION FOLDS FOR SURROGATE MODELS

Specify the number of surrogate cross-validation folds to use (from 0 to
10). When running experiments, Driverless AI automatically splits the
training data and uses the validation data to determine the performance
of the model parameter tuning and feature engineering steps. For a new
interpretation, Driverless AI uses 3 cross-validation folds by default
for the interpretation.

MLI_QBIN_COUNT``

NUMBER OF COLUMNS TO BIN FOR SURROGATE MODELS

Specify the number of columns to bin for surrogate models. This value
defaults to 0.

SAMPLE SIZE FOR SURROGATE MODELS

When the number of rows is above this limit, sample for surrogate
models. The default value is 100000.

MLI_NUM_QUANTILES``

NUMBER OF BINS FOR QUANTILE BINNING

Specify the number of bins for quantile binning. By default, this value
is set to -10.

MLI_DIA_SAMPLE_SIZE``

When the number of rows is above this limit, sample for Disparate Impact
Analysis (DIA). The default value is 100000.

MLI_PD_SAMPLE_SIZE``

partial dependence plot. The default value is 25000.

MLI_PD_NUMCAT_NUM_CHART``

UNIQUE FEATURE VALUES COUNT DRIVEN PARTIAL DEPENDENCE PLOT BINNING AND
CHART SELECTION

mli_pd_numcat_threshold~~~~~~~~~~~~~~~~~~~~~~~~~~~  .. container:: dropdown     **Threshold for PD/ICE Binning and Chart Selection**     Ifmli_pd_numcat_num_chart`` is enabled, and if the number of

MLI_SA_SAMPLING_LIMIT``

KLIME_CLUSTER_COL``

K-LIME CLUSTERING COLUMNS

For k-LIME interpretations, optionally specify which columns to have
k-LIME clustering applied to.

NOTE: This setting is not found in the config.toml file.

For k-LIME interpretations, specify one or more columns to generate
decile bins (uniform distribution) to help with MLI accuracy. Columns
selected are added to top n columns for quantile binning selection. If a
column is not numeric or not in the dataset (transformed features), then
the column will be skipped.

UPGRADING THE DRIVERLESS AI IMAGE

WARNINGS:
It is not possible to upgrade from version 1.2.2 or earlier to the
latest version. You have to manually remove the 1.2.2 container and then
reinstall the latest Driverless AI version.

Upgrading from Version 1.3.0 to 1.5.1
1. SSH into the IP address of the image instance and copy the existing
    experiments to a backup location:
2.  wget the newer image. Replace VERSION and BUILD below with the
    Driverless AI version. 3. Use the docker load command to load the image:
4. Run docker images to find the new image tag. 5. Start the Driverless AI Docker image and replace TAG below with the
    Docker 19.03) command. Upgrading from version 1.5.2 or Later
Upgrading to versions 1.5.2 and later is no longer done via Docker. Instead, perform the following steps if you are upgrading to version
1.5.2 or later. Replace dai_NEWVERSION.deb below with the new Driverless
AI version (for example, dai_1.8.4.1_amd64.deb). Note that this upgrade
process inherits the service user and group from /etc/dai/User.conf and
/etc/dai/Group.conf.

USING THE MOJO SCORING PIPELINE WITH SPARK/SPARKLING WATER

NOTE: The Driverless AI 1.5 release will be the last release with
TOML-based MOJO2. Releases after 1.5 will include protobuf-based MOJO2. MOJO scoring pipeline artifacts can be used in Spark to deploy
predictions in parallel using the Sparkling Water API. This section
shows how to load and run predictions on the MOJO scoring pipeline in
Spark using Scala and the Python API. In the event that you upgrade H2O Driverless AI, we have a good news! Sparkling Water is backwards compatible with MOJO versions produced by
older Driverless AI versions. Requirements
-   You must have a Spark cluster with the Sparkling Water JAR file
    passed to Spark. -   To run with PySparkling, you must have the PySparkling zip file. The H2OContext does not have to be created if you only want to run
predictions on MOJOs using Spark. This is because the scoring is
independent of the H2O run-time.

This can be achieved via --jars argument of the Spark
launcher scripts. NOTE: In Local Spark mode, use --driver-class-path to specify path to
the license file. PySparkling
First, start PySpark with PySparkling Python package and Driverless AI
license. ./bin/pyspark --jars license.sig --py-files pysparkling.zip
or, you can download official Sparkling Water distribution from H2O
Download page. Follow the steps on the Sparkling Water download page. Once you are in the Sparkling Water directory, you can call:
    ./bin/pysparkling --jars license.sig
At this point, you should have available a PySpark interactive terminal
where you can try out predictions. If you would like to productionalize
the scoring process, you can use the same configuration, except instead
of using ./bin/pyspark, you would use ./bin/spark-submit to submit your
job to a cluster.

# If you want to use old behavior when all output columns were stored inside an array,
    # set it to False. However we strongly encourage users to use True which is defined as a default value. settings = H2OMOJOSettings(namedMojoOutputColumns = True)
    # Load the pipeline. 'settings' is an optional argument. If it's not specified, the default values are used. mojo = H2OMOJOPipelineModel.createFromMojo("file:///path/to/the/pipeline.mojo", settings)
    # Load the data as Spark's Data Frame
    dataFrame = spark.read.csv("file:///path/to/the/data.csv", header=True)
    # Run the predictions.

./bin/spark-submit`` to submit your job to a cluster. // First, specify the dependencies
    import ai.h2o.sparkling.ml.models. {H2OMOJOPipelineModel, H2OMOJOSettings}
    // The 'namedMojoOutputColumns' option ensures that the output columns are named properly. // If you want to use old behavior when all output columns were stored inside an array,
    // set it to false. However we strongly encourage users to use true which is defined as a default value. val settings = H2OMOJOSettings(namedMojoOutputColumns = true)
    // Load the pipeline. 'settings' is an optional argument. If it's not specified, the default values are used. val mojo = H2OMOJOPipelineModel.createFromMojo("file:///path/to/the/pipeline.mojo", settings)
    // Load the data as Spark's Data Frame
    val dataFrame = spark.read.option("header", "true").csv("file:///path/to/the/data.csv")
    // Run the predictions.

UI LANGUAGE

The Driverless AI UI is available in English (default), Japanese,
Chinese (Simplified), and Korean. This section describes how you can use
the app_language config setting/environment variable to change the
language of the UI before starting Driverless AI. When using app_language, the following options can be specified:
-   en: English (default)
-   ja: Japanese
-   cn: Chinese (Simplified)
-   ko: Korean

Examples
The following examples show how to change the app language from English
to Japanese. Docker Image Installs
To change the application language in Docker images, specify the
APP_LANGUAGE environment variable. Note that this variable must be
prepended with DRIVERLESS_AI_. Replace nvidia-docker with docker in the
example below if necessary.

VARIABLE IMPORTANCE IN DRIVERLESS AI

Global Feature Importance
-   Model Specific Feature Importance: After completion of an experiment
    Driverless AI, reports the variable importance that is model or
    algorithm specific. For example for Tree based models, this
    importance is gain based. i.e It computes the average reduction in
    impurity across all trees in the forest due to each feature. Features that tend to split nodes closer to the root of a tree have
    a larger importance value. For say an n fold model the variable
    importance is averaged across the folds, normalized and reported. For an ensemble model, the importance is multiplied by the
    respective model weights and normalized. -   Permutation Feature Importance: Permutation-based feature importance
    is a _model-agnostic_ approach.

If the feature has little predictive power, shuffling
    its values should have less impact on the model’s performance. If a
    feature is highly predictive, however, shuffling its values should
    decrease the model’s performance. ref. Driverless AI applies permutation based feature importance for
    upfront feature selection before genetic algorithm <ga> when the
    feature space is large. Local Feature Importance
-   LIME: Local interpretable model-agnostic explanations (LIME) is a
    _model agnostic_ technique aiming to explain which features are most
    important in specific areas of the feature space. The main idea of
    LIME is to compute a local surrogate model in the area of interest. This surrogate model is an easily interpretable model such as a
    linear model or a decision tree trained to mimic the behavior of the
    more complex model of interest.

By feeding these perturbed data points to
    the complex model a relation between the the perturbed features and
    the model prediction emerges which is then captured by the surrogate
    model. Driverless MLI builds LIME models <surrogate-tab> as part of model
    interpretation toolkit. -   Shapley: Shapley values can be used for local feature importance. They can be used to explain which feature(s) contribute most to a
    specific prediction, say fraud or not fraud. Shapley values are not
    designed to answer the "what if" questions that LIME’s local
    surrogate models are designed for. Shapely has its origin in game theory where the problem at hand is
    to determine a fair payoff for all players in the team based on
    their individual capabilities or performance. Shapley value is
    defined as an average expected marginal contribution of one player
    after all possible combinations have been considered.

MOJO2 JAVADOC

The downloaded MOJO.ZIP file contains the entire scoring pipeline.

IMAGE PROCESSING IN DRIVERLESS AI

Image processing in Driverless AI is a powerful tool that can be used to
gain insight from digital images. The following sections describe
Driverless AI's image processing capabilities. -   image-processing-supported-file-types
-   Uploading Image dataset <upload-image-data> to Driverless AI
-   Image Transformer <image-embeddings>: Use image transformers when a
    dataset contains both images and other feature types. -   Image Model <image-model>: Use an Image model when the only feature
    in the dataset is an image. -   Deploying an Image Model <deploy-image> to Production
Note
- Image models from Driverless AI version 1.9.x aren't supported in
1.10.x. - Image and NLP use cases in Driverless AI benefit significantly
from GPU usage. For more information, see GPU usage in DAI <gpu_in_dai>.

These Images are still compatible with Driverless AI and will
appear in the INSIGHTS column if DAI has insights to provide for them. Ideally Driverless AI can support all OpenCV Image formats. Uploading Data for Image Processing
Driverless AI supports multiple methods for uploading image datasets:
-   Archive with images in directories for each class. Labels for each
    class are automatically created based on directory hierarchy
-   Archive with images and a CSV file that contains at least one column
    with image names and a target column (best method for regression). Note that each image name must include the correct file extension. -   CSV file with local paths to the images on the disk
-   CSV file with remote URLs to the images
Modeling Images
Driverless AI features two different approaches to modeling images.

The resulting vector is then used for modeling in
Driverless AI. There are several options in the EXPERT SETTINGS panel that let you
configure the Image Vectorizer transformer. This panel is available from
within the experiment page above the Scorer knob. Refer to
image-settings for more information on these options. NOTES:
-   This modeling approach supports classification and regression
    experiments. -   This modeling approach supports the use of mixed data types (any
    number of image columns, text columns, numeric or categorical
    columns)
-   The Image Vectorizer transformer can also be enabled with the
    Pipeline Building Recipe <pipeline-building-recipe> expert setting,
    which is located in the Experiment tab. Automatic Image Model
Automatic Image Model is an AutoML model that accepts only an image and
a label as input features. This model automatically selects
hyperparameters such as learning rate, optimizer, batch size, and image
input size.

Automatic Image Model uses TensorFlow
ImageNet models <tensorflow_image_pretrained_models> and starts the
training process from them. The possible architectures list includes all
the well-known models: (SE)-ResNe(X)ts; DenseNets; EfficientNets; etc. Unique insights that provide information and sample images for the
current best individual model are available for Automatic Image Model. To view these insights, click on the INSIGHTS option while an experiment
is running or after an experiment is complete. Refer to image-insights
for more information. Each individual model score (together with the neural network
architecture name) is available in the Iteration Data panel. The last
point in the Iteration Data is always called ENSEMBLE. This indicates
that the final model ensembles multiple individual models.

The experiment preview
section updates to include information about Automatic Image Model:
[]
NOTES:
-   This modeling approach only supports a single image column as an
    input. -   This modeling approach does not support any transformers. -   This modeling approach supports classification and regression
    experiments. -   This modeling approach does not support the use of mixed data types
    because of its limitation on input features. -   This modeling approach does not use Genetic Algorithm <ga> (GA). -   The use of one or more GPUs is strongly recommended for this
    modeling approach. -   If an internet connection is available, ImageNet pretrained weights
    are downloaded automatically. If an internet connection is not
    available, weights must be downloaded from
    http://s3.amazonaws.com/artifacts.h2o.ai/releases/ai/h2o/pretrained/dai_image_models_1_10.zip
    and extracted into tensorflow_image_pretrained_models_dir
    (./pretrained/image/ by default).

DRIVERLESS AI SECURITY

Objective
This document describes different aspects of Driverless AI security and
provides guidelines to secure the system by reducing its surface of
vulnerability. This section covers the following areas of the product:
  -   security_user_access
      -   security_auth (Also see dai_auth)
      -   Authorization
  -   security_data
      -   security_data_import
      -   security_data_export
      -   security_logs
      -   security_data_isolation
  -   security_client_server
      -   security_response_headers
      -   security_recommended_headers
      -   security_other_headers
  -   security_web_ui
  -   security_custom_recipe
  -   security_config (Also see
      in depth documentation <configuration-security> on configuration
      security in DAI)

Important things to know
Warning
WARNING Security in a default installation of Driverless AI is DISABLED!

For production
environments, we recommend following this document and performing a
secure Driverless AI installation. ------------------------------------------------------------------------

User Access
Authentication
Driverless AI supports Client Certificate, LDAP, Local, mTLS, OpenID,
PAM, none, and unvalidated (default) authentication. These can be
configured by specifying the environment variables when starting the
Driverless AI Docker image or by specifying the appropriate
configuration options in the config.toml file. For more info, see
dai_auth.

method. authentic ation_defau lt_timeout_ hours   7 2                 Consult your security           Number of hours
                                                                requirements. after which a
                                                                                                user has to
                                                                                                relogin. --------------------------------------------------------------------------------------------------------------
mTLS Authentication
Driverless AI supports Mutual TLS authentication (mTLS) by setting a
specific verification mode along with a certificate authority file, an
SSL private key, and an SSL certificate file. For more information, see
the mtls_auth. Authorization Methods
Driverless AI does not currently perform any authorization.

available/configured
                                                                                           data sources. ma x_fi le_u ploa d_si ze   1048 576000 00B                Configure based on expected   Limit maximum size of
                                                             file size and size of         uploaded file. Driverless AI deployment. su ppor ted_ file _typ es   see config .toml               It is recommended to limit    Supported file formats
                                                             file types to extension used  listed in filesystem
                                                             in the target environment     browsers. (e.g., parquet).

It is
                                                                                           recommended to show
                                                                                           only configured data
                                                                                           sources.

Note: if dataset download is
                                                                        disabled, we strongly suggest to
                                                                        disable custom recipes as well to
                                                                        remove another way how data could
                                                                        be exported from the application. enable_art ifacts_uploa d           `fa lse `        false            Replace all downloads on the
                                                                        experiment page to "exports", and
                                                                        allow users to push to the artifact
                                                                        store configured with
                                                                        artifacts_store. (See notes below.)

(See notes below.) artifacts_ file_system_ directory   tm p             tmp              File system location where
                                                                        artifacts will be copied in case
                                                                        artifacts_store is set to
                                                                        file_system. (See notes below.) ---------------------------------------------------------------------------------------------------------
NOTES ABOUT ARTIFACTS:
-   Currently, file_system is the only option that can be specified for
    artifacts_store. Additional options will be available in future
    releases. -   The location for artifacts_file_system_directory is expected to be a
    directory on your server. -   When these artifacts are enabled/configured, the menu options on the
    completed_experiment page change.

Refer to
    export_artifacts for more information. Logs
The Driverless AI produces several logs:
  -   audit logs
  -   server logs
  -   experiment logs
The administrator of Driverless AI application (i.e., person who is
responsible for configuration and setup of the application) has control
over content which is written to the logs. -------------------------------------------------------------------------------------------------------
  Option                                      De fa   Recomme   Description
                                              ul t Va nded      
                                              lu e    Value     
  ------------------------------------------- ------- --------- -----------------------------------------
  audit_lo g_retentio n_period                ` 5 `   0 (disabl Number of days to keep audit logs. The
                                              (d ay   e audit   value 0 disable rotation.

.t om l           
  log_leve l                                  ` 1 `   see       Define verbosity of logging
                                                      config. toml      
  collect_ server_log s_in_exper iment_logs   ` fa ls false     Dump server logs with experiment. e `               Dangerous because server logs can contain
                                                                information about experiments of other
                                                                users using Driverless AI. h2o_reci pes_log_le vel                     No ne   ---       Log level for OSS H2O instances used by
                                                                custom recipes. debug_lo g                                  ` fa ls false     Enable debug logs. e `               
  write_re cipes_to_e xperiment_ logger       ` fa ls false     Dump a custom recipe source code into
                                              e `               logs.

It is       |
| ta_dire |   |                      | recommended to hide it to        |
|  ctory  | r |                      | protect data_directory from      |
|         |   |                      | browsing and corruption. |
|         | u |                      |                                  |
|         |   |                      |                                  |
|         | e |                      |                                  |
|         |   |                      |                                  |
+---------+---+----------------------+----------------------------------+
| file_ p |   | true                 | Enable path filter for           |
| ath_fi  | f |                      | file-system browser (file data   |
| ltering |   |                      | source).

|
|         |   |                      |                                  |
|         | s |                      |                                  |
|         |   |                      |                                  |
|         | e |                      |                                  |
|         |   |                      |                                  |
| file_ p |   | Include a list of    | List of absolute path prefixes   |
| ath_fi  | [ | folder paths or      | to restrict access to in         |
| lter_in |   | {{DAI_USERNAME}} for | file-browser. |
|  clude  | ] | the logged user's    |                                  |
|         |   | directory.

|                                  |
| autod o |   | ""                   | Directory where Driverless AI    |
| c_addi  | " |                      | searches for the updated AutoDoc |
| tional_ |   |                      | templates. Providing empty value |
|  templa | " |                      | "" disables this functionality.

pem"   Correct private key. Private key to setup
                                                                           HTTPS/SSL
                                                                           communication. ssl_ crt_fi le     "/etc/dai/ cert.pem"           Correct public         Public certificate to
                                                    certifikate. setup HTTPS/SSL. ssl_ no_ssl v2     true                           true                   Prevents an SSLv2
                                                                           connection. ssl_ no_ssl v3     true                           true                   Prevents an SSLv3
                                                                           connection. ssl_ no_tls v1     true                           true                   Prevents an TLSv1
                                                                           connectiona.

ssl_ no_tls v1_2   false                          false (disable TLSv1.2 Prevents a TLSv1.2
                                                    only if TLSv1.3 is     connection. available)             
  ssl_ no_tls v1_3   false                          false                  Prevents a TLSv1.3
                                                                           connection. -----------------------------------------------------------------------------------------------
HTTP Cookie Attributes
By default, HTTP cookies used by Driverless AI are issued with the
following attributes:
  -   HTTPOnly: True
  -   SameSite: Lax
If either of these needs to be overridden, or if more custom attributes
need to be set, you can use the config http_cookie_attributes to specify
key-value pairs of so-called cookie morsels. For a list of supported
keys, see the official Python documentation.

|                                |                  |
+------+-----------+--------------------------------+------------------+
| Co n | Content   | default-src https: ; font-sr c | https://develope |
| ten  | Security  |  'self'; script-src 'self' 'u  | r.mozilla.org/en |
| t-Se | Policy    | nsafe-eval' 'unsafe-inline'; s | -US/docs/Web/HTT |
|  cur | (CSP) is  |  tyle-src 'self' 'unsafe-inlin | P/Headers/Conten |
| i ty | an added  | e '; object-src 'none'         | t-Security-Polic |
| -P o | layer of  |                                | y                |
| lic  | security  | Note: The Driverless AI is     | https://develope |
| y    | that      | still requires to have         | r.mozilla.org/en |
|      | helps to  | unsafe-eval and unsafe-inline  | -US/docs/Web/HTT |
|      | detect    | configured, which              | P/CSP            |
|      | and       | potentionally makes the server | https://infosec. |
|      | mitigate  | vulnerable to XSS attacks.

|                                |                  |
|      | Controls  |                                |                  |
|      | from      |                                |                  |
|      | where the |                                |                  |
|      | page can  |                                |                  |
|      | download  |                                |                  |
|      | source.

|                                |                  |
|      | The value |                                |                  |
|      | here      |                                |                  |
|      | overrides |                                |                  |
|      | the       |                                |                  |
|      | default,  |                                |                  |
|      | which is  |                                |                  |
|      | SAMEORI G |                                |                  |
|      | IN.

|                                |                  |
Other Headers to Consider
  ------------------------------------------------------------------------
  Header             Documentation
  ------------------ -----------------------------------------------------
  Public-Key-P ins   https://developer.mozilla.org/en-US/docs/Web/HTTP/P
  CORS-related       ublic_Key_Pinning
  headers            https://developer.mozilla.org/en-US/docs/Web/HTTP/C
                     ORS
------------------------------------------------------------------------

Web UI Security
Note
The Driverless AI UI is design to be user-friendly, and by default all
features like auto-complete are enabled. Disabling the user-friendly
features increases security of the application, but impacts
user-friendliness and usability of the application.

allow_loc alstorage           tr ue   fals e   Disable use of web browser local
                                                 storage. show_all_ filesystems         tr ue   fals e   Show all available data sources in
                                                 WebUI (even though there are not
                                                 configured). It is recommended to show
                                                 only configured data sources. verify_se ssion_ip            `fa lse true     Verifies each request IP against IP
                                `                which initialized the session. allow_con current_ses sions   tr ue   fals e   Disable concurrent sessions (logins). enable_xs rf_protecti on      tr ue   true     Enable XSRF (cross-site request
                                                 forgery) protection. enable_se cure_cookie s       `fa lse true     Enable SECURE cookie flag.

-------------------------------------------------------------------------------------

Custom Recipe Security
Note
By default Driverless AI enables custom recipes as a main route for the
way data-science teams can extend the application capabilities. In
enterprise environments, it is recommended to follow best software
engineering practices for development of custom recipes (i.e., code
reviews, testing, stage releasing, etc.) and bundle only a pre-defined
and approved set of custom Driverless AI extensions.

enable_custom_reci pes_upload               tru e    false     Enable uploading of custom
                                                                 recipes. enable_custom_reci pes_from_url             tru e    false     Enable downloading of custom
                                                                 recipes from external URL. include_custom_rec ipes_by_default          fal se   false     Include custom recipes in
                                                                 default inclusion lists.

Due to that, you may not be able to score
new datasets with those experiments and you'll need to re-upload the
recipes again. It is still recommended to ENABLE this options for
security reasons. ------------------------------------------------------------------------

Configuration Security
Driverless AI provides the option to store sensitive or secure
configuration information in an encrypted keystore as an alternative to
keeping security settings as clear text in the config.toml file. Encrypting sensitive information is highly recommended. For more
information, see configuration-security. Baseline Secure Configuration
The following Driverless AI configuration is an example of secure
configuration. Ensure that all necessary config options are specified. For more information, see config_file.

IMPORTING DATASETS

Supported file types
-   arff
-   avro
-   bin
-   bz2
-   dat
-   feather
-   gz
-   tgz
-   tsv
-   txt
-   xls
-   xlsx
-   xz
-   zip
Note

Adding datasets
GB. or
    them from the enabled_file_systems setting in the config.toml file. (Refer to Using the config.toml file section for more information.)

-   If Driverless AI was started with data connectors enabled for Azure
    appear in the ADD DATASET (OR DRAG & DROP) dropdown menu. Refer to
    the Enabling Data Connectors section for more information. -   When specifying to add a dataset using DATA RECIPE URL, the URL must
    the same columns.

EXPERIMENT SUMMARY

An experiment summary is available for each completed experiment. Click
the DOWNLOAD SUMMARY & LOGS button to download the
H2OAI_EXPERIMENT_SUMMARY_<EXPERIMENT>.ZIP file. []
The files within the experiment summary zip provide textual explanations
of the graphical representations that are shown on the Driverless AI UI. Details of each artifact are described below. Experiment AutoDoc
A report file (AUTODOC) is included in the experiment summary. This
report provides insight into the training data and any detected shifts
in distribution, the validation schema selected, model parameter tuning,
feature evolution and the final set of features chosen during the
experiment. For more information, see autodoc. Experiment Artifacts Overview
The Experiment Summary contains artifacts that provide overviews of the
experiment. -   PREVIEW.TXT: Provides a preview of the experiment.

-   SUMMARY: Provides the same summary that appears in the lower-right
    portion of the UI for the experiment. (Available in txt or json.) -   CONFIG.JSON: Provides a list of the settings used in the experiment. -   CONFIG_OVERRIDES_TOML_STRING.TXT: Provides any overrides for this
    experiment that were made to the config.toml file. -   ARGS_DO_AUTO_DL.JSON: The internal arguments used in the Driverless
    AI experiment based on the dataset and accuracy, time and
    interpretability settings. -   EXPERIMENT_COLUMN_TYPES.JSON: Provides the column types for each
    column included in the experiment. -   EXPERIMENT_ORIGINAL_COLUMN.JSON: A list of all columns available in
    the dataset that was used in the experiment. -   EXPERIMENT_PIPELINE_ORIGINAL_REQUIRED_COLUMNS.JSON: For columns used
    in the experiment, this includes the column name and type. -   EXPERIMENT_SAMPLING_DESCRIPTION.JSON: A description of the sampling
    performed on the dataset.

Tuning Artifacts
During the Driverless AI experiment, model tuning is performed to
determined the optimal algorithm and parameter settings for the provided
dataset. For regression problems, target tuning is also performed to
determine the best way to represent the target column (i.e. does taking
the log of the target column improve results). The results from these
tuning steps are available in the Experiment Summary. -   TUNING_LEADERBOARD: A table of the model tuning performed along with
    the score generated from the model and training time. (Available in
    txt or json.) -   TARGET_TRANSFORM_TUNING_LEADERBOARD.TXT: A table of the transforms
    applied to the target column along with the score generated from the
    model and training time. (This will be empty for binary and
    multiclass use cases.) Features Artifacts
Driverless AI performs feature engineering on the dataset to determine
the optimal representation of the data.

The complete list of features used
in the final model is available in the Experiment Summary artifacts. The Experiment Summary also provides a list of the original features and
their estimated feature importance. For example, given the features in
the final Driverless AI model, we can estimate the feature importance of
the original features. ----------------------------------------------------------------
    Feature                                    Feature Importance
    ------------------------------------------ ---------------------
    NumToCatWoE:PAY_AMT2                       1
    PAY_3                                      0.92
    ClusterDist9:BILL_AMT1:LIMIT_BAL:PAY_3     0.90
    ----------------------------------------------------------------
To calculate the feature importance of PAY_3, we can aggregate the
feature importance for all variables that used PAY_3:
-   NumToCatWoE:PAY_AMT2: 1 * 0 (PAY_3 not used.) -   PAY_3: 0.92 * 1 (PAY_3 is the only variable used.)

Estimated Feature Importance = (1*0) + (0.92*1) + (0.9*(1/3)) = 1.22
NOTE: The feature importance is converted to relative feature
importance. (The feature with the highest estimated feature importance
will have a relative feature importance of 1). -   ENSEMBLE_FEATURES: A list of features used in the final model, a
    description of the feature, and the relative feature importance. Feature importances for multiple models are linearly blended with
    same weights as the final ensemble of models. (Available in txt,
    table, or json.) -   ENSEMBLE_FEATURES_ORIG: A complete list of all original features
    used in the final model, a description of the feature, the relative
    feature importance, and the standard deviation of relative
    importance. (Available in txt or json.)

(Available in txt or json.) -   ENSEMBLE_FEATURES_PREFIT: A list of features used by the best
    individuals in the final population, each model blended with same
    weights as ensemble if ensemble used blending. (Available in txt,
    table, or json.) -   ENSEMBLE_FEATURES_SHIFT: A list of features used in the final model
    and the difference in relative feature importance between the final
    model and the corresponding feature importance of the final
    population. (Available in txt, table, or json.) -   FEATURES: A list of features used by the best individual pipeline
    (identified by the genetic algorithm) and each feature's relative
    importance. (Available in txt, table, or json.) -   FEATURES_ORIG: A list of original user features used by the best
    individual pipeline (identified by the genetic algorithm) and each
    feature's estimated relative importance. (Available in txt or json.)

(Available in txt, table, or json.) -   LEAKAGE_FEATURES_ORIG.JSON: A list of leaked original features
    provided and an estimate of the relative feature importance of that
    leaked original feature in the final model. -   SHIFT_FEATURES.JSON: A list of all features provided along with the
    relative importance and the shift in standard deviation of relative
    importance of that feature. -   SHIFIT_FEATURES_ORIG.JSON: A list of original features provided and
    an estimate of the shift in relative feature importance of that
    original feature in the final model. Final Model Artifacts
The Experiment Summary includes artifacts that describe the final model. This is the model that is used to score new datasets and create the MOJO
scoring pipeline. The final model may be an ensemble of models depending
on the Accuracy setting. -   COEFS: A list of coefficients and standard deviation of coefficients
    for features. (Available in txt or json.)

-   ENSEMBLE_BASE_LEARNER_FOLD_SCORES: The internal validation scorer
    metrics for each base learner when the final model is an ensemble. (Available in table or json.) Note that this is not available for
    Time Series experiments. -   ENSEMBLE_DESCRIPTION.TXT: A sentence describing the final model. (For example: "Final TensorFlowModel pipeline with ensemble_level=0
    transforming 21 original features -> 54 features in each of 1 models
    each fit on full training data (i.e. no hold-out).") -   ENSEMBLE_COEFS: The coefficient and standard deviation coefficient
    for each feature in the ensemble. (Available as txt or json.) -   ENSEMBLE_COEFS_SHIFT: The coefficient and shift of coefficient for
    each feature in the ensemble. (Available as txt or json.) -   ENSEMBLE_MODEL_DESCRIPTION.JSON/ENSEMBLE_MODEL_EXTRA_DESCRIPTION: A
    json file describing the model(s) and for ensembles how the model
    predictions are weighted.

-   ENSEMBLE_FOLDS_DATA.JSON: A json file describing the folds used for
    the final model(s). This includes the size of each fold of data and
    the performance of the final model on each fold. (Available if a
    fold column was specified.) -   ENSEMBLE_FEATURES_ORIG: A list of the original features provided and
    an estimate of the relative feature importance of that original
    feature in the ensemble of models. (Available in txt or json.) -   ENSEMBLE_FEATURES: A complete list of all features used in the final
    ensemble of models, a description of the feature, and the relative
    feature importance. (Available in txt, table, or json.) -   LEAKAGE_COEFS.JSON: A list of coefficients and standard deviation of
    coefficients for leaked features. -   PIPELINE: A visual representation of the experiment pipeline. -   SHIFT_COEFS.JSON: A list of coefficients and the shift in standard
    deviation for those coefficients used in the experiment.

-   ENSEMBLE_SCORES.JSON: The scores of the final model for our list of
    scorers. -   ENSEMBLE_CONFUSION_MATRIX_TEST: The confusion matrix for the test
    data if test data is provided. Note that this is not available for
    Time Series experiments. -   ENSEMBLE_CONFUSION_MATRIX_WITH_VALIDATION: The confusion matrix for
    the internal validation data. Note that this is not available for
    Time Series experiments. -   ENSEMBLE_CONFUSION_MATRIX_STATS_VALIDATION: The confusion matrix
    statistics on internal validation data. Note that this is not
    available for Time Series experiments. -   ENSEMBLE_CONFUSION_MATRIX_STATS_TEST.JSON: Confusion matrix
    statistics on the test data. This is only available if test data is
    provided. Note that this is not available for Time Series
    experiments. -   ENSEMBLE_GAINS_TEST: The lift and gains table for test data if test
    data is provided. (Visualization of lift and gains can be seen in
    the UI.)

-   ENSEMBLE_GAINS_WITH_VALIDATION: The lift and gains table for the
    internal validation data. (Visualization of lift and gains can be
    seen in the UI.) Note that this is not available for Time Series
    experiments. -   ENSEMBLE_ROC_TEST: The ROC and Precision Recall table for test data
    if test data is provided. (Visualization of ROC and Precision Recall
    curve can be seen in the UI.) Note that this is not available for
    Time Series experiments. -   ENSEMBLE_ROC_WITH_VALIDATION: The ROC and Precision Recall table for
    the internal validation data. (Visualization of ROC and Precision
    Recall curve can be seen in the UI.) Note that this is not available
    for Time Series experiments. -   FS_NORMALIZED_VARIMP: The normalized frequency variable importance
    values. (Available in table or json.) Note that this is not
    available for Time Series experiments. -   FS_UNNORMALIZED_VARIMP: The unnormalized frequency variable
    importance values.

IMPUTATION IN DRIVERLESS AI

The impute feature lets you fill in missing values with substituted
values. Missing values can be imputed based on the column's mean,
median, minimum, maximum, or mode value. You can also impute based on a
specific percentile or by a constant value.

The imputation is precomputed on all data or inside the pipeline (based
on what's in the train split).

The following guidelines should be followed when performing imputation:

-   For constant imputation on numeric columns, constant must be
    numeric.
-   For constant imputation on string columns, constant must be a
    string.
-   For percentile imputation, the percentage value must be between 0
    and 100.

NOTES:

-   This feature is experimental.
-   Time columns cannot be imputed.

Enabling Imputation

Imputation is disabled by default. It can be enabled by setting

enable_imputation=truein the config.toml (for native installs) or via theDRIVERLESS_AI_ENABLE_IMPUTATION=true``
environment variable (Docker image installs). This enables imputation
functionality in transformers.

RUNNING AN EXPERIMENT WITH IMPUTATION

Once imputation is enabled, you will have the option when running an
experiment to add imputation columns.

1.  Click on COLUMNS IMPUTATION in the Experiment Setup page.

2.  Click on ADD IMPUTATION in the upper-right corner.
3.  Select the column that contains missing values you want to impute.
4.  Select the imputation type. Available options are:

5.  Optionally allow Driverless AI to compute the imputation value
    during validation instead of using the inputted imputed value.
6.  Click SAVE when you are done.

7.  At this point, you can add additional imputations, delete the
    imputation you just created, or close this form and return to the
    experiment. Note that each column can have only a single imputation.

INTERNAL VALIDATION TECHNIQUE

This section describes the technique behind internal validation in
Driverless AI. For the experiment, Driverless AI will either:
(1) split the data into a training set and internal validation set
(2) use cross validation to split the data into n folds
Driverless AI chooses the method based on the size of the data and the
Accuracy setting. For method 1, part of the data is removed to be used
for internal validation. (NOTE: This train and internal validation split
may be repeated if the data is small so that more data can be used for
training.) For method 2, however, no data is wasted for internal validation. With
cross validation, the whole dataset is utilized, and each model is
trained on a different subset of the training data. The following
visualization shows an example of cross validation with 5 folds. []
Driverless AI randomly splits the data into the specified number of
folds for cross validation.

Driverless AI will not automatically create the internal validation data
randomly if a user provides a Fold Column or a Validation Dataset. If a
Fold Column or a Validation Dataset is provided, Driverless AI will use
that data to calculate the performance of the Driverless AI models and
to calculate all performance graphs and statistics. (NOTE: The fold and
weight columns are not used as features in modeling.) If the experiment is a Time Series use case, and a Time Column is
selected, Driverless AI will change the way the internal validation data
is created. In the case of temporal data, it is important to train on
historical data and validate on more recent data. Driverless AI does not
perform random splits, but instead respects the temporal nature of the
data to prevent any data leakage. In addition, the train/validation
split is a function of the time gap between train and test as well as
the forecast horizon (amount of time periods to predict).

SCORING ON ANOTHER DATASET

After you generate a model, you can use that model to make predictions
on another dataset.

1.  Click the EXPERIMENTS link in the top menu and select the experiment
    that you want to use.
2.  On the completed Experiment page, click MODEL ACTIONS > PREDICT.
3.  Select the new dataset (test set) that you want to score on. Note
    that this new dataset must include the same columns as the dataset
    used in selected experiment.
4.  Select the columns from the test set to include in the predictions
    frame.
5.  Click DONE to start the scoring process.
6.  Click the DOWNLOAD PREDICTIONS button after scoring is complete.

NOTE: This feature runs batch scoring on a new dataset. You may notice
slow speeds if you attempt to perform single-row scoring.

AUTODOC SETTINGS

This section includes settings that can be used to configure AutoDoc.

MAKE_AUTOREPORT``

MAKE AUTODOC

Specify whether to create an AutoDoc for the experiment after it has
finished running. This is enabled by default.

Specify a name for the AutoDoc report. This is set to "report" by
default.

Specify a path for the AutoDoc template:

-   To generate a custom AutoDoc template, specify the full path to your
    custom template.
-   To generate the standard AutoDoc, specify the default value for this
    setting.

AUTODOC_OUTPUT_TYPE``

AUTODOC FILE OUTPUT TYPE

Specify the AutoDoc output type. Choose from the following file types:

-   docx (Default)
-   md

AUTODOC SUBTEMPLATE TYPE

-   md
-   docx

AUTODOC_PD_MAX_ROWS``

PDP MAX NUMBER OF ROWS

Specify the number of rows for Partial Dependence Plots.

take when generating a report. Set this value to -1 to disable the time
limit. This is set to 20 seconds by default.

GENETIC ALGORITHM IN DRIVERLESS AI

Driverless AI aims to determine the BEST PIPELINE for a dataset. This
involves data transformation, feature engineering, model hyperparameter
tuning, scoring and ensembling. The GENETIC ALGORITHM process is a trial-and-error selection process,
but it is reproducible. In Driverless AI,
genetic algorithm <enable_genetic_algorithm> is performed during the
Feature Evolution stage <full_pic> of an experiment. Feature Evolution
is a competition between slowly mutating parameters to find best
individuals <ga_dai>. The FEATURE EVOLUTION is not completely random and
is informed from the variable importance <vi_in_dai> interactions tables
of the modeling algorithms. Driverless AI Brain <feature_brain1> caches
information about the set of best genes, interactions and parameters in
the population and also information from previous experiments (if
enabled), can be used during genetic algorithm mutations.

It can be
enabled for model hyperparameter tuning in combination with genetic
algorithm for feature engineering. CUSTOM CODE can also be written to
toggle inbuilt mutation strategy. For details see
additional information<some_details> section. During model building and feature tuning processes, _overfitting_ is
prevented by doing bootstrapping and cross validation, while
_underfitting_ is prevented by balancing exploitation vs exploration in
genetic algorithm. -   Understanding Genetic Algorithm <ga_dai> and its Driverless AI
    equivalent. -   The Full Picture <full_pic> : The end to end pipeline in Driverless
    AI. -   Reading the logs <read_the_log> : Workflow as seen in the Experiment
    logs. -   Some additional details <some_details>

Understanding Genetic Algorithm
Genetic Algorithm is a search heuristic inspired by the process of
natural selection where the fittest individuals are selected to produce
offspring for the next generation.

It is conceptually like
      the real gene where information is present to make say a protein. -   A transformer is the actual code that applies the gene. -   An individual consists of a genome that includes a set of _genes_,
      i.e. information about which transformations and with what
      parameters to perform. It also includes _model hyperparameters_
      and some _additional information_ like the target transformations
      applied etc. -   Individuals create a population that goes through a randomly
      chosen pair-wise tournament process <tournament_style> to decide
      the winners. -   Fitness score for an individual is model evaluation or scores
      based on the scoring metric. Below are the steps involved in a Genetic Algorithm and their Driverless
AI equivalent:
INITIALIZATION
Consider all the probable solutions to the given problem. This creates
the population. The most popular technique for initialization is the use
of random binary strings.

FITNESS ASSIGNMENT
Assign a fitness score to every individual, which further determines the
probability of being chosen for reproduction. The higher the fitness
score, the higher the chances of being chosen for reproduction. Driverless AI : Fitness score for an individual is model evaluation
based on the scoring metric. SELECTION
Individuals are selected for the reproduction of offspring. The selected
individuals are then arranged in pairs of two to enhance reproduction. These individuals pass on their genes to the next generation. The
genetic algorithm uses the fitness proportionate selection technique to
ensure that useful solutions are used for recombination. Driverless AI : A tournament <tournament_style> is performed within the
population to find the best subset (half) of the population. REPRODUCTION : CROSSOVER MUTATION
This phase involves the creation of a child population. The algorithm
employs variation operators that are applied to the parent population.

mutation : This operator adds new genetic information to the new child
  population. This is achieved by flipping some bits in the chromosome. Mutation solves the problem of local minimum and enhances
  diversification. crossover : This operator swaps the genetic information of two parents
  to reproduce an offspring. It is performed on parent pairs that are
  selected randomly to generate a child population of equal size as the
  parent population. Driverless AI : Winning sub population's genes, features and model
hyperparameters are mutated into new offspring (asexual reproduction). Mutation <mutation_mode> involves _adding, perturbing, or pruning_
genes <ga_dai>. The strategy for adding genes is based on balancing exploitation and
  exploration of importance of original variables. Genes are added that
  explore additional transformations for original variables with high
  importance.

Specific output features can be pruned. Features are pruned when
  variable importance is below a certain threshold (based upon
  interpretability settings). Genes are pruned based on the Information
  Gain Variable Importance for most models (i.e model specific variable
  importance). For some like CUML RF, it is based upon Shapley
  Permutation Importance. REPLACEMENT
Generational replacement takes place in this phase, which is a
replacement of the old population with the new child population. The new
population consists of higher fitness scores than the old population,
Driverless AI : Mutate winning sub-population's Genes (add, prune and
perturb), Features, Model hyper parameters to fill-up the population
back to pre-tournament size. TERMINATION
After replacement has been done, a stopping criterion is used to provide
the basis for termination. The algorithm will terminate after the
threshold fitness solution has been attained.

Driverless AI: Score the individuals and either terminate the evolution
if stopping criteria is reached or continue the selection process. The Full Picture
Here we describe in details the working of the different stages that
Driverless performs in sequence during an experiment to output the best
pipeline for the dataset-
1)  Convert Accuracy, Time and Interpretabilty knob <ati_knobs> settings
    to number of iterations and models to be built. 2)  Do TARGET TUNING: Find the best representation of the target
    variable for the problem at hand. This is achieved by building
    (LightGBM if available) models with simple allowed feature
    transformations and model parameters (chosen from the internal
    recipe pool) and choosing the target transformation with highest
    score. The target_transform_tuning_leaderboard_simple.json file in
    summary zip or Experiment GUI lists the built models with their
    scores and parameters.

Then, a simple
          model is built on each feature with significant variable
          importance. The models with high AUC (for classification) or
          R2 score (regression) are reported to the user as potential
          leak features. B)  Shift Detection <check_distribution_shift>: To detect shift in
          distribution between the training, validation or testing
          datasets, Driverless AI trains a binomial model to predict
          which dataset a row belongs to. Say, if the model built using
          only a specific feature as predictor, is able to separate the
          training and testing data with a high accuracy say an AUC of
          0.9, this indicates that there is a drift in the distribution
          of that feature in the training and testing data. Shifted
          features should either be dropped. Or more meaningful
          aggregate features be created by using them as labels/bins.

4)  Model and Feature TUNING PHASE: Tuning is random selection of
    parameters to find best individuals <ga_dai>. A)  Driverless creates a diverse set of individuals. First, it
          goes through and creates a "SEQUENCE" of models (based on
          allowed algorithms), adding them with simple feature
          transformations and model parameters. These allowed algorithms
          and feature transformations are displayed in the preview of
          the experiment. The DEFAULT includes simple genes like
          original numeric, date, tfidf or bert embeddings for text
          data, Target encodings, Frequency encodings, Weight of
          evidence encodings, clustering, interactions, etc. These
          default features are simple and support MOJO creation.

Then, if more individuals are needed in the population,
          "RANDOM" models are added. These have same model types
          (algorithms) as in SEQUENCE but with mutated parameters calls
          to the model to get _random hyper parameters and (default +
          extra) random features_. A "GLM ONE HOT ENCODED" model is evaluated and if seem to be
          performing well on the dataset, is added as an individual. A _reference individual_ "CONSTANT MODEL" is added to the mix,
          so that we know what best constant predictions (predict the
          same thing whatever the input data) would give for a score. This is how a DIVERSE POPULATION OF INDIVIDUALS is created.

c)  Every scored individual has gone through feature
                transformations, training the model, prediction making,
                and metric scoring. C)  Then a tournament <tournament_style> is performed amongst the
          individuals to get the best individuals to be passed on to the
          _evolution phase_. D)  An "EXTRA_FS" model is added in case "FS" strategy (feature
          selection strategy) is chosen ( for high interpretability
          settings) and it _replaces_ one of the above non-reference
          individuals. This special individual has features that are
          pre-pruned based on the permutation importance <vi_in_dai> of
          the dataset. The Tuning stage leaderboard of an experiment lists all the wining
    individuals (i.e models that scored highest during the tournament). The summary zip artifact includes it as the
    tuning_leaderboard_simple.json or txt file.

During
    evolution phase, we start off with the best individuals (highest
    score) from the _tuning phase_. The tuning phase may have created
    more or less individuals than required in the population. So first
    step is to either prune or add new individuals to create the desired
    population size. The evolution_begin_leaderboard_simple.json file
    lists these individuals (the unscored are the new added individuals
    to bring the population to the right size). A)  Every iteration of the experiment, each individual creates a
          new model based on its genes. B)  Population of individuals is trained on the training data,
          with early stopping if available. C)  Population is scored for given metric, with bootstrapping if
          chosen (default).

In the genetic algorithm, Mutation involves
          adding, pruning, or perturbing genes. Specific output features
          can also be pruned. The strategy for adding genes is based on
          balancing exploitation and exploration of importance of
          original variables. Genes are added that explore additional
          transformations for original variables with high importance. Genes are pruned based on the Information Gain Variable
          Importance for most models, for some like CUML RF, it is based
          upon Shapley Permutation Importance. Features are pruned when
          variable importance is below a certain threshold (based upon
          interpretability settings). See also
          Mutation strategies <mutation_mode>. F)  Back to A...
6)  Ensembling and Final Scoring Pipeline creation: Ensemble the final
    models and build Final Pipeline for production with a MOJO and/or
    Python scoring pipelines <deployment>.

Feature cost is not equal to the
      number of features used in the model but is based on their
      complexity (or interpretability) i.e. transformation
      used/interactions applied/time taken. For example a low cost model
      may have greater number of more interpretable features than a high
      cost model (i.e. cost number != number of feature used). This
      parameter is used in the workflow during genetic algorithm to
      decide if need to reduce feature count given interpretability dial
      settings of the experiment. -   Certain individuals in the _Evolution Begin leaderboard table_ are
      unscored. This can happen if:
        -   They violated some constraint on feature counts imposed for
            given choice of interpretability settings and so were
            changed, and the score no longer applies.

-   Also see additional details<some_details>. Reading the Logs
The Experiment preview gives an estimate of the number of iterations
done and the total number of models(including cross validation models)
that are built during the various stages of the experiment. For example
(from logs) -
Click here <exp_log.txt> to download and view the sample log file used
in this example.

The listed model count
estimate considers each nfold model as a count. As this is a regression problem, target tuning is performed and 18
models are created to decide the best
target transformation <target_transformer> for the dataset. This create
3 models with 3 fold cross validation each with 2 repeats, i.e two
different views of the dataset (in train/valid split). This is done in
two iterations. Next 4 iterations are be used for model and feature parameter tuning. This involves creation of approximately 8*3*2
(individuals*folds*repeats) ~ 48 models. The output models from tuning stage undergo Feature Evolution by genetic
algorithm. The genetic algorithm is performed on 8 individuals
(population size). The next 10 iterations are used for feature evolution
and around (10 * 8/2[population subset] * (3*2) (foldcv*repeats) ~240
new models are scored. The upper limit to it is 528 models. Early
stopping is performed if the scores do not improve after 5 iterations.

These estimates are based on Accuracy/Time/Interpretabilty dial
settings, types of models selected, and other expert settings for the
experiment. Then the transformer allowed search space for the experiment and per
feature is mapped out.

The dataset is split internally to create
cross validation folds for model building and tuning. In this example,
Feature evolution stage will require 3 folds for cross validation and
and two repeats i.e data views are done. The for final pipeline will
also perform 3 folds cv. After splitting the datasets in to folds for
internal validations, a Kolmogorov-Smirnov statistics is calculated to
see if the folds have similar distribution of data.

INFO   | Transformers used (accuracy=7, time=2, interpretability=8, num_classes=1): ['InteractionsTransformer', 'OriginalTransformer']
    INFO   | Genes used (accuracy=7, time=2, interpretability=8, num_classes=1): ['InteractionsGene', 'OriginalGene']
    INFO   | Tournament style: model
BACKEND TUNING job is run to make sure that everything is working as
expected. INFO   | Auto-tuning modeling backend: start. INFO   | Backend candidate Job# 0 Name: LightGBMModel using GPU (if applicable) with Booster: lightgbm
    INFO   | Backend candidate Job# 1 Name: LightGBMModel using CPU with Booster: lightgbm
    ...
    INFO   | Auto-tuning modeling backend: end : Duration: 299.8936 s
LEAKAGE DETECTION A model is run to determine the predictive power of
each feature on the target. Then, a simple model is built on each
feature with significant variable importance. The models with high AUC
(for classification) or R2 score (regression) are reported to the user
as potential leak.

of the target variable
to optimize for scorer So 3 models with 6 fold cross validation in 2
iterations. Each model tries a different target transformation. INFO   | Tuned 18/18 target transform tuning models. Tuned [LIGHTGBM] Tuning []
    INFO   | Target transform search: end : Duration: 389.6202 s
    INFO   | Target transform: TargetTransformer_identity_noclip
PARAMETER AND FEATURE TUNING STAGE starts from 3rd iteration and 4
iterations are spent in building ~48 models (8*3*2). 8 Individuals are built and made sure that the features included in the
models satisfy the interpretablity conditions (see nfeatures_max and
ngenes_max). Also an additional FS individual is added during the 6th
iteration. See tuning phase <full_pic> for reference. Hence this stage
builds greater than 48 models.

INFO   | Submitted    2 and Completed    2 non-identity feature engineering tasks out of   12 total tasks (including   10 identity)
    INFO   | Transforming FS valid
    INFO   | Submitted    2 and Completed    2 non-identity feature engineering tasks out of   12 total tasks (including   10 identity)
    INFO   | 10 features created during FS Permute
    INFO   | Using model LIGHTGBM for FS
The FEATURE EVOLUTION using genetic algorithm starts from the 7th
iteration and in each iteration for the next 10 iterations, scores
approximately (8/2)(subpopulation) *(3*2) =24 new models. At the end of the 16th iteration, the experiment has not converged so
the Feature evolution is stopped. It is made sure that the features
included in the models satisfy the interpretablity conditions and are
less than the maximum allowed limits (see nfeatures_max and ngenes_max). Best individual and population is stored in the Driverless AI brain for
restart or refitting of the experiment.

After sampling expected population size: 1. INFO   | Final population size after sampling: 1 (0 reference) with models_final=3 and num_ensemble_folds=3
    INFO   | Final Model sampled population with population of 8 individuals (best scores=['1.540669'])
In iteration 17, three fold cross validation is performed on the final
ENSEMBLE model, a few checks are done on the features used, predictions
and python and mojo SCORING PIPELINES are created. Logs and summary
ARTIFACTS are collected. INFO   | Completed 3/3 final ensemble models.

Reducing number of features for ALL MODELS is applicable only when
      (one of below satisfied):
        -   num. of columns, is greater than max_orig_cols_selected or,
        -   num of non-numeric columns, is greater than
            max_orig_nonnumeric_cols_selected or,
        -   num. of numeric columns, is greater than
            max_orig_numeric_cols_selected
      Given the above requirements for all models is not satisfied;
      reducing number of features only for the FS INDIVIDUAL (EXTRA_FS)
      is applicable only when (one of below satisfied) :
        -   num. of columns, is greater than fs_orig_cols_selected or,
        -   num. of non-numeric columns, is greater than
            fs_orig_numeric_cols_selected or,
            fs_orig_nonnumeric_cols_selected
    See tuning phase <full_pic> and permutation importance <vi_in_dai>.

This set is simple
          and supports MOJO creation for productionization of pipeline. -   model_origin as RANDOM allows both features and model
          hyper-parameters to call their mutate lists or functions. -   model_origin as EXTRA_FS is for the extra individuals added
          through Feature Selection(FS) based on permutation importance. -   model_origin as REF# denotes for reference individuals
          provided as a baseline(eg. ConstantModel). -   model_origin as GLM_OHE denotes features generated by GLM +
          OHE. 3)  Driverless AI Brain: During an experiment building, Brain caches the
    best iterations, parameters, models, genes and populations. These
    are used for informed lookups, cross overs during mutation,
    restarts <checkpointing> and refits <retrain> of experiment. For
    details see feature_brain_level <feature_brain1>.

-   Batched mode tends to do multiple types of the same
          transformation together. -   Full mode does even more types of the same transformation
          together. 5)  Mutation via custom recipe: Users can control and specify their own
    mutation strategy and the list of parameters to mutate on, by
    writing their own custom python code and hooking it up with the
    inbuilt Driverless AI Genetic Algorithm. Here is an example of such
    a recipe. The get_one function passes on the list of values to
    genetic algorithm or Optuna for that parameter. Reach out to
    support@h2o.ai if need more help with writing your own
    custom recipies <custom-recipes>. 6)  Optuna: Driverless AI supports OPTUNA for model hyperparameter
    tuning during the Tuning phase <full_pic> of an experiment. Optuna
    employs a Bayesian optimization algorithm called Tree-structured
    Parzen Estimator for hyperparameter optimization.

MTLS AUTHENTICATION EXAMPLE

SSL private key, and an SSL certificate file. The diagram below is a
visual representation of the mTLS authentication process. []

Description of Configuration Attributes
Use the following configuration options to configure mTLS. -   ssl_client_verify_mode: Sets the client verification mode. Choose
    certificate file, provided by your organization. This certificate
    will be used to verify the client certificate when client
    authentication is enabled. If this is not specified, clients are
    verified using the default system certificates. -   ssl_key_file: Specifies your web server private key file. This is
    normally created by your organization's sys admin.

This is normally created by your organization's sys admin. -   ssl_client_key_file: Required if
    ssl_client_verify_mode = "CERT_REQUIRED". Specifies the private key
    file that Driverless AI uses to authenticate itself. This is
    normally created by your organization's sys admin. -   ssl_client_crt_file: Required if
    ssl_client_verify_mode = "CERT_REQUIRED". Specifies the private
    client certificate file that Driverless AI will use to authenticate
    itself. This is normally created by your organization's sys admin. -   auth_tls_crl_file: Specifies the path to the certificate revocation
    list file that will be used to verify the client certificate. This
    file contains a list of revoked user IDs. Configuration Scenarios
The table below describes user certificate behavior for mTLS
authentication based on combinations of the configuration options
described above.

The | validated    |
|                    |              | certificate is   | and the      |
| AND                |              | used for         | revoked file |
|                    |              | connecting to    | is provided  |
| authentication_m e |              | the Driverless   | in           |
| thod='tls_authent  |              | AI server as     | AUTH_TLS_C R |
| ication'           |              | well as for      | L_FILE       |
|                    |              | authentication. |              |
+--------------------+--------------+------------------+--------------+

Enabling mTLS Authentication
To enable mTLS authentication in Docker images, specify the
authentication environment variable that you want to use. Each variable
must be prepended with DRIVERLESS_AI.

APPENDIX A: THIRD-PARTY INTEGRATIONS

H2O Driverless AI integrates with a (continuously growing) number of
third-party products. Please contact sales@h2o.ai to schedule a
discussion with one of our Solution Engineers for more information. If you are interested in a product not yet listed here, please ask us
about it!

APPENDIX C: INSTALLED COMPONENTS AND DEPENDENCIES

H2O Driverless AI is an artificial intelligence (AI) platform that
automates some of the most difficult data science and machine learning
workflows such as feature engineering, model validation, model tuning,
model selection and model deployment. It aims to achieve highest
predictive accuracy, comparable to expert data scientists, but in much
shorter time thanks to end-to-end automation. Driverless AI also offers
automatic visualizations and machine learning interpretability (MLI). Especially in regulated industries, model transparency and explanation
are just as important as predictive performance. This section describes components that included with the Driverless AI
Docker image and information on additional Driverless AI dependencies.

DATATABLE-0.2.2+MASTER.501.NOOMP-CP38-CP38-LINUX_X86_64.WHL:
data.table is one of the add-on packages for the R programming language. It provides a high-performance version of base R's data.frame with
syntax and feature enhancements for ease of use, convenience, and
programming speed. H2O4GPU-0.2.0+MASTER.B1EF476-CP38-CP38-LINUX_X86_64.WHL: H2O4GPU is a
collection of GPU solvers provided by H2Oai with APIs in Python and R.
The Python API builds upon the easy-to-use scikit-learn API and its
well-tested CPU-based algorithms. It can be used as a drop-in
replacement for scikit-learn (i.e. import h2o4gpu as sklearn) with
support for GPUs on selected (and ever-growing) algorithms. H2O4GPU
inherits all the existing scikit-learn algorithms and falls back to CPU
algorithms when the GPU algorithm does not support an important existing
scikit-learn class option. The R package is a wrapper around the H2O4GPU
Python package, and the interface follows standard R conventions for
modeling.

Python and Other Dependencies for Driverless AI
PYTHON 3.6: Python is a programming language that lets you work more
quickly and integrate your systems more effectively. PYTHON-DATEUTIL: The dateutil module provides powerful extensions to the
standard datetime module, available in Python. PYCRYPTO 2.6.1: The Python Cryptography Toolkit (pycrypto) is a
collection of both secure hash functions (such as SHA256 and RIPEMD160)
and various encryption algorithms (AES, DES, RSA, ElGamal, etc.). The
package is structured to make adding new modules easy. This section is
essentially complete, and the software interface will almost certainly
not change in an incompatible way in the future; all that remains to be
done is to fix any bugs that show up. If you encounter a bug, please
report it in the Launchpad bug tracker.

NUMPY 1.14.0 NumPy is the fundamental package for scientific computing
with Python. It contains among other components:
  -   A powerful N-dimensional array object
  -   Sophisticated (broadcasting) functions
  -   Tools for integrating C/C++ and Fortran code
  -   Useful linear algebra, Fourier transform, and random number
      capabilities
  Besides its obvious scientific uses, NumPy can also be used as an
  efficient multi-dimensional container of generic data. Arbitrary
  data-types can be defined. This allows NumPy to seamlessly and
  speedily integrate with a wide variety of databases. NumPy is licensed
  under the BSD license, enabling reuse with few restrictions. PANDAS 0.22.0: The Python Data Analysis Library, pandas is an open
source, BSD-licensed library providing high-performance, easy-to-use
data structures and data analysis tools for the Python programming
language.

There's no need to manually
add query strings to your URLs or to form-encode your POST data. Keep-alive and HTTP connection pooling are 100% automatic, thanks
to urllib3. SCIKIT-LEARN 0.19.1: Simple and efficient tools for data mining and data
analysis, accessible to everybody, and reusable in various contexts. scikit-learn is built on NumPy, SciPy, and matplotlib open source,
commercially usable BSD license. SCIPY 1.0.0: SciPy (pronounced “Sigh Pie”) is a Python-based ecosystem
of open-source software for mathematics, science, and engineering. SETPROCTITLE 1.1.10: The setproctitle module allows a process to change
its title (as displayed by system tools such as ps and top). Changing
the title is mostly useful in multi-process systems, for example when a
master process is forked: changing the children’s title allows to
identify the task each process is busy with. The technique is used
by PostgreSQL and the OpenSSH Server for example.

An extensive list of result statistics are available for
each estimator. The results are tested against existing statistical
packages to ensure that they are correct. The package is released under
the open source Modified BSD (3-clause) license. TOML 0.9.3.1: This is a Python library for parsing and creating TOML. The module passes the TOML test suite which is a fork of BurntSushi’s
TOML test suite. TOML is a configuration file format that is easy to
read due to obvious semantics and aims to be "minimal". TOML is designed
to map unambiguously to a dictionary. CLANG: Python bindings for clang from clang release branches
CLANG+LLVM-4.0.0-X86_64-LINUX-GNU-UBUNTU-16.04/ CLANG: The LLVM compiler
infrastructure supports a wide range of projects, from industrial
strength compilers to specialized JIT applications to small research
projects. APT-GET: This is a tool to automatically update your Debian machine and
get and install debian packages/programs.

CURL: PycURL is a Python interface to libcurl, the multiprotocol file
transfer library. Similar to the urllib Python module, PycURL can be
used to fetch objects identified by a URL from a Python program. Beyond
simple fetches however PycURL exposes most of the functionality of
libcurl. APT-UTILS: A package management related utility program. This package
contains some less used command line utilities related to package
management with APT. PYTHON-SOFTWARE-PROPERTIES: This manages the repositories that you
install software from (universe). SOFTWARE-PROPERTIES-COMMON: This manages the repositories that you
install software from (common). IPUTILS-PING: The iputils package is set of small useful utilities for
Linux networking. WGET: GNU Wget is a free software package for retrieving files using
HTTP, HTTPS, FTP and FTPS - the most widely-used Internet protocols.

CPIO: GNU cpio copies files into or out of a cpio or tar archive. The
archive can be another file on the disk, a magnetic tape, or a pipe. GNU
cpio supports the following archive formats: binary, old ASCII, new
ASCII, crc, HPUX binary, HPUX old ASCII, old tar, and POSIX.1 tar. The
tar format is provided for compatibility with the tar program. By
default, cpio creates binary format archives, for compatibility with
older cpio programs. When extracting from archives, cpio automatically
recognizes which kind of archive it is reading and can read archives
created on machines with a different byte-order. NET-TOOLS: A collection of programs that form the base set of the NET-3
networking distribution for the Linux operating system. This package
includes arp(8), hostname(1), ifconfig(8), ipmaddr, iptunnel,
mii-tool(8), nameif(8), netstat(8), plipconfig(8), rarp(8), route(8) and
slattach(8).

ZIP: zip is a compression and file packaging utility for Unix, VMS,
MSDOS, OS/2, Windows 9x/NT/XP, Minix, Atari, Macintosh, Amiga, and Acorn
RISC OS. It is analogous to a combination of the Unix commands tar(1)
and compress(1) and is compatible with PKZIP (Phil Katz's ZIP for MSDOS
systems). DIRMNGR: Dirmngr is a server for managing and downloading certificate
revocation lists (CRLs) for X.509 certificates and for downloading the
certificates themselves. Dirmngr also handles OCSP requests as an
alternative to CRLs. Dirmngr is either invoked internally by gpgsm (from
GnuPG 2) or when running as a system daemon through
the dirmngr-client tool. CURL -SL HTTPS://DEB.NODESOURCE.COM/SETUP_15.X | BASH - &&: This
repository contains the source of
the NodeSource Node.js and io.js Binary Distributions setup and support
scripts. NODEJS: Node.js is a JavaScript runtime built on Chrome's V8 JavaScript
engine.

The node.js package ecosystem, npm, is the
largest ecosystem of open source libraries in the world. BUILD-ESSENTIAL: An informational list of build-essential packages. CCACHE: ccache is a compiler cache. It speeds up recompilation by
caching previous compilations and detecting when the same compilation is
being done again. Supported languages are C, C++, Objective-C and
Objective-C++. ccache is free software, released under the GNU General
Public License version 3 or later. LIBOPENBLAS-DEV: Optimized BLAS (linear algebra) library (development
files)
PBZIP2: PBZIP2 is a parallel implementation of the bzip2 block-sorting
file compressor that uses pthreads and achieves near-linear speedup on
SMP machines. The output of this version is fully compatible with bzip2
v1.0.2 or newer (ie: anything compressed with pbzip2 can be decompressed
with bzip2). PBZIP2 should work on any system that has a pthreads
compatible C++ compiler (such as gcc).

PIP: pip is a package management system used to install and manage
software packages written in Python. Python 2.7.9 and later (on the
python2 series), and Python 3.4 and later include pip (pip3
for Python 3) by default. pip is a recursive acronym that can stand for
either "Pip Installs Packages" or "Pip Installs Python". SETUPTOOLS: Allows you to easily download, build, install, upgrade, and
uninstall Python packages. TENSORFLOW-GPU: An open source machine learning framework for numerical
computation using data flow graphs. PSUTIL: psutil (process and system utilities) is a cross-platform
library for retrieving information on running processes and system
utilization (CPU, memory, disks, network, sensors) in Python. It is
useful mainly for system monitoring, profiling and limiting process
resources and management of running processes.

DASK MULTINODE TRAINING (ALPHA)

Driverless AI can be configured to run in a multinode worker mode where
each worker has a Dask CPU worker and (if the worker has GPUs) a Dask
GPU worker. The main node in this setup has a Dask scheduler. This
document describes the Dask training process and how to configure it. Before setting up Dask multinode training, you must configure
Redis Multinode training in Driverless AI <redis-multinode-training>. NOTE: For Dask multinode examples, see
Dask Multinode examples <multinode-example>. Understanding Dask Multinode Training
Dask multinode training in Driverless AI can be used to run a single
experiment that trains across the multinode cluster. It is EFFECTIVE IN
SITUATIONS where you need to run and complete A SINGLE EXPERIMENT WITH
LARGE AMOUNTS OF DATA or A LARGE HYPER-PARAMETER SPACE SEARCH. The Dask
distributed MACHINES CAN BE CPU ONLY OR CPU + GPU, with Dask experiments
using resources accordingly.

NOTES:
-   Dask multinode training in Driverless AI is currently in a preview
    stage. If you are interested in using Dask multinode configurations,
    contact support@h2o.ai. -   Dask multinode training requires the transfer of data between
    several different workers. For example, if an experiment uses the
    Dask cluster, it must distribute data among cluster workers to be
    trained by XGBoost or Optuna hyper-parameter search. -   Dask tasks are scheduled on a first in, first out (FIFO) basis. -   Users can enable Dask multinode training on a per-experiment basis
    from the expert settings. -   If an experiment chooses to use the Dask cluster (default is true if
    applicable), then A SINGLE EXPERIMENT RUNS ON THE ENTIRE MULTINODE
    CLUSTER. For this reason, using a large number of commodity-grade
    hardware is not useful in the context of Dask multinode.

lightgbm_listen_port.  Edit the Driverless AI config.toml ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  After Driverless AI is installed, edit the following config option in the config.toml file.  .. code:: bash     # Dask settings -- set the IP address of the Dask server. Same as the IP of the main Driverless AI node, and usually same as the Redis/MinIO IP    dask_server_ip = "<host_ip>"  For thedask_server_ipparameter, Driverless AI automatically tries the Redis, MinIO, and local IP addresses to see if it can find the Dask scheduler. In such a case, thedask_server_ip``
parameter does not have to be set.

On EC2 systems, if the main server is

http://ec2-52-71-252-183.compute-1.amazonaws.com:12345/``, it is
recommended to use the nslookup-resolved IP instead of the EC2 IP due to
the way Dask and XGBoost (with rabit) operate. For example,

10.10.4.103. Redis, MinIO, and Dask subsequently use that as the IP in the config.toml file. Ifdask_server_ipis not specified, its value is automatically inferred from Redis or MinIO. Once the worker node starts, use the Driverless AI server IP and Dask dashboard port(s) to view the status of the Dask cluster. .. figure:: images/dask_dashboard.png    :alt:   Description of Configuration Attributes ---------------------------------------  General Dask Settings ~~~~~~~~~~~~~~~~~~~~~  -enable_dask_cluster: Specifies whether to enable a Dask worker on    each multinode worker. -dask_server_ip: IP address used by server for Dask and Dask CUDA    communications. CPU Cluster Dask Settings ~~~~~~~~~~~~~~~~~~~~~~~~~  -dask_server_port: Port used by server for Dask communications. -dask_dashboard_port: Dask dashboard port for Dask diagnostics. -dask_cluster_kwargs: Set Dask CUDA/RAPIDS cluster settings for    single node workers. -dask_scheduler_env: Set Dask scheduler env.

-dask_worker_env: Set Dask worker environment variables. -dask_worker_options: Set Dask worker command-line options. -dask_protocol: Protocol used for Dask communications. -dask_worker_nprocs: Number of processes per Dask worker. -dask_worker_nthreads: Number of threads per process for Dask. GPU CUDA Cluster Dask Settings ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  -dask_cuda_server_port: Port using by server for Dask cuda    communications. -dask_cuda_dashboard_port: Dask dashboard port for dask_cuda    diagnostics. -dask_cuda_cluster_kwargs: Set Dask CUDA/RAPIDS cluster settings    for single node workers. -dask_cuda_scheduler_env: Set Dask CUDA scheduler env. -dask_cuda_scheduler_options: Set Dask CUDA scheduler command-line    options. -dask_cuda_worker_options: Set Dask CUDA worker options. -dask_cuda_worker_env: Set Dask CUDA worker environment variables. -dask_cuda_protocol: Protocol using for dask cuda communications.

Other Cluster Dask Settings ~~~~~~~~~~~~~~~~~~~~~~~~~~~  -lightgbm_listen_port: LightGBM local listening port when using    Dask with LightGBM. Other Related Settings ~~~~~~~~~~~~~~~~~~~~~~  -show_warnings_preview: Show warnings in preview if could use dask    but did not choose relevant model  Example: User Experiment Dask Settings --------------------------------------  To enable XGBoost GBM on a Dask GPU cluster:  1) Set :ref:`num_gpus_for_prediction <num-gpus-for-prediction>` to    number of GPUs on each worker node 2) Go to expert settings and under Recipes tab, push    :ref:`Include specific models <included_models>` and ensure only    selected model is **XGBoostGBMDaskModel**  For more configuration options, see :ref:`user-experiment-dask-settings`. **Notes**:  -  The same steps can be used for a local Dask cluster on a single node    with multiple GPUs.

-  If have Dask cluster or single dask node available as single user,    one can set :ref:`exclusive_mode <exclusive_mode>` to "max" in expert    settings to maximize usage of workers in cluster. User Experiment Dask Settings ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  -use_dask_cluster: Whether to use Dask cluster (True) or only    local cluster for multi-GPU case (False). -enable_xgboost_rapids:    :ref:`Enable RAPIDS-cudf extensions to XGBoost GBM/Dart. <enable_xgboost_rapids>`    (1) -enable_xgboost_gbm_dask:    :ref:`Enable dask_cudf (multi-GPU) XGBoost GBM. <enable_xgboost_gbm_dask>`    (2) -enable_lightgbm_dask:    :ref:`Enable Dask (multi-node) LightGBM. <enable_lightgbm_dask>`    (*Experimental*) (2) -enable_xgboost_dart_dask:    :ref:`Enable dask_cudf (multi-GPU) XGBoost Dart. <enable_xgboost_dart_dask>`    (2) -enable_hyperopt_dask`: :ref:`Enable dask (multi-node/multi-GPU) hyperparameter search <enable_hyperopt_dask>.

SIMPLE CONFIGURATIONS

Below is a list of some simple configurations that can be run with
copy/paste config.toml settings in Driverless AI GUI. Get a quick Final Model: no Genetic Algorithm no Ensembling
These settings can be copy pasted in the Toml editor in the Expert
Settings. The experiment preview can be checked to make sure the changes
have taken effect. The Toml editor of a completed experiment will also
list them at the end of the experiment. Toml editor
    enable_genetic_algorithm = "off"
    fixed_ensemble_level = 0

Use Original Features With Genetic Algorithm
This example does no transformations on numeric features and only a
single simple encoding on categorical features, i.e. _no interactions,
target-encoding, dates, text, etc_. It only does model selection and
tuning via GA. The examples can be copy pasted in the Toml editor in the Expert
Settings. The experiment preview gets modified and can be inspected to
confirm the changes have taken effect.

It does not drop any features form the original set but
    you may still not see some features if the modeling algorithm does
    not use it say for splitting while fitting the model)
      Toml editor
          included_transformers = ["OriginalTransformer","FrequentTransformer"]
          max_feature_interaction_depth = 1
          no_drop_features = true
2)  This applies identity transformer on numeric features and
    OneHotEncoding transformation on categorical features. Toml editor
          included_transformers = ["OriginalTransformer","OneHotEncodingTransformer"]

Build models with your choice of algorithm and parameters
These settings can be copy pasted in the
Add to config.toml via toml string under the Expert Experiment settings
of an experiment. Always check the Driverless preview to make sure the
changes have taken effect before launching the experiment.

1)  This example builds a single GBM model with 2 folds cross
      validation and user provided parameters with no genetic algorithm. Add to config.toml via toml string
          ""  included_models = ['XGBOOSTGBM']\n
              params_xgboost = "{'max_depth': 2, 'max_leaves': 4, 'n_estimators': 50, 'learning_rate': 0.03}"\n
              fixed_num_folds = 2 \n
              feature_brain_level = 0 \n 
              enable_genetic_algorithm = "off" \n
          ""
  2)  This example builds a single TensorFlow model on original numeric
      features with user defined parameters. The model is allowed to
      predict outside the range of the numeric target column. The model
      is evaluated with a 4 fold cross validation scheme. Mojo creation,
      pipeline visualization and genetic algorithm is turned off. Experiment logs can be viewed to verify the parameter used by the
      TensorFlow model.

During genetic algorithm, it
      does feature engineering and will do model tuning by toggling
      OTHER PARAMS not set by the user.The Scores tab can be used to
      inspect the built models. Add to config.toml via toml string
          ""  included_models = ["LightGBMModel"] \n
              params_lightgbm = "{ 'n_estimators': 100, 'max_leaves': 64, 'random_state': 1234}" \n
          ""

Disable leakage checks, shift detection, mojo creation and ensembling
list them at the end of the experiment. Leakages or shift detection can be turned off if user is aware that the
data is good or when building a TensorFlow model.

NEW EXPERIMENTS

This page describes how to start a new experiment in Driverless AI. Note
An experiment setup wizard that guides you through the process of
setting up an experiment is also available. For more information, see
dai_wizard. 1. Run an experiment by selecting [CLICK FOR ACTIONS] button beside the
    training dataset that you want to use. Click PREDICT to begin an
    experiment. Alternatively, you can click the NEW EXPERIMENT ->
    STANDARD SETUP button on the EXPERIMENTS page, which prompts you to
    select a training dataset. (To go to the _dai_wizard, click NEW
    EXPERIMENT -> WIZARD SETUP.) Clicking STANDARD SETUP takes you
    directly to the dataset list page:
You can also get to the dataset list page from the EXPERIMENT SETUP page
by clicking TRAINING DATASET, TEST DATASET, or VALIDATION DATASET. The
dataset list page lets you view a list of datasets that are available
for selection.

Note that if your DAI instance has more than 15 datasets, the dataset
list page is organized into pages, with each page listing up to 15
datasets. 2. The Experiment Settings form displays and auto-fills with the
    selected dataset. Optionally enter a custom name for this
    experiment. If you do not add a name, Driverless AI will create one
    for you. 3. Optionally specify a validation dataset and/or a test dataset. 4. Specify the target (response) column. Note that not all explanatory
    functionality will be available for multiclass classification
    scenarios (scenarios with more than two outcomes). When the target
    column is selected, Driverless AI automatically provides the target
    column type and the number of rows. If this is a classification
    problem, then the UI shows unique and frequency statistics (Target
    Freq/Most Freq) for numerical columns. If this is a regression
    problem, then the UI shows the dataset mean and standard deviation
    values.

The next step is to set the parameters and settings for the
    experiment. (Refer to the Experiment Settings section for more
    information about these settings.) You can set the parameters
    individually, or you can let Driverless AI infer the parameters and
    then override any that you disagree with. Available parameters and
    settings include the following:
6. After your settings are made, review the Experiment Preview to learn
    what each of the settings means. NOTE: When changing the algorithms
    used via expert-settings, you may notice that those changes are not
    applied. Driverless AI determines whether to include models and/or
    recipes based on a hierarchy of those expert settings. Refer to the
    Why do my selected algorithms not show up in the Experiment Preview?<expert_settings_recipe_hierarchy>
    FAQ for more information. 7. Click LAUNCH EXPERIMENT to start the experiment.

-   The iteration data (internal validation) for each cross validation
    fold along with the specified scorer value. Click on a specific
    iteration or drag to view a range of iterations. Double click in the
    graph to reset the view. In this graph, each "column" represents one
    iteration of the experiment. During the iteration, Driverless AI
    will train n models. (This is called individuals in the experiment
    preview.) So for any column, you may see the score value for those n
    models for each iteration on the graph. -   The variable importance values. To view variable importance for a
    specific iteration, just select that iteration in the Iteration Data
    graph. The Variable Importance list will automatically update to
    show variable importance information for that iteration. Hover over
    an entry to view more info.

(Note that Trace is used for development/debugging and
    to show what the system is doing at that moment.) -   For classification problems, the lower right section includes a
    toggle between an ROC curve, Precision-Recall graph, Lift chart,
    Gains chart, and GPU Usage information (if GPUs are available). For
    regression problems, the lower right section includes a toggle
    between a Residuals chart, an Actual vs. Predicted chart, and GPU
    Usage information (if GPUs are available). (Refer to the Experiment
    Graphs section for more information.) Upon completion, an Experiment
    Summary section will populate in the lower right section. -   The bottom portion of the experiment screen will show any warnings
    that Driverless AI encounters. You can hide this pane by clicking
    the X icon. []

Finishing/Aborting Experiments
You can finish and/or abort experiments that are currently running. -   FINISH Click the FINISH button to stop a running experiment.

-   ABORT: After clicking FINISH, you have the option to click ABORT,
      which terminates the experiment. (You will be prompted to confirm
      the abort.) Aborted experiments will display on the Experiments
      page as Failed. You can restart aborted experiments by clicking
      the right side of the experiment, then selecting RESTART FROM LAST
      CHECKPOINT. This will start a new experiment based on the aborted
      one. Alternatively, you can started a new experiment based on the
      aborted one by selecting NEW EXPERIMENT WITH SAME SETTINGS. Refer
      to Checkpointing, Rerunning, and Retraining for more information. Aborting Experiment Report
The final step that Driverless AI performs during an experiment is to
complete the experiment report. During this step, you can click ABORT to
skip this report. []

1. Abort the experiment. 2.

exclusive_mode------------------  .. container:: dropdown     **Exclusive level of access to node resources**     There are three levels of access:        -  safe: this level assumes that there might be another experiment          also running on same node. -  moderate: this level assumes that there are no other          experiments or tasks running on the same node, but still only          uses physical core counts. -  max: this level assumes that there is absolutly nothing else          running on the node except the experiment     The default level is "safe" and the equivalent config.toml parameter    isexclusive_mode`. If :ref:`multinode <multinode-training> is
    enabled, this option has no effect, unless
    worker_remote_processors=1 when it will still be applied. Each
    exclusive mode can be chosen, and then fine-tuned using each expert
    settings.

MAX_CORES``

NUMBER OF CORES TO USE

Specify the number of cores to use per experiment. Note that if you
specify 0, all available cores will be used. Lower values can reduce
memory usage but might slow down the experiment. This value defaults to
0(all). One can also set it using the environment variable
OMP_NUM_THREADS or OPENBLAS_NUM_THREADS (e.g., in bash: 'export
OMP_NUM_THREADS=32' or 'export OPENBLAS_NUM_THREADS=32')

MAX_FIT_CORES``

MAXIMUM NUMBER OF CORES TO USE FOR MODEL FIT

Specify the maximum number of cores to use for a model's fit call. Note
that if you specify 0, all available cores will be used. This value
defaults to 10.

use_dask_cluster--------------------  .. container:: dropdown     **If full dask cluster is enabled, use full cluster**     Specify whether to use full multinode distributed cluster (True) or    single-node dask (False). In some cases, using entire cluster can be    inefficient. E.g. several DGX nodes can be more efficient, if used    one DGX at a time for medium-sized data. The equivalent config.toml    parameter isuse_dask_cluster``.

MAX_PREDICT_CORES``

MAXIMUM NUMBER OF CORES TO USE FOR MODEL PREDICT

Specify the maximum number of cores to use for a model's predict call.
Note that if you specify 0, all available cores will be used. This value
defaults to 0(all).

MAX_PREDICT_CORES_IN_DAI``

MAXIMUM NUMBER OF CORES TO USE FOR MODEL TRANSFORM AND PREDICT WHEN
DOING MLI, AUTODOC

Specify the maximum number of cores to use for a model's transform and
predict call when doing operations in the Driverless AI MLI GUI and the
Driverless AI R and Python clients. Note that if you specify 0, all
available cores will be used. This value defaults to 4.

BATCH_CPU_TUNING_MAX_WORKERS``

TUNING WORKERS PER BATCH FOR CPU

Specify the number of workers used in CPU mode for tuning. A value of 0
uses the socket count, while a value of -1 uses all physical cores
greater than or equal to 1. This value defaults to 0(socket count).

CPU_MAX_WORKERS``

NUMBER OF WORKERS FOR CPU TRAINING

Specify the number of workers used in CPU mode for training:

-   0: Use socket count (Default)
-   -1: Use all physical cores >= 1 that count

NUM_GPUS_PER_EXPERIMENT``

#GPUS/EXPERIMENT

Specify the number of GPUs to use per experiment. A value of -1
(default) specifies to use all available GPUs. Must be at least as large
as the number of GPUs to use per model (or -1). In multinode context
when using dask, this refers to the per-node value.

MIN_NUM_CORES_PER_GPU``

NUM CORES/GPU

Specify the number of CPU cores per GPU. In order to have a sufficient
number of cores per GPU, this setting limits the number of GPUs used.
This value defaults to 2.

num_gpus_per_model----------------------  .. container:: dropdown     **#GPUs/Model**     Specify the number of GPUs to user per model. The equivalent    config.toml parameter isnum_gpus_per_model`` and the default value

    is 1. Currently num_gpus_per_model other than 1 disables GPU
    locking, so is only recommended for single experiments and single
    users. Setting this parameter to -1 means use all GPUs per model. In
    all cases, XGBoost tree and linear models use the number of GPUs
    specified per model, while LightGBM and Tensorflow revert to using 1
    GPU/model and run multiple models on multiple GPUs. FTRL does not
    use GPUs. Rulefit uses GPUs for parts involving obtaining the tree
    using LightGBM. In multinode context when using dask, this parameter
    refers to the per-node value.

num_gpus_for_prediction---------------------------  .. container:: dropdown     **Num. of GPUs for Isolated Prediction/Transform**     Specify the number of GPUs to use forpredictfor models andtransformfor transformers when running outside offit/fit_transform. Ifpredictortransformare called    in the same process asfit/fit_transform, the number of GPUs    will match. New processes will use this count for applicable models    and transformers. Note that enablingtensorflow_nlp_have_gpus_in_productionwill override this setting    for relevant TensorFlow NLP transformers. The equivalent config.toml    parameter isnum_gpus_for_prediction`` and the default value is

    "0".

    NOTE: When GPUs are used, TensorFlow, PyTorch models and
    transformers, and RAPIDS always predict on GPU. And RAPIDS requires
    Driverless AI python scoring package also to be used on GPUs. In
    multinode context when using dask, this refers to the per-node
    value.

gpu_id_start----------------  .. container:: dropdown     **GPU Starting ID**     Specify Which gpu_id to start with. If using CUDA_VISIBLE_DEVICES=...    to control GPUs (preferred method), gpu_id=0 is the first in that    restricted list of devices. For example, ifCUDA_VISIBLE_DEVICES='4,5'thengpu_id_start=0`` will refer to
    device #4.

ASSUMED_SIMULTANEOUS_DT_FORKS_MUNGING``

ASSUMED/EXPECTED NUMBER OF MUNGING FORKS

Expected maximum number of forks, used to ensure datatable doesn't
overload system. For actual use beyond this value, system will start to
have slow-down issues. THe default value is 3.

MAX_MAX_DT_THREADS_MUNGING``

MAXIMUM OF THREADS FOR DATATABLE FOR MUNGING

Maximum number of threads for datatable for munging.

MAX_DT_THREADS_MUNGING``

MAX NUMBER OF THREADS TO USE FOR DATATABLE AND OPENBLAS FOR MUNGING AND
MODEL TRAINING

Specify the maximum number of threads to use for datatable and OpenBLAS
during data munging (applied on a per process basis):

-   0 = Use all threads
-   -1 = Automatically select number of threads (Default)

MAX_DT_THREADS_READWRITE``

MAX NUMBER OF THREADS TO USE FOR DATATABLE READ AND WRITE OF FILES

Specify the maximum number of threads to use for datatable during data
reading and writing (applied on a per process basis):

MAX_DT_THREADS_STATS_OPENBLAS``

MAX NUMBER OF THREADS TO USE FOR DATATABLE STATS AND OPENBLAS

Specify the maximum number of threads to use for datatable stats and
OpenBLAS (applied on a per process basis):

ALLOW_REDUCE_FEATURES_WHEN_FAILURE``

WHETHER TO REDUCE FEATURES WHEN MODEL FAILS (GPU OOM PROTECTION)
Big models (on big data or with lot of features) can run out of memory
on GPUs. This option is primarily useful for avoiding model building
failure due to GPU Out Of Memory (OOM). Currently is applicable to all
non-dask XGBoost models (i.e. GLMModel, XGBoostGBMModel,
XGBoostDartModel, XGBoostRFModel),during normal fit or when using
Optuna. This is acheived by reducing features until model does not fail. For
example, If XGBoost runs out of GPU memory, this is detected, and
(regardless of setting of skip_model_failures), we perform feature
selection using XGBoost on subsets of features. The dataset is
progressively reduced by factor of 2 with more models to cover all
features. This splitting continues until no failure occurs. Then all
sub-models are used to estimate variable importance by absolute
information gain, in order to decide which features to include.

REDUCE_REPEATS_WHEN_FAILURE``

NUMBER OF REPEATS FOR MODELS USED FOR FEATURE SELECTION DURING FAILURE
RECOVERY

With
allow_reduce_features_when_failure <allow_reduce_features_when_failure>,
this controls how many repeats of sub-models are used for feature
selection. A single repeat only has each sub-model consider a single
sub-set of features, while repeats shuffle hich features are considered
allowing more chance to find important interactions. More repeats can
lead to higher accuracy. The cost of this option is proportional to the
repeat count. The default value is 1.

FRACTION_ANCHOR_REDUCE_FEATURES_WHEN_FAILURE``

FRACTION OF FEATURES TREATED AS ANCHOR FOR FEATURE SELECTION DURING
FAILURE RECOVERY

With
this controls the fraction of features treated as an anchor that are
fixed for all sub-models. Each repeat gets new anchors. For tuning and
evolution, the probability depends upon any prior importance (if
present) from other individuals, while final model uses uniform
probability for anchor features. The default fraction is 0.1.

XGBOOST_REDUCE_ON_ERRORS_LIST``

ERRORS FROM XGBOOST THAT TRIGGER REDUCTION OF FEATURES

Error strings from XGBoost that are used to trigger re-fit on reduced
sub-models. See allow_reduce_features_when_failure.

LIGHTGBM_REDUCE_ON_ERRORS_LIST``

ERRORS FROM LIGHTGBM THAT TRIGGER REDUCTION OF FEATURES

Error strings from LightGBM that are used to trigger re-fit on reduced

NUM_GPUS_PER_HYPEROPT_DASK``

GPUS / HYPEROPTDASK

Specify the number of GPUs to use per model hyperopt training task. To
use all GPUs, set this to -1. For example, when this is set to -1 and
there are 4 GPUs available, all of them can be used for the training of
a single model across a Dask cluster. Ignored if GPUs are disabled or if
there are no GPUs on system. In multinode context, this refers to the
per-node value.

DETAILED_TRACES``

ENABLE DETAILED TRACES

Specify whether to enable detailed tracing in Driverless AI trace when
running an experiment. This is disabled by default.

DEBUG_LOG``

ENABLE DEBUG LOG LEVEL

If enabled, the log files will also include debug logs. This is disabled

LOG_SYSTEM_INFO_PER_EXPERIMENT``

ENABLE LOGGING OF SYSTEM INFORMATION FOR EACH EXPERIMENT

Specify whether to include system information such as CPU, GPU, and disk
space at the start of each experiment log. Note that this information is
already included in system logs. This is enabled by default.

SUPPORTED ALGORITHMS

Constant Model
A Constant Model predicts the same constant value for any input data. The constant value is computed by optimizing the given scorer. For
example, for MSE/RMSE, the constant is the (weighted) mean of the target
column. For MAE, it is the (weighted) median. For other scorers like
MAPE or custom scorers, the constant is found with an optimization
process. For classification problems, the constant probabilities are the
observed priors. A constant model is meant as a baseline reference model. If it ends up
being used in the final pipeline, a warning will be issued because that
would indicate a problem in the dataset or target column (e.g., when
trying to predict a random outcome). Decision Tree
A Decision Tree is a single (binary) tree model that splits the training
data population into sub-groups (leaf nodes) with similar outcomes.

FTRL
Follow the Regularized Leader (FTRL) is a DataTable implementation [1]
of the FTRL-Proximal online learning algorithm proposed in [4]. This
implementation uses a hashing trick and Hogwild approach [3] for
parallelization. FTRL supports binomial and multinomial classification
for categorical targets, as well as regression for continuous targets. GLM
Generalized Linear Models (GLM) estimate regression models for outcomes
following exponential distributions. GLMs are an extension of
traditional linear models. They have gained popularity in statistical
data analysis due to:
-   the flexibility of the model structure unifying the typical
    regression methods (such as linear regression and logistic
    regression for binary classification)
-   the recent availability of model-fitting software
-   the ability to scale well with large datasets
Driverless AI uses the XGBoost GLM implementation (booster=gblinear) for
modeling. This GLM is subject to early stopping.

Isolation Forest isolates observations by randomly selecting a
are highly likely to be anomalies. LightGBM
LightGBM is a gradient boosting framework developed by Microsoft that
uses tree based learning algorithms. It was specifically designed for
lower memory usage and faster training speed and higher efficiency. Similar to XGBoost, it is one of the best gradient boosting
implementations available. It is also used for fitting Random Forest,
DART (experimental), and Decision Tree models inside of Driverless AI.

Driverless AI's NLP BERT models are implemented using PyTorch, for
details see NLP in Driverless AI <nlp-in-dai>. PyTorch Grownet Model
Gradient Boosting Neural Networks or GrowNet applies gradient boosting
to shallow neural networks. The GrowNet is an ensemble consisting of k
neural net models. Each model is fed the original features and the
predictions of the previous model. The predictions of all the models are
summed to produce a final output. Every model can be as simple as having
only one hidden layer. As per the paper, GrowNet is easy to tune and
requires less computational cost and time to train, than _deep neural
networks_ and yet seems to outperform deep neural networks in
regression, classification, and ranking on multiple datasets. Driverless AI integrates the Pytorch implementation of Grownet. The
model expert settings parameter enable_grownet <enable_grownet> controls
the run.

Driverless AI supports both XGBoost RandomForest (XGBRF) and LightGBM
RandomForest (boosting=rf) implementations for modeling. RuleFit
The RuleFit [2] algorithm creates an optimal set of decision rules by
first fitting a tree model, and then fitting a Lasso (L1-regularized)
GLM model to create a linear model consisting of the most important tree
leaves (rules). MOJOs are not currently available for RuleFit models. TensorFlow
TensorFlow is an open source software library for performing high
performance numerical computation. Driverless AI includes
TensorFlow NLP <nlp_fe> recipes based on CNN ad BiGRU (RNN) Deeplearning
models and Tensorflow Imagenet models <image-processing-in-dai> for
image data. A TensorFlow model is a fully connected neural network with a few hidden
layers (that is, a multilayer perceptron). It has a few tuning
parameters that can add wide and deep or attention. TensorFlow is considered a model like XGB, LGBM, or GLM.

Only C++ MOJOs are currently available for TensorFlow models. XGBoost
XGBoost is a supervised learning algorithm that implements a process
called boosting to yield accurate models. Boosting refers to the
ensemble learning technique of building many models sequentially, with
each new model attempting to correct for the deficiencies in the
previous model. In tree boosting, each new model that is added to the
ensemble is a decision tree. XGBoost provides parallel tree boosting
(also known as GBDT, GBM) that solves many data science problems in a
fast and accurate way. For many problems, XGBoost is one of the best
gradient boosting machine (GBM) frameworks today. Driverless AI supports XGBoost GBM and XGBoost DART models. Zero-Inflated Models
Zero-inflated models fit the data with excess zero counts in the target
variable for example in insurance claim use case. In Driverless AI, this
model trains a classifier that attempts to classify zero and non-zero
values.

IMBALANCED MODELING IN DRIVERLESS AI

This page describes Driverless AI's imbalanced modeling capabilities. -   imbalanced_modeling_overview
-   imbalanced_algorithms_enabling

Overview
Driverless AI offers imbalanced algorithms for use cases where there is
a binary, imbalanced target. These algorithms are enabled by default if
the target column is considered imbalanced. While they are enabled,
Driverless AI may decide to not use them in the final model to avoid
poor performance. Note
While Driverless AI does try imbalanced algorithms by default, they have
not generally been found to improve model performance. Note that using
imbalanced algorithms also results in a significantly larger final
model, because multiple models are combined with different balancing
ratios. Imbalanced algorithms
Driverless AI provides two types of imbalanced algorithms:
ImbalancedXGBoost and ImbalancedLightGBM.

These
models use different samples of data and may use different sampling
ratios. (By trying multiple ratios, DAI is more likely to come up with a
robust model.) Note
- When your experiment is complete, you can find more details about what
bagging was performed in the experiment AutoDoc <autodoc>. For a sample
AutoDoc, view the blog post on this topic. -   For more information on imbalanced modeling sampling methods, see
    imbalanced-sampling. Enabling imbalanced algorithms
The following steps describe how to enable only imbalanced algorithms:
1. On the EXPERIMENT SETUP page, click EXPERT SETTINGS. 2. In the Expert Settings window, click on the TRAINING -> MODELS
    subtab. 3. For the INCLUDE SPECIFIC MODELS setting, click the SELECT VALUES
    button. 4. On the SELECTED INCLUDED MODELS page, click UNCHECK ALL, and then
    select only the imbalanced algorithms: ImbalancedXGBoost and
    ImbalancedLightGBM. Click DONE to confirm your selection. 5.

ENABLE_TENSORFLOW_TEXTCNN``

ENABLE WORD-BASED CNN TENSORFLOW MODELS FOR NLP

Specify whether to use out-of-fold predictions from Word-based CNN
TensorFlow models as transformers for NLP. This option is ignored if
TensorFlow is disabled. We recommend that you disable this option on
systems that do not use GPUs.

ENABLE_TENSORFLOW_TEXTBIGRU``

ENABLE WORD-BASED BIGRU TENSORFLOW MODELS FOR NLP

Specify whether to use out-of-fold predictions from Word-based BiG-RU

ENABLE_TENSORFLOW_CHARCNN``

ENABLE CHARACTER-BASED CNN TENSORFLOW MODELS FOR NLP

Specify whether to use out-of-fold predictions from Character-level CNN

ENABLE_PYTORCH_NLP_MODEL``

ENABLE PYTORCH MODELS FOR NLP

Specify whether to enable pretrained PyTorch models and fine-tune them
for NLP tasks. This is set to AUTO by default. You need to set this to
ON if you want to use the PyTorch models like BERT for modeling. Only
the first text column will be used for modeling with these models. We
recommend that you disable this option on systems that do not use GPUs.

ENABLE_PYTORCH_NLP_TRANSFORMER``

ENABLE PRE-TRAINED PYTORCH TRANSFORMERS FOR NLP

Specify whether to enable pretrained PyTorch models for NLP tasks. This
is set to AUTO by default, and is enabled for text-dominated problems
only. You need to set this to ON if you want to use the PyTorch models
like BERT for feature engineering (via fitting a linear model on top of
pretrained embeddings). We recommend that you disable this option on

NOTES:

-   This setting requires an Internet connection.

PYTORCH_NLP_PRETRAINED_MODELS``

SELECT WHICH PRETRAINED PYTORCH NLP MODELS TO USE

Specify one or more pretrained PyTorch NLP models to use. Select from
the following:

-   bert-base-uncased (Default)
-   distilbert-base-uncased (Default)
-   xlnet-base-cased
-   xlm-mlm-enfr-1024
-   roberta-base
-   albert-base-v2
-   camembert-base
-   xlm-roberta-base

NOTES:

-   Models that are not selected by default may not have MOJO support.
-   Using BERT-like models may result in a longer experiment completion
    time.

TENSORFLOW_MAX_EPOCHS_NLP``

MAX TENSORFLOW EPOCHS FOR NLP

When building TensorFlow NLP features (for text data), specify the
maximum number of epochs to train feature engineering models with (it
might stop earlier). The higher the number of epochs, the higher the run
time. This value defaults to 2 and is ignored if TENSORFLOW MODELS is
disabled.

ENABLE_TENSORFLOW_NLP_ACCURACY_SWITCH``

ACCURACY ABOVE ENABLE TENSORFLOW NLP BY DEFAULT FOR ALL MODELS

Specify the accuracy threshold. Values equal and above will add all
enabled TensorFlow NLP models at the start of the experiment for
text-dominated problems when the following NLP expert settings are set
to AUTO:

-   Enable word-based CNN TensorFlow models for NLP
-   Enable word-based BigRU TensorFlow models for NLP
-   Enable character-based CNN TensorFlow models for NLP

If the above transformations are set to ON, this parameter is ignored.

At lower accuracy, TensorFlow NLP transformations will only be created
as a mutation. This value defaults to 5.

PYTORCH_NLP_FINE_TUNING_NUM_EPOCHS``

NUMBER OF EPOCHS FOR FINE-TUNING OF PYTORCH NLP MODELS

Specify the number of epochs used when fine-tuning PyTorch NLP models.

PYTORCH_NLP_FINE_TUNING_BATCH_SIZE``

BATCH SIZE FOR PYTORCH NLP MODELS

Specify the batch size for PyTorch NLP models. This value defaults to
10.

NOTE: Large models and batch sizes require more memory.

PYTORCH_NLP_FINE_TUNING_PADDING_LENGTH``

MAXIMUM SEQUENCE LENGTH FOR PYTORCH NLP MODELS

Specify the maximum sequence length (padding length) for PyTorch NLP
models. This value defaults to 100.

NOTE: Large models and padding lengths require more memory.

PYTORCH_NLP_PRETRAINED_MODELS_DIR``

PATH TO PRETRAINED PYTORCH NLP MODELS

Specify a path to pretrained PyTorch NLP models. To get all available
models, download
http://s3.amazonaws.com/artifacts.h2o.ai/releases/ai/h2o/pretrained/bert_models.zip,
then extract the folder and store it in a directory on the instance
where Driverless AI is installed:

    pytorch_nlp_pretrained_models_dir = /path/on/server/to/bert_models_folder

tensorflow_nlp_pretrained_embeddings_file_path--------------------------------------------------  .. container:: dropdown     **Path to Pretrained Embeddings for TensorFlow NLP Models**     Specify a path to pretrained embeddings that will be used for the    TensorFlow NLP models. Note that this can be either a path in the    local file system (/path/on/server/to/file.txt) or an S3 location    (s3://``). NOTES:
  -   If an S3 location is specified, an S3 access key ID and S3 secret
      access key can also be specified with the
      tensorflow_nlp_pretrained_s3_access_key_id and
      tensorflow_nlp_pretrained_s3_secret_access_key expert settings
      respectively. -   You can download the Glove embeddings from here and specify the
      local path in this box. -   You can download the fasttext embeddings from here and specify the
      local path in this box. -   You can also train your own custom embeddings.

tensorflow_nlp_pretrained_s3_access_key_id----------------------------------------------  .. container:: dropdown     **S3 access key ID to use when**tensorflow_nlp_pretrained_embeddings_file_path**is set to an S3    location**     Specify an S3 access key ID to use whentensorflow_nlp_pretrained_embeddings_file_path` is set to an S3 location. For more information, see :ref:`the entry on the tensorflow_nlp_pretrained_embeddings_file_path <tensorflow_nlp_pretrained_embeddings_file_path>

tensorflow_nlp_pretrained_s3_secret_access_key--------------------------------------------------  .. container:: dropdown     **S3 secret access key to use when**tensorflow_nlp_pretrained_embeddings_file_path**is set to an S3    location**     Specify an S3 secret access key to use whentensorflow_nlp_pretrained_embeddings_file_path` is set to an S3 location. For more information, see :ref:`the entry on the tensorflow_nlp_pretrained_embeddings_file_path <tensorflow_nlp_pretrained_embeddings_file_path>

TENSORFLOW_NLP_PRETRAINED_EMBEDDINGS_TRAINABLE``

FOR TENSORFLOW NLP, ALLOW TRAINING OF UNFROZEN PRETRAINED EMBEDDINGS

Specify whether to allow training of all weights of the neural network
graph, including the pretrained embedding layer weights. If this is
disabled, the embedding layer will be frozen. All other weights,
however, will still be fine-tuned. This is disabled by default.

TEXT_FRACTION_FOR_TEXT_DOMINATED_PROBLEM``

FRACTION OF TEXT COLUMNS OUT OF ALL FEATURES TO BE CONSIDERED A
TEXT-DOMINANTED PROBLEM

Specify the fraction of text columns out of all features to be
considered as a text-dominated problem. This value defaults to 0.3.

Specify when a string column will be treated as text (for an NLP
problem) or just as a standard categorical variable. Higher values will
favor string columns as categoricals, while lower values will favor
string columns as text. This value defaults to 0.3.

TEXT_TRANSFORMER_FRACTION_FOR_TEXT_DOMINATED_PROBLEM``

FRACTION OF TEXT PER ALL TRANSFORMERS TO TRIGGER THAT TEXT DOMINATED

considered a text-dominated problem. This value defaults to 0.3.

STRING_COL_AS_TEXT_THRESHOLD``

THRESHOLD FOR STRING COLUMNS TO BE TREATED AS TEXT

Specify the threshold value (from 0 to 1) for string columns to be
treated as text (0.0 - text; 1.0 - string). This value defaults to 0.3.

TEXT_TRANSFORMERS_MAX_VOCABULARY_SIZE``

MAX SIZE OF THE VOCABULARY FOR TEXT TRANSFORMERS

Max number of tokens created during fitting of Tfidf/Count based text
transformers. If multiple values are provided, will use the first one
for initial models, and use remaining values during parameter tuning and
feature evolution. The default value is [1000, 5000]. Values smaller
than 10000 are recommended for speed.

EXPERT SETTINGS

This section describes the Expert Settings that are available when
starting an experiment. Driverless AI provides a variety of options in
the Expert Settings that let you customize your experiment. Use the
search bar to refine the list of settings or locate a specific setting. The default values for these options are derived from the configuration
options in the config.toml file. When a setting is changed from the
default value, it is highlighted in the interface to indicate that the
default value is not currently selected. For information on Expert Settings usage and a full list of
configuration options, see config_usage and dai_config. Note
By default, the feature brain pulls in any better model regardless of
the features even if the new model disabled those features. For full
control over features pulled in via changes in these Expert Settings,
users should set the FEATURE BRAIN LEVEL option to 0.

INSTALL ON NVIDIA GPU CLOUD/NGC REGISTRY

Driverless AI is supported on the following NVIDIA DGX products, and the
installation steps for each platform are the same. -   NVIDIA GPU Cloud
-   NVIDIA DGX-1
-   NVIDIA DGX-2
-   NVIDIA DGX Station

  ---------------------------------------------------------------
  Provider                     GPUs   Min Memory   Suitable for
  ---------------------------- ------ ------------ --------------
  NVIDIA GPU Cloud             Yes                 Serious use
  NVIDIA DGX-1/DGX-2           Yes    128 GB       Serious use
  NVIDIA DGX Station           Yes    64 GB        Serious Use

Installing the NVIDIA NGC Registry
NOTE: These installation instructions assume that you are running on an
NVIDIA DGX machine. Driverless AI is only available in the NGC registry
for DGX machines. 1.

(Note that NVIDIA Compute is no
    longer supported by NVIDIA.) 2. In the REGISTRY > PARTNERS menu, select H2OAI-DRIVERLESS. 3. At the bottom of the screen, select one of the H2O Driverless AI
    tags to retrieve the pull command. 4. On your NVIDIA DGX machine, open a command prompt and use the
    specified pull command to retrieve the Driverless AI image. For
    example:
    machine:
6. Set up the data, log, license, and tmp directories on the host
    machine:
7. At this point, you can copy data into the data directory on the host
    machine. The data will be visible inside the Docker container. 8. Enable persistence of the GPU. Note that this only needs to be run
    once. Refer to the following for more information:
    http://docs.nvidia.com/deploy/driver-persistence/index.html. 9. Run docker images to find the new image tag. 10.

Depending on your install version, use the
    (harmless) warning message. 11. Connect to Driverless AI with your browser:

Use Ctrl+C to stop Driverless AI. Upgrading Driverless AI
The steps for upgrading Driverless AI on an NVIDIA DGX system are
similar to the installation steps. WARNINGS:
    upgrading. -   Build MLI models before upgrading. -   Build MOJO pipelines before upgrading.

The upgrade process inherits the service user and group from
upgrade. NOTE: Use Ctrl+C to stop Driverless AI if it is still running. Requirements
As of 1.7.0, CUDA 9 is no longer supported. Your host environment must
have CUDA 10.0 or later with NVIDIA drivers >= 440.82 installed (GPU
only). Driverless AI ships with its own CUDA libraries, but the driver
must exist in the host environment. Go to
https://www.nvidia.com/Download/index.aspx to get the latest NVIDIA
Tesla V/P/K series driver. Upgrade Steps
1. On your NVIDIA DGX machine, create a directory for the new
    Driverless AI version. 2. Copy the data, log, license, and tmp directories from the previous
    Driverless AI directory into the new Driverless AI directory. 3. Run docker pull nvcr.io/h2oai/h2oai-driverless-ai:latest to retrieve
    the latest Driverless AI version. 4.

USING LICENSE MANAGER (BETA)

The following sections describe how to use License Manager with
Driverless AI. Presently it is in beta state and is optional. Please
contact support@h2o.ai to get License manager artifacts. -   understanding-lm
-   configure-lm

Understanding License Manager
License Manager is a software that is used to assist in the monitoring
of license usage for H2O.ai products. It allows for the application of a
single global license that can optionally implement specific
restrictions (for example, a restriction on the maximum number of
concurrent Driverless AI users can be specified). The license is applied
to the License Management server, not to individual products. Configuring Driverless AI to Use License Manager
Although Driverless AI can technically be started without the license
manager server running, you would not be able to log in and use the
software if Driverless AI is unable to communicate with a running
license management server.

To learn how to start a License Manager server and configure Driverless
AI to use License Manager, refer to the following sections:
-   start-lm-server
-   lm-setup
-   start-dai-with-lm
Starting a License Manager Server
To start a License Manager server, refer to the following steps:
1. Obtain a license manager install artifact from H2O.ai. Choose from
    the following:
      -   DEB
      -   RPM
      -   Docker
      -   Linux binary
2. Install the artifact:
      -   DEB - dpkg -i /path/to/lms.deb
      -   RPM - rpm -ivh /path/to/lms.rpm
      -   Docker - docker load < /path/to/lms.tar.gz
      -   Linux binary - No install necessary. Only a Linux-based
          machine is required
3. Start the License Manager server. This process may vary depending on
    the install type. systemd-based artifacts may require some changes
    to startup scripts if custom startup is needed.

By default, the license manager UI is available at
    http://license-manager-ip-address:9999. License Manager Server Setup
1. To acquire a license, contact support@h2o.ai. 2. Create a new project or use the default project with a
    useful/explicit name. 3. Enable the new project. 4. Navigate to the Licenses panel in License Manager UI and load the
    license to the License Manager server. Links to the Licenses panel
    are located in the left-hand side bar of the interface. []
Starting Driverless AI with License Manager
To configure Driverless AI to use License Manager on startup, use the
config.toml <config_file> file. The following TOML options can also be
set with environment variables. NOTE: The Driverless AI instance must have the ability to communicate
with the License Manager server over a network.

OUT OF MEMORY HANDLING IN DRIVERLESS AI

This page describes options for reducing memory usage to avoid out of
memory errors during the final model building stage. REDUCING ESTIMATED MEMORY USAGE AND THE NUMBER OF CORES USED PER
EXPERIMENT
To avoid out of memory errors in situations where many different
transformers are used at the same time, set the following options as
environment variables when starting DAI. Note that these configuration
options can also be set in the config.toml file <understanding-configs>. -   final_munging_memory_reduction_factor: Specify a factor by which to
    reduce estimated memory usage during the final ensemble feature
    engineering stage. Larger values use less memory, with 1 using the
    highest amount of memory. -   max_cores: Specify the number of cores to use per experiment. Note
    that if you specify 0, all available cores will be used.

max_workers_final_base_models = 1to automatically limit the number of models built at the same time to 1. This option is useful in situations where a specific transformer or model uses more memory than expected. **Limiting the total number of features**  You can limit the total number of features with the :ref:`config_nfeatures_max` configuration option. For example, if you encounter an out of memory error due to having a large number of features, you can set this option and refit the best model to see if the error is resolved. **Limiting the maximum number of genes per model**  You can specify the maximum number of genes (transformer instances) per model with the :ref:`config_ngenes_max` configuration option. **Additional options**  -  :ref:`config_munging_memory_overhead_factor:`: Specify memory usage    per transformer per input data size.

PAM AUTHENTICATION EXAMPLE

The following sections describe how to enable Pluggable Authentication
Modules (PAM) in Driverless AI. You can do this by specifying
environment variables in the Docker image or by updating the config.toml
file. NOTE: This assumes that the user has an understanding of how to grant
permissions in their own environment in order for PAM to work. Specifically for Driverless AI, be sure that the Driverless AI processes
owner has access to /ETC/SHADOW (without root); otherwise authentication
will fail. Docker Image Installs
NOTE: The following instructions are only applicable with a CentOS 7
host. In this example, the host Linux system has PAM enabled for
authentication and Docker running on that Linux system. The goal is to
enable PAM for Driverless AI authentication while the Linux system hosts
the user information. 1. Verify that the username ("eric" in this case) is defined in the
    Linux system. 2.

INSTALL THE DRIVERLESS AI AWS COMMUNITY AMI

WATCH THE INSTALLATION VIDEO here. Note that some of the images in this
same.

Log in to your AWS account at https://aws.amazon.com. 2. In the upper right corner of the Amazon Web Services page, set the
    location drop-down. (NOTE: We recommend selecting the US East region
    because H2O's resources are stored there. It also offers more
    instance types than other regions.) 3. Select the EC2 option under the Compute section to open the EC2
    Dashboard. 4. Click the LAUNCH INSTANCE button under the Create Instance section. 5. Under Community AMIs, search for H2OAI, and then select the version
    that you want to launch. 6. On the Choose an Instance Type page, select GPU COMPUTE in the
    FILTER BY dropdown. This will ensure that your Driverless AI
    instance will run on GPUs. Select a GPU compute instance from the
    available options. (We recommend at least 32 vCPUs.) Click the NEXT:
    CONFIGURE INSTANCE DETAILS button. 7. Specify the Instance Details that you want to configure.

Click NEXT: ADD STORAGE. 8. Specify the Storage Device settings. Note again that Driverless AI
    requires 10 GB to run and will stop working of less than 10 GB is
    available. The machine should have a minimum of 30 GB of disk space. Click NEXT: ADD TAGS. 9. If desired, add unique Tag name to identify your instance. Click
    NEXT: CONFIGURE SECURITY GROUP. 10. Add the following security rules to enable SSH access to Driverless
    AI, then click REVIEW AND LAUNCH.

12. A popup will appear prompting you to select a key pair. This is
    required in order to SSH into the instance. You can select your
    existing key pair or create a new one. Be sure to accept the
    acknowledgement, then click LAUNCH INSTANCES to start the new
    instance. 13. Upon successful completion, a message will display informing you
    that your instance is launching. Click the VIEW INSTANCES button to
    see information about the instance including the IP address. The
    CONNECT button on this page provides information on how to SSH into
    your instance. 14. Open a Terminal window and SSH into the IP address of the AWS
    instance. Replace the DNS name below with your instance DNS. 15. If you selected a GPU-compute instance, then you must enable
    persistence and optimizations of the GPU. The commands vary
    depending on the instance type. Note also that these commands need
    to be run once every reboot.

At this point, you can copy data into the data directory on the host
    machine using scp. For example:
17. Connect to Driverless AI with your browser. Sign in to Driverless AI
    with the username H2OAI and use the AWS InstanceID as the password. You will be prompted to enter your Driverless AI license key when
    you log in for the first time. Stopping the EC2 Instance
The EC2 instance will continue to run even when you close the
aws.amazon.com portal. To stop the instance:
1. On the EC2 Dashboard, click the RUNNING INSTANCES link under the
    Resources section. 2. Select the instance that you want to stop. 3. In the ACTIONS drop down menu, select INSTANCE STATE > STOP. 4. A confirmation page will display. Click YES, STOP to stop the
    instance. Upgrading the Driverless AI Community Image
WARNINGS:
    earlier.

We recommend you take the following steps before
to the start and h2oai scripts. 1. SSH into the IP address of the image instance and copy the existing
2.  wget the newer image. The command below retrieves version 1.2.2:
3. In the /HOME/UBUNTU/SCRIPTS/ folder, edit both the start.sh and
    h2oai.sh scripts to use the newer image. 4.

SHARING EXPERIMENTS

This page describes how to share DRIVERLESS AI (DAI) experiments by
exporting and importing experiments or by using Remote Storage. -   export_import
-   remote_storage
  Sharing Method                      Requirements
  Exporting and Importing Experiments Requires only DAI
  Experiments                         
  Remote Storage                      Requires H2O AI Cloud (HAIC)
                                      <https://docs.h2o.ai/haic/
                                      latest/>__

Exporting and Importing Experiments
As of version 1.10, DAI supports exporting and importing DAI
experiments. You can download experiments as a .dai file that can be
imported by other DAI users.

All exported Driverless AI experiments have the .dai
file extension. Importing an Experiment
To import an experiment, click the IMPORT EXPERIMENT button on the
Experiment listing page, and then select the DAI experiment file you
want to import from your local file system. You can also drag the DAI
experiment file from your local file system to the Experiment listing
page. If the selected experiment used custom recipes, the custom recipes
associated with the experiment are also imported. Datasets associated with imported experiments are not imported as part
of the experiment import process. Instead, only a minimal set of
metadata is imported. To take advantage of certain features such as
interpreting experiments and previewing datasets, you must manually
import the datasets associated with the imported experiment. Warning
To ensure that the import process is not interrupted, do not refresh the
page while the experiment is being imported.

Remote Storage
Note
This method requires H2O AI CLOUD (HAIC). For more information on HAIC,
see the HAIC documentation. Note
Unsupervised experiments are not currently supported by both Remote
Storage and H2O MLOps. Remote storage is only available to H2O AI Cloud (HAIC) users. In most
cases, experiments that are placed in a Project are automatically added
to Remote Storage. However, if the Project is created by clicking NEW
EXPERIMENT > CREATE LEADERBOARD, the experiments in that Project are not
automatically added to Remote Storage. To add an experiment in a
Leaderboard Project to Remote Storage, navigate to the Project and open
the drop-down options menu for the experiment, and then click LINK
REMOTELY. If a project is shared with you by another DAI user, the experiments and
datasets associated with that project are initially greyed out,
indicating that they live only in the Remote Storage. Before they can be
viewed and used, you must import them.

[]
Note that every dataset and experiment linked to the project is labeled
with one of the following:
-   LOCAL: The experiment only exists in local storage. This occurs when
    you run leaderboards. The experiments that are created as part of a
    leaderboard are not automatically linked to remote storage. If you
    want these experiments to be linked to remote storage, you must link
    them manually. -   REMOTE: The dataset or experiment exists in remote storage and is
    available for import. -   LOCAL & REMOTE: The dataset or experiment exists in both local and
    remote storage. If only an experiment is imported, but not its datasets, some
functionalities are not available such as diagnostics or MLI. Both the
experiment and its datasets must be imported to use all of the
experiment's functionalities. Experiments in Remote Storage are made available in H2O MLOps and can be
shared with other users.

Projects saved in Remote Storage are made available in newly
created instances of DAI. This means that in cases where you need to
keep an old experiment, model interpretation, or AutoDoc for reference
purposes, keeping the specific DAI instance containing them isn't
necessary. Instead, you can create a project, link the relevant
experiment and data, and delete the DAI instance. The model can then be
deployed to H2O MLOps, from which you can download the AutoDoc
associated with the model. In addition, you can create a new DAI
instance, import the project, and run and view the model interpretation. Following this practice can help lower costs by eliminating the need to
keep specific instances of DAI. You can configure Remote Storage with the enable_storage parameter in
the config.toml file. Sharing With Other Users
To share your project with other users, go to the Projects page and open
the drop-down menu for the project you want to share, then click SHARE.

Select one of the following roles:
-   DEFAULT: This role is equivalent to granting write access to a user. Users with this role can make any modification to the shared
    project, including renaming the project, adding datasets, adding
    experiments, adding a note, and rerunning experiments. Users that
    are granted this role can perform any action that they are able to
    perform on projects they create and own. Warning
    Users with the DEFAULT role can delete projects that have been
    shared with them. If a user with the DEFAULT role deletes a project,
    it is also deleted for both the original owner and other shared
    users. -   READER: This role is equivalent to granting read-only access to a
    user. Users with this role can view the project and import datasets
    or experiments from it, but cannot make modifications to the project
    such as adding or deleting datasets or experiments.

FAQ

difficult data science and machine learning workflows such as feature
engineering, model validation, model tuning, model selection and model
deployment. It aims to achieve highest predictive accuracy, comparable
to expert data scientists, but in much shorter time thanks to end-to-end
automation. Driverless AI also offers automatic visualizations and
machine learning interpretability (MLI). Especially in regulated
industries, model transparency and explanation are just as important as
predictive performance. Modeling pipelines (feature engineering and
models) are exported (in full fidelity, without approximations) both as
Python modules and as Java standalone scoring artifacts. This section provides answers to frequently asked questions.

You can also
post questions on the H2O.ai Community Slack workspace in the
#DRIVERLESSAI channel. If you have not signed up for the H2O.ai
https://www.h2o.ai/community/. GENERAL
-   How is Driverless AI different than any other black box ML
    algorithm? -   How often do new versions come out? INSTALLATION/UPGRADE/AUTHENTICATION
-   How can I change my username and password? -   Can Driverless AI run on CPU-only machines? -   How can I upgrade to a newer version of Driverless AI? -   What kind of authentication is supported in Driverless AI? -   How can I automatically turn on persistence each time the GPU system
    reboots? -   How can I start Driverless AI on a different port than 12345? -   Can I set up TLS/SSL on Driverless AI? -   Can I set up TLS/SSL on Driverless AI in AWS? -   Why do I receive a "package dai-<version>.x86_64 does not verify: no
    digest" error during the installation?

How can I fix that? -   Is it possible for multiple users to share a single Driverless AI
    instance? -   Can multiple Driverless AI users share a GPU server? -   How can I retrieve a list of Driverless AI users? -   Start of Driverless AI fails on the message "Segmentation fault
    (core dumped)" on Ubuntu 18/RHEL 7.6. How can I fix this? -   Which Linux systems does Driverless AI support? DATA
-   Is there a file size limit for datasets? -   How can I import CSV files that use UTF-8 encoding into Excel? -   Can a byte order mark be used when writing CSV files with datatable? -   Which version of Longhorn is supported by Driverless AI? -   Is it possible to download a transformed test dataset in Driverless
    AI? CONNECTORS
-   Why can't I import a folder as a file when using a data connector on
    Windows? -   I get a ClassNotFoundException error when I try to select a JDBC
    connection. How can I fix that?

How can I fix that? -   I get a "Permission Denied" error during Hive import. How do I fix
    this? RECIPES
-   Where can I retrieve H2O's custom recipes? -   How can I create my own custom recipe? -   Are MOJOs supported for experiments that use custom recipes? -   How can I use BYOR in my airgapped installation? -   When enabling recipes in Driverless AI, can I install Python
    packages from my organization's internal Python package index? EXPERIMENTS
-   How much memory does Driverless AI require in order to run
    experiments? -   How many columns can Driverless AI handle? -   How should I use Driverless AI if I have large data? -   How does Driverless AI detect the ID column? -   Can Driverless AI handle data with missing values/nulls? -   How does Driverless AI deal with categorical variables? What if an
    integer column should really be treated as categorical? -   How are outliers handled?

-   Does Driverless AI treat numeric variables as categorical variables? -   Which algorithms are used in Driverless AI? -   Why do my selected algorithms not show up in the Experiment Preview? -   How can we turn on TensorFlow Neural Networks so they are evaluated? -   Does Driverless AI standardize the data? -   What objective function is used in XGBoost? -   Does Driverless AI perform internal or external validation? -   How does Driverless AI prevent overfitting? -   How does Driverless AI avoid the multiple hypothesis (MH) problem? -   How does Driverless AI suggest the experiment settings? -   What happens when I set Interpretability and Accuracy to the same
    number? -   Can I specify the number of GPUs to use when running Driverless AI? -   How can I create the simplest model in Driverless AI? -   Why is my experiment suddenly slow?

-   Why does the final model performance appear to be worse than
    previous iterations? -   How can I find features that may be causing data leakages in my
    Driverless AI model? -   How can I see the performance metrics on the test data? -   How can I see all the performance metrics possible for my
    experiment? -   What if my training/validation and testing data sets come from
    different distributions? -   Does Driverless AI handle weighted data? -   How does Driverless AI handle fold assignments for weighted data? -   Why do I see that adding new features to a dataset deteriorates the
    performance of the model? -   How does Driverless AI handle imbalanced data for binary
    classification experiments? -   How is feature importance calculated in Driverless AI? -   I want to have only one LightGBM model in the final pipeline. How
    can I achieve this? -   I want to have only one LightGBM model and no FE. How can I do this?

-   When should fast approximation be turned off? -   Why does the confusion matrix sometimes show decimals instead of
    whole numbers? -   Is data sampling for multiclass use cases supported? FEATURE TRANSFORMATIONS
-   Where can I get details of the various transformations performed in
    an experiment? PREDICTIONS
-   How can I download the predictions onto the machine where Driverless
    AI is running? -   Why are predicted probabilities not available when I run an
    experiment without ensembling? DEPLOYMENT
-   What drives the size of a MOJO? -   Are MOJOs thread safe? -   Running the scoring pipeline for my MOJO is taking several hours. How can I get this to run faster? -   Why have I encountered a "Best Score is not finite" error? TIME SERIES
-   What if my data has a time dependency? -   What is a lag, and why does it help? -   Why can't I specify a validation data set for time-series problems?

Is it because of
    creating the lag features on the test set? -   In regards to applying the target lags to different subsets of the
    time group columns, are you saying Driverless AI perform
    auto-correlation at "levels" of the time series? For example,
    consider the Walmart dataset where I have Store and Dept (and my
    target is Weekly Sales). Are you saying that Driverless AI checks
    for auto-correlation in Weekly Sales based on just Store, just Dept,
    and both Store and Dept? -   How does Driverless AI detect the time period? -   What is the logic behind the selectable numbers for forecast horizon
    length? -   Assume that in my Walmart dataset, all stores provided data at the
    week level, but one store provided data at the day level. What would
    Driverless AI do?

What would Driverless AI do? -   Why does the number of weeks that you want to start predicting
    matter? -   Are the scoring components of time series sensitive to the order in
    which new pieces of data arrive? I.e., is each row independent at
    scoring time, or is there a real-time windowing effect in the
    scoring pieces? -   What happens if the user, at predict time, gives a row with a time
    value that is too small or too large? -   What's the minimum data size for a time series recipe? -   How long must the training data be compared to the test data? -   How does the time series recipe deal with missing values? -   Can the time information be distributed across multiple columns in
    the input data (such as [year, day, month]? -   What type of modeling approach does Driverless AI use for time
    series? -   What's the idea behind exponential weighting of moving averages? LOGGING
-   How can I reduce the size of the Audit Logger?

HOW OFTEN DO NEW VERSIONS COME OUT? Installation/Upgrade/Authentication
HOW CAN I CHANGE MY USERNAME AND PASSWORD? CAN DRIVERLESS AI RUN ON CPU-ONLY MACHINES? HOW CAN I UPGRADE TO A NEWER VERSION OF DRIVERLESS AI? WHAT KIND OF AUTHENTICATION IS SUPPORTED IN DRIVERLESS AI? HOW CAN I AUTOMATICALLY TURN ON PERSISTENCE EACH TIME THE GPU SYSTEM
REBOOTS? HOW CAN I START DRIVERLESS AI ON A DIFFERENT PORT THAN 12345? CAN I SET UP TLS/SSL ON DRIVERLESS AI? CAN I SET UP TLS/SSL ON DRIVERLESS AI IN AWS? I RECEIVED A "PACKAGE DAI-<VERSION>.X86_64 DOES NOT VERIFY: NO DIGEST"
ERROR DURING THE INSTALLATION. HOW CAN I FIX THIS? I RECEIVED A "MUST HAVE EXACTLY ONE OPENCL PLATFORM 'NVIDIA CUDA'"
ERROR. HOW CAN I FIX THAT? IS IT POSSIBLE FOR MULTIPLE USERS TO SHARE A SINGLE DRIVERLESS AI
INSTANCE? CAN MULTIPLE DRIVERLESS AI USERS SHARE A GPU SERVER? HOW CAN I RETRIEVE A LIST OF DRIVERLESS AI USERS?

HOW CAN I FIX THIS? WHICH LINUX SYSTEMS DOES DRIVERLESS AI SUPPORT? Data
IS THERE A FILE SIZE LIMIT FOR DATASETS? HOW CAN I IMPORT CSV FILES THAT USE UTF-8 ENCODING INTO EXCEL? CAN A BYTE ORDER MARK BE USED WHEN WRITING CSV FILES WITH DATATABLE? WHICH VERSION OF LONGHORN IS SUPPORTED BY DRIVERLESS AI? IS IT POSSIBLE TO DOWNLOAD A TRANSFORMED TEST DATASET IN DRIVERLESS AI? Connectors
WHY CAN'T I IMPORT A FOLDER AS A FILE WHEN USING A DATA CONNECTOR ON
WINDOWS? I GET A CLASSNOTFOUNDEXCEPTION ERROR WHEN I TRY TO SELECT A JDBC
CONNECTION. HOW CAN I FIX THAT? I GET A ORG.DATANUCLEUS.EXCEPTIONS.NUCLEUSUSEREXCEPTION: PLEASE CHECK
YOUR CLASSPATH AND PLUGIN SPECIFICATION ERROR WHEN ATTEMPTING TO CONNECT
TO HIVE. HOW CAN I FIX THAT? I GET A "PERMISSION DENIED" ERROR DURING HIVE IMPORT. HOW DO I FIX THIS? Recipes
WHERE CAN I RETRIEVE H2O'S CUSTOM RECIPES? HOW CAN I CREATE MY OWN CUSTOM RECIPE? ARE MOJOS SUPPORTED FOR EXPERIMENTS THAT USE CUSTOM RECIPES?

WHEN ENABLING RECIPES IN DRIVERLESS AI, CAN I INSTALL PYTHON PACKAGES
FROM MY ORGANIZATION'S INTERNAL PYTHON PACKAGE INDEX? Yes—you can use the pip_install_options
  TOML option <understanding-configs> to specify your organization's
  internal Python package index as follows:
      pip_install_options="['--extra-index-url', 'http://my-own-repo:port']"
  For more information on the --extra-index-url <url> pip install
  option, refer to the official pip documentation. Experiments
HOW MUCH MEMORY DOES DRIVERLESS AI REQUIRE IN ORDER TO RUN EXPERIMENTS? HOW MANY COLUMNS CAN DRIVERLESS AI HANDLE? HOW SHOULD I USE DRIVERLESS AI IF I HAVE LARGE DATA? HOW DOES DRIVERLESS AI DETECT THE ID COLUMN? CAN DRIVERLESS AI HANDLE DATA WITH MISSING VALUES/NULLS? HOW DOES DRIVERLESS AI DEAL WITH CATEGORICAL VARIABLES? WHAT IF AN
INTEGER COLUMN SHOULD REALLY BE TREATED AS CATEGORICAL? HOW ARE OUTLIERS HANDLED?

DOES DRIVERLESS AI TREAT NUMERIC VARIABLES AS CATEGORICAL VARIABLES? WHICH ALGORITHMS ARE USED IN DRIVERLESS AI? WHY DO MY SELECTED ALGORITHMS NOT SHOW UP IN THE EXPERIMENT PREVIEW? When changing the algorithms used via EXPERT SETTINGS > MODEL and EXPERT
SETTINGS > RECIPES, you may notice in the Experiment Preview that those
changes are not applied. Driverless AI determines whether to include
models and/or recipes based on a hierarchy of those expert settings as
well as data types (numeric, categorical, text, image, etc.) and system
properties (GPUs, multiple GPUs, etc.). []
-   Setting an Algorithm to "OFF" in Expert Settings: If an algorithm is
    turned OFF in Expert Settings (for example, GLM MODELS) when
    running, then that algorithm will not be included in the experiment.

-   Algorithms Not Specified as "OFF" and Included from Recipes: If a
    Driverless AI algorithm is specified as either "AUTO" or "ON" and
    additional models are selected for the experiment in the INCLUDE
    SPECIFIC MODELS option, than those algorithms may or may not be
    included in the experiment. Driverless AI will determine the
    algorithms to use based on the data and experiment type. -   To show warnings in the preview for which models were not used, set
    show_inapplicable_models_preview = true in config.toml
WHY DO MY SELECTED TRANSFORMERS NOT SHOW UP IN THE EXPERIMENT PREVIEW? When changing the transformers used via EXPERT SETTINGS > TRANSFORMERS
and EXPERT SETTINGS > RECIPES, you may notice in the Experiment Preview
that those changes are not applied. Driverless AI determines whether to
include transformers can be used based upon data types (numeric,
categorical, text, image, etc.) and system properties (GPUs, multiple
GPUs, etc.).

-   To show warnings in the preview for which models were not used, set
    show_inapplicable_transformers_preview = true in config.toml
HOW CAN WE TURN ON TENSORFLOW NEURAL NETWORKS SO THEY ARE EVALUATED? DOES DRIVERLESS AI STANDARDIZE THE DATA? WHAT OBJECTIVE FUNCTION IS USED IN XGBOOST? DOES DRIVERLESS AI PERFORM INTERNAL OR EXTERNAL VALIDATION? HOW DOES DRIVERLESS AI PREVENT OVERFITTING? HOW DOES DRIVERLESS AI AVOID THE MULTIPLE HYPOTHESIS (MH) PROBLEM? HOW DOES DRIVERLESS AI SUGGEST THE EXPERIMENT SETTINGS? WHAT HAPPENS WHEN I SET INTERPRETABILITY AND ACCURACY TO THE SAME
NUMBER? CAN I SPECIFY THE NUMBER OF GPUS TO USE WHEN RUNNING DRIVERLESS AI? HOW CAN I CREATE THE SIMPLEST MODEL IN DRIVERLESS AI? For information on why your experiment isn't performing as expected, see
experiment_performance. WHEN I RUN MULTIPLE EXPERIMENTS WITH DIFFERENT SEEDS, WHY DO I SEE
DIFFERENT SCORES, RUNTIMES, AND SIZES ON DISK IN THE EXPERIMENTS LISTING
PAGE?

HOW CAN I FIND FEATURES THAT MAY BE CAUSING DATA LEAKAGES IN MY
DRIVERLESS AI MODEL? HOW CAN I SEE THE PERFORMANCE METRICS ON THE TEST DATA? HOW CAN I SEE ALL THE PERFORMANCE METRICS POSSIBLE FOR MY EXPERIMENT? WHAT IF MY TRAINING/VALIDATION AND TESTING DATA SETS COME FROM DIFFERENT
DISTRIBUTIONS? DOES DRIVERLESS AI HANDLE WEIGHTED DATA? HOW DOES DRIVERLESS AI HANDLE FOLD ASSIGNMENTS FOR WEIGHTED DATA? WHY DO I SEE THAT ADDING NEW FEATURES TO A DATASET DETERIORATES THE
PERFORMANCE OF THE MODEL? HOW DOES DRIVERLESS AI HANDLE IMBALANCED DATA FOR BINARY CLASSIFICATION
EXPERIMENTS? HOW IS FEATURE IMPORTANCE CALCULATED IN DRIVERLESS AI? I WANT TO HAVE ONLY ONE LIGHTGBM MODEL IN THE FINAL PIPELINE. HOW CAN I
DO THIS? I WANT TO HAVE ONLY ONE LIGHTGBM MODEL AND NO FE. HOW CAN I DO THIS? WHAT IS FAST APPROXIMATION IN DRIVERLESS AI? WHEN SHOULD FAST APPROXIMATION BE TURNED OFF? WHY DOES THE CONFUSION MATRIX SOMETIMES SHOW DECIMALS INSTEAD OF WHOLE
NUMBERS?

Feature Transformations
WHERE CAN I GET DETAILS OF THE VARIOUS TRANSFORMATIONS PERFORMED IN AN
EXPERIMENT? Predictions
HOW CAN I DOWNLOAD THE PREDICTIONS ONTO THE MACHINE WHERE DRIVERLESS AI
IS RUNNING? WHY ARE PREDICTED PROBABILITIES NOT AVAILABLE WHEN I RUN AN EXPERIMENT
WITHOUT ENSEMBLING? Deployment
WHAT DRIVES THE SIZE OF A MOJO? ARE MOJOS THREAD SAFE? RUNNING THE SCORING PIPELINE FOR MY MOJO IS TAKING SEVERAL HOURS. HOW
CAN I GET THIS TO RUN FASTER? WHY HAVE I ENCOUNTERED A "BEST SCORE IS NOT FINITE" ERROR? Time Series
WHAT IF MY DATA HAS A TIME DEPENDENCY? WHAT IS A LAG, AND WHY DOES IT HELP? WHY CAN'T I SPECIFY A VALIDATION DATA SET FOR TIME-SERIES PROBLEMS? WHY
DO YOU LOOK AT THE TEST SET FOR TIME-SERIES PROBLEMS
WHY DOES THE GAP BETWEEN TRAIN AND TEST MATTER? IS IT BECAUSE OF
CREATING THE LAG FEATURES ON THE TEST SET?

FOR EXAMPLE, CONSIDER THE WALMART DATASET
WHERE I HAVE STORE AND DEPT (AND MY TARGET IS WEEKLY SALES). ARE YOU
SAYING THAT DRIVERLESS AI CHECKS FOR AUTO-CORRELATION IN WEEKLY SALES
BASED ON JUST STORE, JUST DEPT, AND BOTH STORE AND DEPT? HOW DOES DRIVERLESS AI DETECT THE TIME PERIOD? WHAT IS THE LOGIC BEHIND THE SELECTABLE NUMBERS FOR FORECAST HORIZON
LENGTH? ASSUME THAT IN MY WALMART DATASET, ALL STORES PROVIDED DATA AT THE WEEK
LEVEL, BUT ONE STORE PROVIDED DATA AT THE DAY LEVEL. WHAT WOULD
DRIVERLESS AI DO? ASSUME THAT IN MY WALMART DATASET, ALL STORES AND DEPARTMENTS PROVIDED
DATA AT THE WEEKLY LEVEL, BUT ONE DEPARTMENT IN A SPECIFIC STORE
PROVIDED WEEKLY SALES ON A BI-WEEKLY BASIS (EVERY TWO WEEKS). WHAT WOULD
DRIVERLESS AI DO? WHY DOES THE NUMBER OF WEEKS THAT YOU WANT TO START PREDICTING MATTER? ARE THE SCORING COMPONENTS OF TIME SERIES SENSITIVE TO THE ORDER IN
WHICH NEW PIECES OF DATA ARRIVE?

CREATING CUSTOM PLOTS

To create a custom plot, click the ADD GRAPH button in the upper-right
corner and select one of the available plot types. After selecting a
plot, configure the available settings for that plot type and click
SAVE. The custom plot appears on the Visualization page once it has been
created. The following example creates a custom histogram plot for the
CreditCard-Train dataset:
The following is a complete list of available graph types. Bar chart
This plot presents categorical data with rectangular bars that are
proportional to the values they represent. The type of marker used to
represent bars determines the bar chart type. The most common marker is
the bar marker, which ranges from a lower value (usually zero) to an
upper value. Also available are the Cleveland dot plot (replaces the bar
with a dot located at the upper value) and the area chart (covers the
bars with a solid area marker).

They may represent counts (if no
_y_ variable is specified) or the average value of the _y_ variable per
category (if the _y_ variable is specified). When creating a bar chart, specify the following options:
    -   _x_ variable name: Specify the name of the _x_ variable
    -   _y_ variable name: Specify the name of the _y_ variable
    -   Transpose: Specify whether to switch the _X_-axis and _Y_-axis
    -   Sort: Specify whether to sort bars alphabetically by _x_ values
    -   Mark: Specify a marker type. Select POINT to create a Cleveland
        dot plot

Boxplot
This plot presents the fractiles of a distribution. The center of the
box represents the median, the edges of a box represent the lower and
upper quartiles, and the ends of the "whiskers" represent that range of
values. When outliers occur, the adjacent whisker is shortened to the
next lower or upper value. For variables having only a few values, the
boxes can be compressed.

When more than
one value falls within a small neighborhood, the dots are stacked. When creating a dotplot, specify the following options:
    -   Variable name: Specify the name of the variable on which dots
        are calculated
    -   Mark: Specify a marker type

Grouped Boxplot
This plot is a boxplot where categories are organized into groups and
subgroups. When creating a grouped boxplot, specify the following options:
    -   Variable name: Specify the variable that you want the box to
        represent
    -   Group variable name: Specify the name of the grouping variable

Heatmap
See data heatmap. When creating a heatmap, specify the following
options:
  -   Variable names: Specify one or more variables to use.

Choose from RECTANGULAR and
      SYMMETRIC

Histogram
This plot is a graphical display of data that uses bars of differing
height. Each bar groups numbers into ranges by its width, and taller
bars show that more data falls within a specific range. This plot is
often used to display the shape and spread of a continuous variable. When creating a histogram, specify the following options:
    -   Variable name: Specify the variable name
    -   Transformation: Specify whether to use a transformation. Choose
        from LOG and SQUARE ROOT
    -   Number of bars: Specify the number of bars to use
    -   Mark: Specify a marker type. Use AREA to create a density
        polygon

Linear Regression
This plot predicts a set of values on a variable _y_ from values on a
variable _x_ by fitting a linear function (ax + b) so that for any value
on the _x_ variable, this function yields the most probable value on the
_y_ variable.

When creating a linear regression plot, specify the following options:
    -   Mark: Specify a marker type. Choose from POINT and SQUARE

LOESS Regression
variable _x_ by fitting a locally linear function (ax + b) that
determines the most probable _y_ variable values based on the available
_x_ variable values. The effectiveness of this prediction in a sample of
values is represented by the discrepancies between the _y_ values and
their corresponding predicted values. When creating a LOESS regression plot, specify the following options:
    -   Mark: Specify a marker type.

This is set to 0.5 by
        default

Parallel Coordinates Plot
This plot is used for comparing multiple variables. Each variable has
its own vertical axis in the plot, and each profile connects the values
on the axes for a single observation. If the data contains clusters,
these profiles are color-coded by their cluster number. When creating a parallel coordinates plot, specify the following
  options:
    -   Variable names: Specify one or more variables to use. If none
        are specified, all the variables in the dataset are used
    -   Permute: Specify whether to reorder variables using singular
        value decomposition (SVD)
    -   Cluster: Specify whether to include _k_-Means cluster variables. Unique colors are assigned for each cluster ID

Probability Plot
This plot evaluates the skewness of a distribution by plotting two
cumulative distribution functions against each other.

Choose from NORMAL
        and UNIFORM

Quantile Plot
This plot compares two probability distributions by plotting their
quantiles against each other. When creating a quantile plot, specify the following options:
    -   Distribution: Specify a distribution type. Choose from NORMAL

Scatterplot
This plot represents the values of two variables (_y_ and _x_) in a
frame that contains one point for each row of the input sample data. They are useful for analyzing the joint distribution of two variables.

MODEL SCORES

You can view detailed information about model scores after an experiment
is complete by clicking on the SCORES option. []
The Model Scores page that opens includes the following tables:
-   MODEL AND FEATURE TUNING LEADERBOARD: This leaderboard shows scoring
    information based on the scorer that was selected in the experiment. This information is also available in the TUNING_LEADERBOARD.JSON
    file of the experiment_summary. You can download that file directly
    from the bottom of this table. -   FINAL PIPELINE SCORES ACROSS CROSS-VALIDATION FOLDS AND MODELS: This
    table shows the final pipeline scores across cross-validation folds
    and models. Note that if Constant Model was enabled (default), then
    that model is added in this table as a baseline (reference) only and
    will be dropped in most cases. This information is also included in
    the ENSEMBLE_BASE_LEARNER_FOLD_SCORES.JSON file of the
    experiment_summary.

CONFIGURATION AND AUTHENTICATION

config-usage config_docs/index

config_toml setting-environment-variables user-settings connectors
notifications export-artifacts language multinode snowflake-integration
pip-install

authentication

LINUX X86_64 INSTALLS

This section provides installation steps for RPM, deb, and tar installs
in Linux x86_64 environments.

linux-rpm linux-deb linux-tarsh

DOCKER IMAGE INSTALLATION

This section provides instructions for installing the Driverless AI
Docker image.

install/linux-docker-images install/mac-osx install/windows

For instructions on installing Driverless AI in native Linux
environments, refer to native_installs.

Note that from version 1.10, DAI Docker image runs with internal

tinithat is equivalent to using--initfrom Docker. If both are enabled in the launch command, tini prints a (harmless) warning message. For GPU users, as GPU needs--pid=host``
for nvml, which makes tini not use pid=1, so it will show the warning
message (still harmless).

We recommend

nproc=16384:16384` in Docker launch command. But if user plans to build

--shm-size=4g`` is recommended for Driverless AI Docker command.

WHICH PIPELINE SHOULD I USE? Driverless AI Python Scoring Pipeline
Driverless AI Python Scoring Pipeline is implemented as a Python WHL
file. While this allows for a single process scoring engine, the scoring
service is generally implemented as a client/server architecture and
supports interfaces for TCP and HTTP. When running the Python Scoring
Pipeline:
  -   HTTP is supported by virtually any language. HTTP supports RESTful
      calls via curl, wget, or supported packages in various scripting
      languages. -   TCP is a bit more complex, though faster. TCP also requires
      Thrift, which currently does not handle NAs. k-LIME reason codes and Shapley reason codes WHL file can be obtained
for ALL MODELS from MLI Standalone Python SCORING PIPELINE from the MLI
experiment page. Driverless AI MOJO Scoring Pipeline
Driverless AI MOJO Scoring Pipeline is flexible and is faster than the
Python Scoring Pipeline. It requires some coding.

CLOUD INSTALLATION

Driverless AI is available on the following cloud platforms:

-   H2O AI Cloud (HAIC)
-   AWS - Amazon Machine Image (AMI) <install-on-aws>
-   Azure <install-on-azure>
-   Google Cloud <install-on-google-compute>

The installation steps for AWS, Azure, and Google Cloud assume that you
have a license key for Driverless AI. For information on how to obtain a
license key for Driverless AI, visit
in, or you can save it as a .sig file and place it in the LICENSE folder
that you will create during the installation process.

install/aws install/azure install/google-compute

INSTALL ON GOOGLE COMPUTE

Driverless AI can be installed on Google Compute using one of two
methods:

-   Install the Google Cloud Platform offering. This installs Driverless
    AI via the available GCP Marketplace offering.
-   Install and Run in a Docker Container on Google Compute Engine. This
    installs and runs Driverless AI from scratch in a Docker container
    on Google Compute Engine.

Select your desired installation procedure below:

google-cloud-platform google-docker-container

DOWNLOADING DATASETS

In Driverless AI, you can download datasets from the Datasets Overview
page.

To download a dataset, click on the dataset or select the [CLICK FOR
ACTIONS] button beside the dataset that you want to download, and then
select DOWNLOAD from the submenu that appears.

NOTE: The option to download datasets will not be available if the

enable_dataset_downloadingoption is set tofalse` when starting
Driverless AI. This option can be specified in the :ref:`config.toml
<sample-configtoml> file.

AUTHENTICATION METHODS

configuration options in the config.toml file. NOTES:
-   You can enable multiple authentication methods with the
    additional_authentication_methods config.toml setting. These are
    enabled alongside the default method specified with the
    authentication_method config.toml setting. Login forms for each
    additional method are available on the
    /login/<authentication_method> path. -   If multiple authentication methods are enabled, each method must be
    set up so that it results in the same username to provide access to
    the same resources. -   Driverless AI is also integrated with IBM Spectrum Conductor and
    supports authentication from Conductor.

ABOUT VERSION SUPPORT

Each X.Y.Z long-term support (LTS) release of Driverless AI is supported
for 18 months. For example, the end of support date for 1.10.4 is April
13, 2024, which is 18 months after the release date of October 13, 2022.
Note that the end of support date for each base version is also applied
to each X.Y.Z.{1,2,3...} release.

To view end of support dates for recent DAI LTS releases, see the
Driverless AI prior releases page.

INSTALLING THE R CLIENT

The R Client is available on the Driverless AI UI and from the command
line. The installation process includes downloading the R client and
then installing the source package. Prerequisites
The R client requires R version 3.3 or greater. In addition, the
following R packages must be installed:
-   Rcurl
-   jsonlite
-   rlang
-   methods

Download the R Client
The R Client can be downloaded from within Driverless AI or from the
command line. Download from UI
On the Driverless AI top menu, select the RESOURCES > R CLIENT link. This downloads the DAI_<VERSION>.TAR.GZ file. []
Download from Command Line
The Driverless AI R client is exposed as the /clients/ HTTP end point.

ENABLE_CONSTANT_MODEL``

CONSTANT MODELS

Specify whether to enable constant models <constant_models>. This is set
to AUTO (enabled) by default.

enable_decision_tree------------------------  .. container:: dropdown     **Decision Tree Models**     Specify whether to build Decision Tree models as part of the    experiment. This is set to **Auto** by default. In this case,    Driverless AI will build Decision Tree models if interpretability is    greater than or equal to the value ofdecision_tree_interpretability_switch(which defaults to 7) and    accuracy is less than or equal todecision_tree_accuracy_switch``

    (which defaults to 7).

ENABLE_GLM``

GLM MODELS

Specify whether to build GLM models (generalized linear models) as part
of the experiment (usually only for the final model unless it's used
exclusively). GLMs are very interpretable models with one coefficient
per feature, an intercept term and a link function. This is set to AUTO
by default (enabled if accuracy <= 5 and interpretability >= 6).

ENABLE_XGBOOST_GBM``

XGBOOST GBM MODELS

Specify whether to build XGBoost models as part of the experiment (for
both the feature engineering part and the final model). XGBoost is a
type of gradient boosting method that has been widely successful in
recent years due to its good regularization techniques and high
accuracy. This is set to AUTO by default. In this case, Driverless AI
will use XGBoost unless the number of rows * columns is greater than a
threshold. This threshold is a config setting that is 100M by default
for CPU and 30M by default for GPU.

ENABLE_LIGHTGBM``

LIGHTGBM MODELS

Specify whether to build LightGBM models as part of the experiment.
LightGBM Models are the default models. This is set to AUTO (enabled) by
default.

ENABLE_XGBOOST_DART``

XGBOOST DART MODELS

Specify whether to use XGBoost's Dart method when building models for
experiment (for both the feature engineering part and the final model).
This is set to AUTO (disabled) by default.

enable_xgboost_rapids-------------------------  .. container:: dropdown     **Enable RAPIDS-cuDF extensions to XGBoost GBM/Dart**     Specify whether to enable RAPIDS extensions to XGBoost GBM/Dart. **If    selected, python scoring package can only be used on GPU system**.    The equivalent config.toml parameter isenable_xgboost_rapids`` and

    the default value is False. Disabled for dask multinode models due
    to bug in dask_cudf and xgboost.

ENABLE_XGBOOST_RF``

ENABLE XGBOOST RF MODEL

Specify whether to enable XGBoost RF mode without early stopping. This
setting is disabled unless switched on.

enable_xgboost_gbm_dask---------------------------  .. container:: dropdown     **Enable Dask_cuDF (multi-GPU) XGBoost GBM**     Specify whether to enable Dask_cudf (multi-GPU) version of XGBoost    GBM. Disabled unless switched on. Only applicable for single final    model without early stopping. **No Shapley possible**. The equivalent    config.toml parameter isenable_xgboost_gbm_dask`` and the default

    value is "auto".

enable_xgboost_dart_dask----------------------------  .. container:: dropdown     **Enable Dask_cuDF (multi-GPU) XGBoost Dart**     Specify whether to enable Dask_cudf (multi-GPU) version of XGBoost    GBM/Dart. This option is disabled unless switched on. Only applicable    for single final model without early stopping. **No Shapley is    possible**. The equivalent config.toml parameter isenable_xgboost_dart_daskand the default value is "auto". It is    recommended to run Dask_cudf on multi gpus; if for say debugging    purposes, user would like to enable them on 1 GPU, then setuse_dask_for_1_gpu``
to True via config.toml setting.

enable_lightgbm_dask------------------------  .. container:: dropdown     **Enable Dask (multi-node) LightGBM**     Specify whether to enable multi-node LightGBM. It is disabled by    default unless switched on. The equivalent config.toml parameter isenable_lightgbm_dask``
and default value is "auto".

To enable multinode Dask see
Dask Multinode Training <dask-multinode-training>.

enable_hyperopt_dask------------------------  .. container:: dropdown     **Enable Dask (multi-node/multi-GPU) hyperparameter search**     Specify whether to enable Dask (multi-node/multi-GPU) version of    hyperparameter search. "auto" and "on" are same currently. Dask mode    for hyperparameter search is enabled if:        1) Have a :ref:`Dask multinode cluster <dask-multinode-training>`          or multi-GPU node and model uses 1 GPU for each model( see          :ref:`num-gpus-per-model`).       2) Not already using a Dask model.     The equivalent config.toml parameter isenable_hyperopt_dask`` and

    the default value is "auto".

num_inner_hyperopt_trials_prefinal--------------------------------------  .. container:: dropdown     **Number of trials for hyperparameter optimization during model    tuning only**     Specify the number of trials for **Optuna** hyperparameter    optimization for tuning and evolution of models. If using **RAPIDS**    or **DASK**, this parameter specifies the number of trials for    hyperparameter optimization within XGBoost GBM/Dart and LightGBM and    hyperparameter optimization keeps data on GPU entire time. 0 means no trials. For small data, 100 is fine, while for larger data    smaller values are reasonable if need results quickly. If using    RAPIDS or DASK, hyperparameter optimization stays on GPU the entire    time. The equivalent config.toml parameter isnum_inner_hyperopt_trials_prefinal`` and the default value is
    0.

num_inner_hyperopt_trials_final-----------------------------------  .. container:: dropdown     **Number of trials for hyperparameter optimization for final model    only**     Number of trials for **Optuna** hyperparameter optimization for final    models. If using **RAPIDS** or **DASK**, this is number of trials for    rapids-cudf hyperparameter optimization within XGBoost GBM/Dart and    LightGBM, and hyperparameter optimization keeps data on GPU entire    time.     0 means no trials.For small data, 100 is ok choice, while for larger    data smaller values are reasonable if need results quickly. This    setting applies to final model only, even if    num_inner_hyperopt_trials=0. The equivalent config.toml parameter isnum_inner_hyperopt_trials_final``
and the default value is 0.

num_hyperopt_individuals_final----------------------------------  .. container:: dropdown     **Number of individuals in final ensemble to use Optuna on**     Number of individuals in final model (all folds/repeats for given    base model) to optimize with **Optuna** hyperparameter tuning. The    default value is -1, means all. 0 is same as choosing no Optuna    trials. Might be only beneficial to optimize hyperparameters of best    individual (i.e. value of 1) in ensemble.     The default value is -1, means all. The equivalent config.toml    parameter isnum_hyperopt_individuals_final``

optuna_pruner-----------------  .. container:: dropdown     **Optuna Pruners**     `Optuna    Pruner <https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/003_efficient_optimization_algorithms.html#pruning-algorithms>`__    algorithm to use for early stopping of unpromising trials (applicable    to XGBoost and LightGBM that support Optuna callbacks). The default    is **MedianPruner**. To disable choose None.     The equivalent config.toml parameter isoptuna_pruner``

optuna_sampler------------------  .. container:: dropdown     **Optuna Samplers**     `Optuna    Sampler <https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/003_efficient_optimization_algorithms.html#sampling-algorithms>`__    algorithm to use for narrowing down and optimizing the search space    (applicable to XGBoost and LightGBM that support Optuna callbacks).    The default is **TPESampler**. To disable choose None.     The equivalent config.toml parameter isoptuna_sampler``

enable_xgboost_hyperopt_callback------------------------------------  .. container:: dropdown     **Enable Optuna XGBoost Pruning callback**     Specify whether to enable Optuna's XGBoost Pruning callback to abort    unpromising runs. This is True by default. This not is enabled when    tuning learning rate.     The equivalent config.toml parameter isenable_xgboost_hyperopt_callback``

enable_lightgbm_hyperopt_callback-------------------------------------  .. container:: dropdown     **Enable Optuna LightGBM Pruning callback**     Specify whether to enable Optuna's LightGBM Pruning callback to abort    unpromising runs. This is True by default. This not is enabled when    tuning learning rate.     The equivalent config.toml parameter isenable_lightgbm_hyperopt_callback``

ENABLE_TENSORFLOW``

TENSORFLOW MODELS

Specify whether to build TensorFlow models as part of the experiment
(usually only for text features engineering and for the final model
unless it's used exclusively). Enable this option for NLP experiments.
This is set to AUTO by default (not used unless the number of classes is
greater than 10).

TensorFlow models are not yet supported by Java MOJOs (only Python
scoring pipelines and C++ MOJOs are supported).

ENABLE_GROWNET``

PYTORCH GROWNET MODELS

Specify whether to enable PyTorch-based GrowNet <grownet> models. By
default, this parameter is set to AUTO i.e Driverless decides internally
whether to use the algorithm for the experiment. Set it to _on_ to force
the experiment to build a GrowNet model.

ENABLE_FTRL``

FTRL MODELS

Specify whether to build Follow the Regularized Leader (FTRL) models as
part of the experiment. Note that MOJOs are not yet supported (only
Python scoring pipelines). FTRL supports binomial and multinomial
classification for categorical targets, as well as regression for
continuous targets. This is set to AUTO (disabled) by default.

ENABLE_RULEFIT``

RULEFIT MODELS

Specify whether to build RuleFit models as part of the experiment. Note
that MOJOs are not yet supported (only Python scoring pipelines). Note
that multiclass classification is not yet supported for RuleFit models.
Rules are stored to text files in the experiment directory for now. This
is set to AUTO (disabled) by default.

ENABLE_ZERO_INFLATED_MODELS``

ZERO-INFLATED MODELS

Specify whether to enable the automatic addition of
zero-inflated models <zero-inflated-model> for regression problems with
zero-inflated target values that meet certain conditions:

    y >= 0, y.std() > y.mean()")

This is set to AUTO by default.

ENABLE_LIGHTGBM_BOOSTING_TYPES``

LIGHTGBM BOOSTING TYPES

Specify which boosting types to enable for LightGBM. Select one or more
of the following:

-   GBDT: Boosted trees
-   RF_EARLY_STOPPING: Random Forest with early stopping
-   RF: Random Forest
-   DART: Dropout boosted trees with no early stopping

GBDT and RF are both enabled by default.

ENABLE_LIGHTGBM_CAT_SUPPORT``

LIGHTGBM CATEGORICAL SUPPORT

Specify whether to enable LightGBM categorical feature support. This is

NOTES:

-   Only supported for CPU.
-   A MOJO is not built when this is enabled.

ENABLE_LIGHTGBM_CUDA_SUPPORT``

LIGHTGBM CUDA SUPPORT

Specify whether to enable LightGBM CUDA implementation instead of
OpenCL. LightGBM CUDA is supported on Linux x86-64 environments.

SHOW_CONSTANT_MODEL``

WHETHER TO SHOW CONSTANT MODELS IN ITERATION PANEL

Specify whether to show constant models in the iteration panel. This is

PARAMS_TENSORFLOW``

PARAMETERS FOR TENSORFLOW

Specify specific parameters for TensorFlow to override Driverless AI
parameters. The following is an example of how the parameters can be
configured:

    params_tensorflow = '{'lr': 0.01, 'add_wide': False, 'add_attention': True, 'epochs': 30,
    'layers': [100, 100], 'activation': 'selu', 'batch_size': 64, 'chunk_size': 1000, 'dropout': 0.3,
    'strategy': 'one_shot', 'l1': 0.0, 'l2': 0.0, 'ort_loss': 0.5, 'ort_loss_tau': 0.01, 'normalize_type': 'streaming'}'

The following is an example of how layers can be configured:

    [500, 500, 500], [100, 100, 100], [100, 100], [50, 50]

More information about TensorFlow parameters can be found in the Keras
documentation. Different strategies for using TensorFlow parameters can
be viewed here.

MAX_NESTIMATORS``

MAX NUMBER OF TREES/ITERATIONS

Specify the upper limit on the number of trees (GBM) or iterations
(GLM). This defaults to 3000. Depending on accuracy settings, a fraction
of this limit will be used.

n_estimators_list_no_early_stopping---------------------------------------  .. container:: dropdown     **n_estimators List to Sample From for Model Mutations for Models    That Do Not Use Early Stopping**     For LightGBM, the dart and normal random forest modes do not use    early stopping. This setting lets you specify then_estimators``

    (number of trees in the forest) list to sample from for model
    mutations for these types of models.

MIN_LEARNING_RATE_FINAL``

MINIMUM LEARNING RATE FOR FINAL ENSEMBLE GBM MODELS

This value defaults to 0.01. This is the lower limit on learning rate
for final ensemble GBM models.In some cases, the maximum number of
trees/iterations is insufficient for the final learning rate, which can
lead to no early stopping getting triggered and poor final model
performance. Then, one can try increasing the learning rate by raising
this minimum, or one can try increasing the maximum number of
trees/iterations.

MAX_LEARNING_RATE_FINAL``

MAXIMUM LEARNING RATE FOR FINAL ENSEMBLE GBM MODELS

Specify the maximum (upper limit) learning rate for final ensemble GBM
models. This value defaults to 0.05.

MAX_NESTIMATORS_FEATURE_EVOLUTION_FACTOR``

REDUCTION FACTOR FOR MAX NUMBER OF TREES/ITERATIONS DURING FEATURE
EVOLUTION

Specify the factor by which the value specified by the
max-trees-iterations setting is reduced for tuning and feature
evolution. This option defaults to 0.2. So by default, Driverless AI
will produce no more than 0.2 * 3000 trees/iterations during feature
evolution.

MAX_ABS_SCORE_DELTA_TRAIN_VALID``

MAX. ABSOLUTE DELTA BETWEEN TRAINING AND VALIDATION SCORES FOR TREE
MODELS

Modify early stopping behavior for tree-based models (LightGBM,
XGBoostGBM, CatBoost) such that training score (on training data, not
holdout) and validation score differ no more than this absolute value
(i.e., stop adding trees once abs(train_score - valid_score) >
max_abs_score_delta_train_valid). Keep in mind that the meaning of this
value depends on the chosen scorer and the dataset (i.e., 0.01 for
LogLoss is different than 0.01 for MSE). This option is Experimental,
and only for expert use to keep model complexity low. To disable, set to
0.0. By default this option is disabled.

MAX_REL_SCORE_DELTA_TRAIN_VALID``

MAX. RELATIVE DELTA BETWEEN TRAINING AND VALIDATION SCORES FOR TREE
MODELS

holdout) and validation score differ no more than this relative value
max_rel_score_delta_train_valid * abs(train_score)). Keep in mind that
the meaning of this value depends on the chosen scorer and the dataset
(i.e., 0.01 for LogLoss is different than 0.01 for MSE etc). This option
is Experimental, and only for expert use to keep model complexity low.
To disable, set to 0.0. By default this option is disabled.

MIN_LEARNING_RATE``

MINIMUM LEARNING RATE FOR FEATURE ENGINEERING GBM MODELS

Specify the minimum learning rate for feature engineering GBM models.
This value defaults to 0.05.

MAX_LEARNING_RATE``

MAX LEARNING RATE FOR TREE MODELS

Specify the maximum learning rate for tree models during feature
engineering. Higher values can speed up feature engineering but can hurt
accuracy. This value defaults to 0.5.

MAX_EPOCHS``

MAX NUMBER OF EPOCHS FOR TENSORFLOW/FTRL

When building TensorFlow or FTRL models, specify the maximum number of
epochs to train models with (it might stop earlier). This value defaults
to 10. This option is ignored if TENSORFLOW MODELS and/or FTRL MODELS is
disabled.

max_max_depth-----------------  .. container:: dropdown     **Max Tree Depth**     Specify the maximum tree depth. The corresponding maximum value formax_leaves`` is double the specified value. This value defaults to

    12.

max_max_bin---------------  .. container:: dropdown     **Max max_bin for Tree Features**     Specify the maximummax_bin`` for tree features. This value

    defaults to 256.

RULEFIT_MAX_NUM_RULES``

MAX NUMBER OF RULES FOR RULEFIT

Specify the maximum number of rules to be used for RuleFit models. This
defaults to -1, which specifies to use all rules.

ENSEMBLE_META_LEARNER``

ENSEMBLE LEVEL FOR FINAL MODELING PIPELINE

Model to combine base model predictions, for experiments that create a
final pipeline consisting of multiple base models:

-   blender: Creates a linear blend with non-negative weights that add
    to 1 (blending) - recommended
-   extra_trees: Creates a tree model to non-linearly combine the base
    models (stacking) - experimental, and recommended to also set enable
    cross_validate_meta_learner.

fixed_ensemble_level------------------------  .. container:: dropdown     **Ensemble Level for Final Modeling Pipeline**     Specify one of the following ensemble levels:     -  -1 = auto, based upon ensemble_accuracy_switch, accuracy, size of       data, etc. (Default)    -  0 = No ensemble, only final single model on validated       iteration/tree count. Note that holdout predicted probabilities       will not be available. (For more information, refer to this       :ref:`FAQ <predicted-probs>`.)    -  1 = 1 model, multiple ensemble folds (cross-validation)    -  2 = 2 models, multiple ensemble folds (cross-validation)    -  3 = 3 models, multiple ensemble folds (cross-validation)    -  4 = 4 models, multiple ensemble folds (cross-validation)     The equivalent config.toml parameter isfixed_ensemble_level``.

CROSS_VALIDATE_META_LEARNER``

If enabled, use cross-validation to create an ensemble for the meta
learner itself. Especially recommended for
ensemble_meta_learner='extra_trees', to make unbiased training holdout
predictions. No MOJO will be created if this setting is enabled. Not
needed for ensemble_meta_learner='blender'.

CROSS_VALIDATE_SINGLE_FINAL_MODEL``

CROSS-VALIDATE SINGLE FINAL MODEL

Driverless AI normally produces a single final model for low accuracy
settings (typically, less than 5). When the CROSS-VALIDATE SINGLE FINAL
MODEL option is enabled (default for regular experiments), Driverless AI
will perform cross-validation to determine optimal parameters and early
stopping before training the final single modeling pipeline on the
entire training data. The final pipeline will build N + 1 models, with
N-fold cross validation for the single final model. This also creates
holdout predictions for all non-time-series experiments with a single
final model.

Note that the setting for this option is ignored for time-series
experiments or when a validation dataset is provided.

PARAMETER_TUNING_NUM_MODELS``

NUMBER OF MODELS DURING TUNING PHASE

Specify the number of models to tune during pre-evolution phase. Specify
a lower value to avoid excessive tuning, or specify a higher to perform
enhanced tuning. This option defaults to -1 (auto).

IMBALANCE_SAMPLING_METHOD``

SAMPLING METHOD FOR IMBALANCED BINARY CLASSIFICATION PROBLEMS
Specify the sampling method for imbalanced binary classification
problems. This is set to OFF by default. Choose from the following
options:
-   AUTO: sample both classes as needed, depending on data
-   OVER_UNDER_SAMPLING: over-sample the minority class and under-sample
    the majority class, depending on data
-   UNDER_SAMPLING: under-sample the majority class to reach class
    balance
-   OFF: do not perform any sampling
This option is closely tied with the Imbalanced Light GBM and Imbalanced
XGBoost GBM models, which can be enabled/disabled on the Recipes tab
under included_models. Specifically:
-   If this option is ENABLED (set to a value other than OFF) and the
    ImbalancedLightGBM and/or ImbalancedXGBoostGBM models are ENABLED,
    then Driverless AI will check your target imbalance fraction.

IMBALANCE_SAMPLING_THRESHOLD_MIN_ROWS_ORIGINAL``

THRESHOLD FOR MINIMUM NUMBER OF ROWS IN ORIGINAL TRAINING DATA TO ALLOW
IMBALANCED SAMPLING

Specify a threshold for the minimum number of rows in the original
training data that allow imbalanced sampling. This value defaults to
100,000.

IMBALANCE_RATIO_SAMPLING_THRESHOLD``

RATIO OF MAJORITY TO MINORITY CLASS FOR IMBALANCED BINARY CLASSIFICATION
TO TRIGGER SPECIAL SAMPLING TECHNIQUES (IF ENABLED)

For imbalanced binary classification problems, specify the ratio of
majority to minority class. Special imbalanced models with sampling
techniques are enabled when the ratio is equal to or greater than the
specified ratio. This value defaults to 5.

HEAVY_IMBALANCE_RATIO_SAMPLING_THRESHOLD``

RATIO OF MAJORITY TO MINORITY CLASS FOR HEAVILY IMBALANCED BINARY
CLASSIFICATION TO ONLY ENABLE SPECIAL SAMPLING TECHNIQUES (IF ENABLED)

For heavily imbalanced binary classification, specify the ratio of the
majority to minority class equal and above which to enable only special
imbalanced models on the full original data without upfront sampling.
This value defaults to 25.

IMBALANCE_SAMPLING_NUMBER_OF_BAGS``

NUMBER OF BAGS FOR SAMPLING METHODS FOR IMBALANCED BINARY CLASSIFICATION
(IF ENABLED)

Specify the number of bags for sampling methods for imbalanced binary
classification. This value defaults to -1.

IMBALANCE_SAMPLING_MAX_NUMBER_OF_BAGS``

HARD LIMIT ON NUMBER OF BAGS FOR SAMPLING METHODS FOR IMBALANCED BINARY
CLASSIFICATION

Specify the limit on the number of bags for sampling methods for
imbalanced binary classification. This value defaults to 10.

IMBALANCE_SAMPLING_MAX_NUMBER_OF_BAGS_FEATURE_EVOLUTION``

CLASSIFICATION DURING FEATURE EVOLUTION PHASE

imbalanced binary classification. This value defaults to 3. Note that
this setting only applies to shift, leakage, tuning, and feature
evolution models. To limit final models, use the HARD LIMIT ON NUMBER OF
BAGS FOR SAMPLING METHODS FOR IMBALANCED BINARY CLASSIFICATION setting.

IMBALANCE_SAMPLING_MAX_MULTIPLE_DATA_SIZE``

MAX SIZE OF DATA SAMPLED DURING IMBALANCED SAMPLING

Specify the maximum size of the data sampled during imbalanced sampling
in terms of the dataset's size. This setting controls the approximate
number of bags and is only active when the "Hard limit on number of bags
for sampling methods for imbalanced binary classification during feature
evolution phase" option is set to -1. This value defaults to 1.

IMBALANCE_SAMPLING_TARGET_MINORITY_FRACTION``

TARGET FRACTION OF MINORITY CLASS AFTER APPLYING UNDER/OVER-SAMPLING
TECHNIQUES

Specify the target fraction of a minority class after applying
under/over-sampling techniques. A value of 0.5 means that
models/algorithms will be given a balanced target class distribution.
When starting from an extremely imbalanced original target, it can be
advantageous to specify a smaller value such as 0.1 or 0.01. This value
defaults to -1.

FTRL_MAX_INTERACTION_TERMS_PER_DEGREE``

MAX NUMBER OF AUTOMATIC FTRL INTERACTIONS TERMS FOR 2ND, 3RD, 4TH ORDER
INTERACTIONS TERMS (EACH)

Samples the number of automatic FTRL interactions terms to no more than
this value (for each of 2nd, 3rd, 4th order terms). This value defaults
to 10000

ENABLE_BOOTSTRAP``

WHETHER TO ENABLE BOOTSTRAP SAMPLING FOR VALIDATION AND TEST SCORES

Specify whether to enable bootstrap sampling. When enabled, this setting
provides error bars to validation and test scores based on the standard
error of the bootstrap mean. This is enabled by default.

TENSORFLOW_NUM_CLASSES_SWITCH``

FOR CLASSIFICATION PROBLEMS WITH THIS MANY CLASSES, DEFAULT TO
TENSORFLOW

Specify the number of classes above which to use TensorFlow when it is
enabled. Others model that are set to AUTO will not be used above this
number. (Models set to ON, however, are still used.) This value defaults
to 10.

PREDICTION_INTERVALS``

COMPUTE PREDICTION INTERVALS

Specify whether to compute empirical prediction intervals based on
holdout predictions. This is enabled by default.

PREDICTION_INTERVALS_ALPHA``

CONFIDENCE LEVEL FOR PREDICTION INTERVALS

Specify a confidence level for prediction intervals. This value defaults
to 0.9.

MODEL PERFORMANCE ON ANOTHER DATASET

The DIAGNOSE MODEL ON NEW DATASET option lets you view model performance
for multiple scorers based on existing model and dataset. On the completed experiment page, click the DIAGNOSE MODEL ON NEW
DATASET button. NOTE: You can also diagnose a model by selecting DIAGNOSTICS from the
top menu, then selecting an experiment and test dataset. []
Select a dataset to use when diagnosing this experiment. Note that the
dataset must include the target column that is in the original dataset. At this point, Driverless AI will begin calculating all available scores
for the experiment. When the diagnosis is complete, it will be available on the MODEL
DIAGNOSTICS page. Click on the new diagnosis. From this page, you can
download predictions. You can also view scores and metric plots. The
plots are interactive. Click a graph to enlarge. In the enlarged view,
you can hover over the graph to view details for a specific point.

ENVIRONMENT VARIABLES AND CONFIGURATION OPTIONS

Driverless AI provides a number of environment variables that can be
passed when starting Driverless AI or specified in a config.toml file. The complete list of variables is in the config_file section. The steps
for specifying variables vary depending on whether you installed a
Driverless AI RPM, DEB, or TAR SH or whether you are running a Docker
image. Setting Environment Variables and Configuration Options
Each property must be prepended with DRIVERLESS_AI. The example below
starts Driverless AI with environment variables that enable S3 and HDFS
access (without authentication).

DATA LEAKAGE AND SHIFT DETECTION IN DRIVERLESS AI

This page describes data leakage and shift detection in DRIVERLESS AI
(DAI). Overview
-   DATA LEAKAGE: To detect data leakage, DAI runs a model (when
    available, LightGBM) to get the variable importance table, which
    determines the predictive power of each feature on the target
    variable. A simple model is then built on each feature with
    significant variable importance. The models with a high AUC (for
    classification) or R2 (for regression) score are reported to the
    user as potential leak features. -   SHIFT DETECTION: To detect shift in distribution between the
    training, validation or testing datasets, Driverless AI trains a
    binomial model to predict which dataset a row belongs to.

Shifted features can either be dropped or used to
    create more meaningful aggregate features by using them as labels or
    bins. Enabling leakage detection
To enable leakage detection, set the config_check_leakage configuration
option to on (default). When this option is enabled, Driverless AI runs
a model to determine the predictive power of each feature on the target
variable. If leakage detection has been enabled, then the
config_detect_features_leakage_threshold_auc configuration option is
used for per-feature leakage detection if AUC (or R2 for regression) on
original data (label-encoded) is greater-than or equal to the specified
value. By default, this option is set to 0.95. Identifying features responsible for leakage
For significant features (determined by feature importance), a simple
model is built on each feature. The models with a high AUC
(classification) or R2 (regression) score are reported to the user as
potential leaks.

By default, this option is
set to 0.8. Automatically drop features suspected in leakage
A feature is dropped when the single feature model performance exceeds
the threshold for dropping features. You can specify this threshold with
the config_drop_features_leakage_threshold_auc configuration option,
which has a default value of 0.999. When the AUC (or R2 for regression),
GINI, or Spearman correlation is above the specified value, the feature
is dropped. Shift detection
Driverless AI can detect data distribution shifts between
train/valid/test datasets when they are provided. Shift is detected by training a model to distinguish between
train/validation/test datasets by assigning a unique target label to
each of the datasets. If the model turns out to have high accuracy, data
shift is reported with a notification. Shifted features can either be
dropped or used to create more meaningful aggregate features by using
them as labels or bins.

fitted_model.pickle.meta.json`` file in the experiment summary zip
archive.

MOJO SCORING PIPELINES

The MOJO Scoring Pipeline provides a standalone scoring pipeline that
converts experiments to MOJOs, which can be scored in real time. The
MOJO Scoring Pipeline is a scoring engine that can be deployed in any
JAVA environment (Java Runtime) or in PYTHON or R environment (C++
runtime) for scoring in real time or batch. For deployment options see
Deploying the MOJO Pipeline to production <deployment>

scoring-mojo-scoring-pipeline scoring-pipeline-cpp mojo2_javadoc
scoring-klime-mojo-scoring-pipeline

DEPLOY DRIVERLESS AI MODELS

This page describes how to deploy models in Driverless AI. NOTE: AWS Lambda and REST server deployment options are no longer
supported, and have been replaced by deploy_via_triton. Overview
Deploy from the UI
The following deployment options are available directly from the UI:
-   H2O MLOps deployment <deploy_via_mlops> (Note that to use this
    deployment option, you must specify h2o_mlops_ui_url in the
    config.toml file.) -   Triton Inference Server deployment <deploy_via_triton>
Additional deployment options
By default, each completed Driverless AI experiment (unless explicitly
disabled or not available due to modified expert settings) creates at
least one scoring pipeline <Scoring_Pipeline> for scoring in Python,
C++, Java, and R.
The following is a list of deployment options and examples for deploying
Driverless AI MOJO (Java and C++ with Python/R wrappers) and Python
Scoring pipelines for production purposes.

h2o_mlops_ui_urlin theconfig.tomlfile. 1. On the completed experiment page, click **Deploy**. 2. Click the **H2O MLOps** button. This action results in one of the    following outcomes:        -  **The experiment is assigned to a single Project:** You are          redirected to the Project detail page in the H2O MLOps app. -  **The experiment is assigned to multiple Projects:** Select a          project to go to in the H2O MLOps app. Alternatively, create a          new Project to assign the experiment to. If you choose to          create a new Project, you are prompted to enter a name and          description for the Project. Once the new Project has been          created and the experiment has been linked to it, you can click          the **Go to MLOps page** button to navigate to the Project          detail page in the H2O MLOps app. -  **The experiment isn't assigned to any Project:** Select a          Project to link the experiment to.

.. _deploy_via_triton:  Triton Inference Server deployment ----------------------------------  Driverless AI provides a **built-in** Triton Inference Server that can be used for testing, prototyping, or benchmarking. NVIDIA Triton Inference Server <https://github.com/triton-inference-server/server#triton-inference-server>\_ is an open source inference serving software that streamlines AI inferencing. The out-of-the-box built-in Triton Inference Server in Driverless AI is not recommended for production or business-critical deployments. Instead, using an :ref:`external production-grade NVIDIA Triton server <triton_external>` or :ref:`H2O MLOps <deploy_via_mlops>` is recommended. **Notes:**  -  Currently, only CPUs are supported. -  Triton Inference Server deployments cannot be accessed from H2O    MLOps.

-triton_host_remote: Set the hostname or IP address of a remote    Triton inference service (that is, an inference service outside of    DAI). This setting is used whenauto_deploy_triton_scoring_pipelineandmake_triton_scoring_pipelineare not disabled. If this option is    specified set, ensure thattriton_model_repository_dir_remoteandtriton_server_params_remoteare also set. -triton_model_repository_dir_remote: Set the path to the model    repository directory for a remote Triton inference server outside of    Driverless AI. All Triton deployments for all users are stored in    this directory. Requires write access to this directory from    Driverless AI (shared file system). Note that this setting is    optional. If this setting isn't set, each model deployment is    uploaded over the gRPC protocol, but note that this only works for    models that are smaller than 2GB due to protobuf limits.

-triton_server_params_remote: Set the parameters to connect to a    remote Triton server. This option is only required whentriton_host_remoteandtriton_model_repository_dir_remoteare    set. Built-in Triton inference server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  -enable_triton_server_local: Specify whether to enable the    built-in Triton inference server. If this is set to false, you can    still connect to a remote Triton inference server by settingtriton_host. If this is set to true, the built-in Triton    inference server starts. -triton_host_local: Set the hostname or IP address of a built-in    Triton inference service, to be used whenauto_deploy_triton_scoring_pipelineandmake_triton_scoring_pipelineare not disabled. Only needed ifenable_triton_server_localis disabled. This option is required    to be set for some systems like AWS in order for networking packages    to reach the server.

-triton_model_repository_dir_local: Set the path to the model    repository (relative todata_directory) for local Triton    inference server built-in to Driverless AI. Note that all Triton    deployments for all users are stored in this directory. -triton_server_core_chunk_size_local``: Specify the number of cores
    to specify as resource so that C++ MOJO can use its own
    multi-threaded parallel row batching to save memory and increase
    performance. A value of 1 is most portable across any Triton server,
    and is the most efficient use of resources for small (for example,
    1) batch sizes, while 4 is reasonable default (assuming requests are
    batched). DAI DEPLOYMENT WIZARD

The following steps describe how to use the DAI Deployment Wizard to
create and manage Triton Inference Server deployments. 1. On the completed experiment page, click the DEPLOY button. 2. Click the TRITON button. (Note that this step may not be required on
    standalone DAI.) 3.

This Wizard lets you deploy
    Driverless AI experiments to an inference server, to get low-latency
    scoring capabilities for development, prototyping, demos or
    benchmarking. 4. Select at least one of the following, then click the CONTINUE button
    to proceed. -   Predicted target values
      -   Shapley values (reason codes) for engineered features
      -   Shapley values (reason codes) for original features

Deployed models list
A list of all models that have been previously deployed is displayed. They can be active (ready) or inactive (unavailable) deployments. A
deployment can be unloaded if it is not being used temporarily to free
up resources. For each deployed model in this list, the following details are
provided:
-   Model name: Click the name of a model to go to the completed
    experiment page for that model.

-   THROUGHPUT: The approximate number of rows that can be scored in 1
    second when scoring many rows at a time in a batch request. -   SIZE: The size of the MOJO pipeline on disk. - MEMORY USAGE: The size of the MOJO pipeline in memory during scoring
(at the time the MOJO was created). Note that this number may change as
newer MOJO runtimes are released. -   STATE: The following list descibes the various possible states of a
    deployment. -   READY: The deployment is loaded and active (ready to make
          predictions). -   UNLOADING: This indicates a transition state between READY and
          UNAVAILABLE. -   UNAVAILABLE: This indicates an unloaded deployment that can be
          loaded again when needed. -   DETAILS: Displays the name, description, platform, model version,
    input columns, output columns, P99 latency, throughput, and state of
    the deployed model.

-   LOAD ALL: Loads all deployed models that are not in "READY" state,
    and run a very quick benchmark test. In the context of the Triton
    Inference Server, loading and unloading refer to the process of
    adding or removing a model to or from the server's model repository. Once a model is loaded, it is ready to serve inference requests. -   BENCHMARK ALL: Run a benchmark for all of the listed deployed
    models. Note that Shapley values are noticeably slower than regular
    predictions. -   UNLOAD ALL: Unloads all of the deployed models. Unloading a model
    from the Triton Inference Server involves removing the model from
    the server's model repository, which frees up resources that can be
    used by other models. This lets you save memory without having to
    delete models. Models that have been unloaded can be loaded again. -   DELETE ALL: Delete all the listed deployed models. Exercise caution
    when using this option.

-   BACK: Return to the completed experiment page. []

Actions available from the deployment Details page
The following list describes the actions that are available from the
deployment Details page. You can access these actions by clicking
DETAILS next to a specific deployed model in the list of deployed
models. -   QUERY:
-   BENCHMARK:
-   LOAD / UNLOAD: In the context of the Triton Inference Server,
    loading and unloading refer to the process of adding or removing a
    model to or from the server's model repository. You can use this
    button to load or unload a specific model. -   DELETE: Unloads the model and deletes the local directory (only for
    DAI-internal Triton models). -   MODIFY: Redeploys the model. You can choose to deploy any
    combination of Target Predictions/Shapley Values/Original Shapley
    Values. -   EXPORT: Lets you download a zip file of the contents of the folder
    inside any Triton repository, anywhere.

[]
MOJO With Java Runtime deployment options
The following are several options for deploying Driverless AI MOJO with
Java Runtime. The links in the diagram lead to code examples and
templates.

Deployment options from within Driverless AI GUI <deploy_from_gui>. MOJO With C++ Runtime deployment options
Here we list some example scenarios and platforms for deploying
Driverless AI MOJO with C++ Runtime. MOJO C++ runtime can also be run
directly from R/PYTHON terminals. For more information, see
cpp_scoring_pipeline.

CONFIGURATION SECURITY

keeping security settings as clear text in the config.toml file. Updates to config override chain
The Configuration Override Chain has been updated to load the settings
from the encrypted keystore after the settings are read from the plain
text config.toml file. The Environment Variable can still override the
values from the keystore:
    1. h2oai/config/config.toml
    [Internal, not visible to users]
    2. config.toml
    [Place file in a folder/mount file in docker container and provide path
    in "DRIVERLESS_AI_CONFIG_FILE" environment variable]
    3. Keystore file
    [Set keystore_file parameter in config.toml or environment variable
    "DRIVERLESS_AI_KEYSTORE_FILE" to point to a valid DAI keystore file 
    generated using the h2oai.keystore tool.

Environment variable
    [Configuration variables can also be provided as environment 
    variables. They must have the prefix "DRIVERLESS_AI_" followed
    by the variable name in caps. For example, "authentication_method"
    can be provided as "DRIVERLESS_AI_AUTHENTICATION_METHOD"]

Keystore setup workflow
CREATING THE KEYSTORE
Although the keystore file can contain any configuration parameter
supported by the config.toml, it is recommended to store only config
parameters that contain secure/sensitive information in the keystore
file and use the regular config.toml file for other config parameters. STEP 1: CREATE A CLEARTEXT CONFIG SUBSET
To start, create a file CONFIG.CLEAR that follows the TOML syntax of a
regular config.toml file and contains the config parameters that you
want to store securely.

h2oai.keystoretool:  -  The keystore tool needs to be run asrootand within the context    of Driverless AI Python environment provided by thedai-env.shscript. -  Theadd-keyscommand accepts the path to keystore as the first    argument and the clear text config.toml subset as the second. -  If the keystore does not exist, it is created. -  All keys in theconfig.clearare either Inserted or Updated in    the keystore. If a key already exists in the key store, it is    updated. If the keystore contains any keys that are not inconfig.clear, they are not altered. -  Once the keystore file is created, it is recommended to ensure the    following:     -  Ownership is with root user with read and write permissions. -  Change group ownership to the Driverless group (or the appropriate       ID that matches the group ID with which the Driverless processes       run in your system) with read only permissions. No other user or       group should have read access to this file.

::     (user1) $ sudo /bin/bash    # this will get a shell as root.

-list-keys: Lists the keys stored in the Driverless AI keystore. Their values are never displayed. -delete-keys``
Removes the specified key from the keystore. (root) # /opt/h2oai/dai/dai-env.sh python -m h2oai.keystore --help
    ======================================================================
    Usage: python -m h2oai.keystore [OPTIONS] COMMAND [ARGS]...
    Options:
     --help  Show this message and exit. Commands:
     add-keys     Adds/Update DAI secrets keystore (KEYSTORE_PATH) with config...
     delete-keys  Deleted keys provided as -k key1 -k key2 from the keystore...
     list-keys    Lists keys stored in Driverless AI KEYSTORE.

DRIVERLESS AI MOJO SCORING PIPELINE - JAVA RUNTIME (WITH SHAPLEY CONTRIBUTION)

For completed experiments, Driverless AI automatically converts models
to MOJOs (Model Objects, Optimized). The MOJO Scoring Pipeline is a
scoring engine that can be deployed in any Java environment for scoring
in real time. (For information on the C++ scoring runtime with Python
and R wrappers, see
H2O MOJO C++ scoring pipeline <cpp_scoring_pipeline>.) For info on the
available deployment options, see H2O MOJO Deployment <deployment>. MOJOs are tied to experiments. Experiments and MOJOs are not
automatically upgraded when Driverless AI is upgraded. NOTES:
-   This scoring pipeline is not currently available for TensorFlow,
    BERT, RuleFit or Image <deploy-image> models. TensorFlow/Bert are
    supported by C++ Runtime. -   To disable the automatic creation of this scoring pipeline, set the
    MAKE MOJO SCORING PIPELINE expert setting to OFF while building an

-   Shapley contributions come with the downloaded experiment MOJO
    scoring pipeline (Java runtime). -   Shapley contributions <quick-run> for TRANSFORMED FEATURES and
    ORIGINAL FEATURES are currently AVAILABLE for XGBoost (GBM, GLM, RF,
    DART), LightGBM, Zero-Inflated, Imbalanced and DecisionTree models
    (and their ensemble). For ensemble with ExtraTrees meta learner
    (ensemble_meta_learner='extra_trees') models, we suggest to use the
    MLI Python scoring package. Download
Because the Java MOJO runtime is backward compatible, we recommend using
the latest available version. You can download the latest Java MOJO
runtime from https://mvnrepository.com/artifact/ai.h2o/mojo2-runtime. A Quick run
To get a quick output from the downloaded MOJO scoring pipeline in the
console on the example test set:
-   Make sure Java7 or later is installed.

java -Xmx5g -Dai.h2o.mojos.runtime.license.file=license.file -jar mojo2-runtime.jar pipeline.mojo example.csv
-   To get SHAPLEY CONTRIBUTIONS for TRANSFORMED FEATURES run using the
    flag --show-contributions
          java -Xmx5g -Dai.h2o.mojos.runtime.license.file=license.file -jar mojo2-runtime.jar --show-contributions pipeline.mojo example.csv
-   To get SHAPLEY CONTRIBUTIONS for ORIGINAL FEATURES run using the
    flag --show-contributions-original
          java -Xmx5g -Dai.h2o.mojos.runtime.license.file=license.file -jar mojo2-runtime.jar --show-contributions-original pipeline.mojo example.csv
-   To get the prediction interval from the MOJO, run using the flag
    --with-prediction-interval
          java -Xmx5g -Dai.h2o.mojos.runtime.license.file=license.file -jar mojo2-runtime.jar --with-prediction-interval pipeline.mojo example.csv
See run-example.sh for reference. Bigger test files/MOJOs may require
more memory (Xmx) to score.

For ensemble with
      ExtraTrees meta learner (ensemble_meta_learner='extra_trees')
      models we suggest to use the MLI Python scoring package. -   In MOJOs, Shapley values for ORIGINAL FEATURES are approximated
      from the accompanying Shapley values for transformed features with
      the Naive Shapley (even split <dai-shapley>) method. -   The Shapley fast approximation <completed_experiment> uses only
      one model (from the first fold) with no more than the first 50
      trees. For details see fast_approx_num_trees and
      fast_approx_do_one_fold_one_model
      config.toml settings <sample-configtoml>. Prerequisites
The following are required in order to run the MOJO scoring pipeline. -   Java 7 runtime (JDK 1.7) or newer. NOTE: We recommend using Java 11+
    due to a bug in Java. (See
    https://bugs.openjdk.java.net/browse/JDK-8186464.) -   Valid Driverless AI license.

Driverless AI requires a license to be specified in order to run the
MOJO Scoring Pipeline.

-   The default resource name can be changed via the JVM system
          property ai.h2o.mojos.runtime.license.filename. For example:
    # Specify the license via a temporary environment variable

MOJO Scoring Pipeline Files
The MOJO-PIPELINE folder includes the following files:
-   RUN_EXAMPLE.SH: An bash script to score a sample test set. -   PIPELINE.MOJO: Standalone scoring pipeline in MOJO format. -   MOJO2-RUNTIME.JAR: MOJO Java runtime. -   EXAMPLE.CSV: Sample test set (synthetic, of the correct format). -   DOT files: Text files that can be rendered as graphs that provide a
    visual representation of the MOJO scoring pipeline (can be edited to
    change the appearance and structure of a rendered graph). -   PNG files: Image files that provide a visual representation of the
    MOJO scoring pipeline.

On the completed Experiment page, click on the DOWNLOAD MOJO SCORING
    PIPELINE button. 2. In the pop-up menu that appears, click on the DOWNLOAD MOJO SCORING
    PIPELINE button once again to download the SCORER.ZIP file for this
    experiment onto your local machine. Refer to the provided
    instructions for Java, Python, or R.
3. Run the following to score all rows in the sample test set with the
    file paths to the test set (example.csv), MOJO pipeline
    (pipeline.mojo) and license (license.sig) stored in environment
    variables TEST_SET_FILE, MOJO_PIPELINE_FILE,
    DRIVERLESS_AI_LICENSE_KEY:
4. Run the following to score a specific test set (example.csv) with
    MOJO pipeline (pipeline.mojo) and the license file (license.sig):
5. To run the Java application for data transformation directly:

MOJO Scoring Command-Line Options

Java Virtual Machine (JVM) Options
    string.

1. Open a new terminal window, create an experiment folder, and change
    directories to that new folder:
    file called DOCSEXAMPLE.JAVA (for example, using
    vim DocsExample.java). Include the following contents. 3. Compile the source code with the files of the MOJO runtime
    (mojo2-runtime.jar) and MOJO pipeline (pipeline.mojo) copied into
    the experiment:
4. Run the MOJO example with the license (license.sig) copied into the
    experiment:
5. The following output is displayed:

Using the MOJO Scoring Pipeline with Spark/Sparkling Water
TOML-based MOJO2. Releases after 1.5 will include protobuf-based MOJO2.

This section
independent of the H2O run-time. Preparing Your Environment
In order use the MOJO scoring pipeline, Driverless AI license has to be
passed to Spark. This can be achieved via --jars argument of the Spark
the license file.

./bin/pyspark --jars license.sig --py-files pysparkling.zip
job to a cluster. # First, specify the dependencies
    from pysparkling.ml import H2OMOJOPipelineModel, H2OMOJOSettings
    # The 'namedMojoOutputColumns' option ensures that the output columns are named properly. # If you want to use old behavior when all output columns were stored inside an array,
    # set it to False.

settings = H2OMOJOSettings(namedMojoOutputColumns = True)
    # Run the predictions. The predictions contain all the original columns plus the predictions
    # added as new columns
    predictions = mojo.transform(dataFrame)
    # You can easily get the predictions for a desired column using the helper function as
    predictions.select(mojo.selectPredictionUDF("AGE")).collect()
Sparkling Water
First, start Spark with Sparkling Water Scala assembly and Driverless AI
license. ./bin/spark-shell --jars license.sig,sparkling-water-assembly.jar
Download page.

DRIVERLESS AI MLI STANDALONE PYTHON SCORING PACKAGE

This package contains an exported model and Python 3.8 source code
examples for productionizing models built using H2O Driverless AI
Machine Learning Interpretability (MLI) tool. This is only available for
interpreted models and can be downloaded by clicking the SCORING
PIPELINE button on the Interpreted Models page. The files in this package let you obtain reason codes for a given row of
data in a couple of different ways:
    module through remote procedure calls (RPC). MLI Python Scoring Package Files
The SCORING-PIPELINE-MLI folder includes the following notable files:
-   EXAMPLE.PY: An example Python script demonstrating how to import and
    interpret new records.

-   RUN_EXAMPLE_SHAPLEY.SH: Runs example_shapley.py. This compares
    K-LIME and Driverless AI Shapley reason codes. -   TCP_SERVER.PY: A standalone TCP server for hosting MLI services. -   HTTP_SERVER.PY: A standalone HTTP server for hosting MLI services. -   RUN_TCP_SERVER.SH: Runs the TCP scoring service (specifically,
    tcp_server.py). -   RUN_HTTP_SERVER.SH: Runs HTTP scoring service (runs http_server.py). -   EXAMPLE_CLIENT.PY: An example Python script demonstrating how to
    communicate with the MLI server. -   EXAMPLE_SHAPLEY.PY: An example Python script demonstrating how to
    compare K-LIME and Driverless AI Shapley reason codes. -   RUN_TCP_CLIENT.SH: Demonstrates how to communicate with the MLI
    service via TCP (runs example_client.py). -   RUN_HTTP_CLIENT.SH: Demonstrates how to communicate with the MLI
There are two methods for starting the MLI Standalone Scoring Pipeline.

Use
this method if:
-   A valid Driverless AI license key. -   A completed Driverless AI experiment. -   Downloaded MLI Scoring Pipeline. Running the MLI Scoring Pipeline - Recommended
1. Download the TAR SH version of Driverless AI from
    https://www.h2o.ai/download/. 2. Use bash to execute the download. This creates a new DAI-NNN folder. 3. Change directories into the new Driverless AI folder. 4. Run the following to install the Python Scoring Pipeline for your
    completed Driverless AI experiment:
5. Run the following command to run the included scoring pipeline
    example:
This section describes an alternative method for running the MLI
Standalone Scoring Pipeline. This version requires Internet access.

Note that
-   Valid Driverless AI license. -   The scoring module and scoring service are supported only on Linux
    at install time and require internet access. Depending on your
    proxy. -   Apache Thrift (to run the scoring service in TCP mode)
Examples of how to install these prerequisites are below.

Refer to the
  links below for more information:
  -   Anaconda - https://docs.anaconda.com/anaconda/install.html
  -   Miniconda - https://docs.conda.io/en/latest/miniconda.html
INSTALLING THE THRIFT COMPILER
Refer to Thrift documentation at
https://thrift.apache.org/docs/BuildingFromSource for more information. sudo apt-get install automake bison flex g++ git libevent-dev \
      libssl-dev libtool make pkg-config libboost-all-dev ant
    wget https://github.com/apache/thrift/archive/0.10.0.tar.gz
    tar -xvf 0.10.0.tar.gz
    cd thrift-0.10.0
    ./bootstrap.sh
    ./configure
    make
    sudo make install
Run the following to refresh the runtime shared after installing Thrift. sudo ldconfig /usr/local/lib
Running the MLI Scoring Pipeline - Alternative Method
1. On the MLI page, click the SCORING PIPELINE button. 2. Unzip the scoring pipeline, and run the following examples in the
    SCORING-PIPELINE-MLI folder.

All the prerequisites for the scoring
module to work correctly are listed in the 'requirements.txt' file. To
use the scoring module, all you have to do is create a Python
virtualenv, install the prerequisites, and then import and use the
scoring module as follows:
    ----- See 'example.py' for complete example. -----
    from scoring_487931_20170921174120_b4066 import Scorer
    scorer = KLimeScorer()       # Create instance. score = scorer.score_reason_codes([  # Call score_reason_codes()
        7.416,              # sepal_len
        3.562,              # sepal_wid
        1.049,              # petal_len
        2.388,              # petal_wid
    ])
The scorer instance provides the following methods:
-   score_reason_codes(list): Get K-LIME reason codes for one row (list
    of values).

-----
    virtualenv -p python3.8 env
    source env/bin/activate
    pip install --use-deprecated=legacy-resolver -r requirements.txt
    python example.py

K-LIME vs Shapley Reason Codes
There are times when the K-LIME model score is not close to the
Driverless AI model score. In this case it may be better to use reason
codes using the Shapley method on the Driverless AI model. Note that the
reason codes from Shapley will be in the transformed feature space. To see an example of using both K-LIME and Driverless AI Shapley reason
codes in the same Python session, run:
    bash run_example_shapley.sh
For this batch script to succeed, MLI must be run on a Driverless AI
model. If you have run MLI in standalone (external model) mode, there
will not be a Driverless AI scoring pipeline. If MLI was run with transformed features, the Shapley example scripts
will not be exported. You can generate exact reason codes directly from
the Driverless AI model scoring pipeline.

pred_contribs=[True|False]`` to get MLI predictions (KLime/Shapley) on a
new dataset. See the EXAMPLE_SHAPLEY.PY file for more information. MLI SCORING SERVICE - TCP MODE (THRIFT)

The TCP mode lets you use the scoring service from any language
supported by Thrift, including C, C++, C#, Cocoa, D, Dart, Delphi, Go,
Haxe, Java, Node.js, Lua, perl, PHP, Python, Ruby and Smalltalk. To start the scoring service in TCP mode, you will need to generate the
Thrift bindings once, then run the server:
    ----- See 'run_tcp_server.sh' for complete example. -----
    thrift --gen py scoring.thrift
    python tcp_server.py --port=9090
Note that the Thrift compiler is only required at build-time. It is not
a run time dependency, i.e. once the scoring services are built and
tested, you do not need to repeat this installation process on the
machines where the scoring services are intended to be deployed.

----- See 'run_tcp_client.sh' for complete example. -----

    ----- See 'example_client.py' for complete example. -----
    socket = TSocket.TSocket('localhost', 9090)
    transport = TTransport.TBufferedTransport(socket)
    protocol = TBinaryProtocol.TBinaryProtocol(transport)
    client = ScoringService.Client(protocol)
    transport.open()
    row = Row()
    row.sepalLen = 7.416  # sepal_len
    row.sepalWid = 3.562  # sepal_wid
    row.petalLen = 1.049  # petal_len
    row.petalWid = 2.388  # petal_wid
    scores = client.score_reason_codes(row)
    transport.close()
You can reproduce the exact same result from other languages, e.g.

This is usually less performant compared to Thrift, but has the
advantage of being usable from any HTTP client library in your language
of choice, without any dependency on Thrift. For JSON-RPC documentation, see http://www.jsonrpc.org/specification . To start the scoring service in HTTP mode:
    ----- See 'run_http_server.sh' for complete example. -----
    python http_server.py --port=9090
To invoke scoring methods, compose a JSON-RPC message and make a HTTP
POST request to http://host:port/rpc as follows:
    ----- See 'run_http_client.sh' for complete example. -----
    curl http://localhost:9090/rpc \
      --header "Content-Type: application/json" \
      --data @- <<EOF
     {
      "id": 1,
      "method": "score_reason_codes",
      "params": {
        "row": [ 7.486, 3.277, 4.755, 2.354 ]
      }
     }
    EOF
Similarly, you can use any HTTP client library to reproduce the above
result.

PRODUCTIONIZING YOUR MODEL

H2O.ai outputs the best model in an experiment. This model can then be
downloaded and then saved to a production environment.

Run the following commands in Python 3.8 to save the displayed model as
a .csv. Note that Python 3.8 is the only supported Python version for
use with H2O.ai.

    ## final pipeline (logic, not state)
    pipe = population[best_id].get_pipe()

    ## final pipeline state, based on LARGE training data
    train_df_munged, y_munged = pipe.fit_transform(train_df, y)
    #train_df_munged.to_csv("munged_amazon_train.csv", index=False)

    ## Load Kaggle test set without response, convert to munged state
    # test = "../../../../h2oai-benchmarks/Data/Amazon/test.csv"
    # test_df = dt.fread(test).topandas()
    test_df = train_df
    test_df_munged = pipe.transform(test_df)
    #test_df_munged.to_csv("munged_amazon_test.csv", index=False)

REFERENCES

Adebayo, Julius A. "Fairml: Toolbox for diagnosing bias in predictive
modeling." Master’s Thesis, MIT, 2016. Breiman, Leo. "Statistical Modeling: The Two Cultures (with comments and
a rejoinder by the author)." _Statistical Science_ 16, no. 3, 2001. Craven, Mark W. and Shavlik, Jude W. "Extracting tree structured
representations of trained networks." _Advances in Neural Information
Processing Systems_, 1996. Goldstein, Alex, Kapelner, Adam, Bleich, Justin, and Pitkin, Emil. "Peeking inside the black box: Visualizing statistical learning with
plots of individual conditional expectation." _Journal of Computational
and Graphical Statistics_, no. 24, 2015. Groeneveld, R.A. and Meeden, G. (1984), “Measuring Skewness and
Kurtosis.” The Statistician, 33, 391-399. Hall, Patrick, Wen Phan, and SriSatish Ambati. “Ideas for Interpreting
Machine Learning.” O’Reilly Ideas. O’Reilly Media, 2017. Hartigan, J.

Hastie, Trevor, Tibshirani, Robert, and Friedman, Jerome. _The Elements
of Statistical Learning_. Springer, 2008. Lei, Jing, Max G'Sell, Alessandro Rinaldo, Ryan J. Tibshirani, and Larry
Wasserman. "Distribution-Free Predictive Inference for Regression." _Journal of the American Statistical Association_ (just-accepted), 2017. Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. "Why Should I
Trust You? : Explaining the Predictions of Any Classifier." In
Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining, 2016. Tukey, J. W. (1977). _Exploratory Data Analysis_. Addison-Wesley,
Reading, MA. Wilkinson, L. (1999). “Dot plots.” The American Statistician, 53,
276–281. Wilkinson, L., Anand, A., and Grossman, R. (2005), “Graph-theoretic
Scagnostics,” in Proceedings of the IEEE Information Visualization 2005,
pp. 157–164. I.K.

INSTALL THE DRIVERLESS AI AWS MARKETPLACE AMI

A Driverless AI AMI is available in the AWS Marketplace beginning with
Driverless AI version 1.5.2. This section describes how to install and
run Driverless AI through the AWS Marketplace.

Log in to the AWS Marketplace. 2. Search for Driverless AI. 3. Select the version of Driverless AI that you want to install. 4. Scroll down to review/edit your region and the selected
    infrastructure and pricing. 5. Return to the top and select CONTINUE TO SUBSCRIBE. 6.  Review the subscription, then click CONTINUE TO CONFIGURE. 7. If desired, change the Fullfillment Option, Software Version, and
    Region. Note that this page also includes the AMI ID for the
    selected software version. Click CONTINUE TO LAUNCH when you are
    done. 8.  Review the configuration and choose a method for launching
    Driverless AI. Click the USAGE INSTRUCTIONS button in AWS to review
    your Driverless AI username and password. Scroll down to the bottom
    of the page and click LAUNCH when you are done. You will receive a "Success" message when the image launches
successfully.

1. Navigate to the EC2 Console. 2. Select your instance. 3. Open another browser and launch Driverless AI by navigating to
    https://\ <public IP of the instance>:12345. 4. Sign in to Driverless AI with the username H2OAI and use the AWS
    InstanceID as the password. You will be prompted to enter your
    Driverless AI license key when you log in for the first time. Stopping the EC2 Instance
    instance. Upgrading the Driverless AI Marketplace Image
Note that the first offering of the Driverless AI Marketplace image was
1.5.2. As such, it is only possible to upgrade to versions greater than
that.

dai_NEWVERSION.debbelow with the new Driverless AI version (for example,dai_1.5.4_amd64.deb``).
Note that this upgrade process inherits the service user and group from
upgrade.

    # Stop Driverless AI.
    sudo systemctl stop dai

    # Make a backup of /opt/h2oai/dai/tmp directory at this time.

    # Upgrade Driverless AI.
    sudo dpkg -i dai_NEWVERSION.deb

DATA RECIPE URL SETUP

Driverless AI lets you explore data recipe URL data sources from within
configuring Driverless AI to work with data recipe URLs. When enabled
(default), you will be able to modify datasets that have been added to
Driverless AI. (Refer to modify_by_recipe for more information.) NOTES:
    Docker 19.03) command when starting the Driverless AI Docker image. Use docker version to check which version of Docker you are using. Enable Data Recipe URL
This example enables the data recipe URL data connector.

MLI CUSTOM RECIPES

The techniques and methodologies used by Driverless AI for model
interpretation can be extended with recipes (Python code snippets). You
can use your own recipes in combination with or in place of DAI's
built-in recipes. This lets you extend the capabilities of MLI
explainers and out of the box interpretation techniques. The following
steps describe how to upload and enable custom recipes in the Machine
Learning Interpretability (MLI) view. Note
For more information on MLI custom recipes including best practices,
tutorials, explainer templates, and explainer examples, see the official
Recipes for Machine Learning Interpretability in Driverless AI repository <https://github.com/h2oai/driverlessai-recipes/tree/>. To upload a custom recipe:
  1. Navigate to the MLI page and click the NEW INTERPRETATION button. Select UPLOAD MLI RECIPE from the drop-down menu.

INTRODUCTION TO DRIVERLESS AI

introduction_to_dai key-features supported-algorithms workflow

VIEWING EXPLANATIONS

NOTE: Not all explanatory functionality is available for multinomial
classification scenarios. Driverless AI provides explanations for completed models. You can view
these by clicking the EXPLANATIONS button on the MODEL INTERPRETATION >
SURROGATE MODELS DASHBOARD page for an interpreted model. The UI lets you view global, cluster-specific, and local reason codes. You can also export the explanations to CSV. -   GLOBAL REASON CODES: To view global reason codes, click CLUSTER and
    select GLOBAL from the list of options. With Global selected, click
    the EXPLANATIONS button located in the upper-right corner. -   CLUSTER REASON CODES: To view reason codes for a specific cluster,
    click CLUSTER and select a specific cluster from the list of
    options. With a cluster selected, click the EXPLANATIONS button.

INSTALL ON AZURE

This section describes how to install the Driverless AI image from
Azure. NOTE: Prior versions of the Driverless AI installation and upgrade on
Azure were done via Docker. This is no longer the case as of version
1.5.2. WATCH THE INSTALLATION VIDEO here. Note that some of the images in this
same.

You can override the user by providing
    user:
        section for details). -   /etc/dai/User.conf: systemd config file specifying the service
        (optional) environment variable overrides.

-   /opt/h2oai/dai/tmp: Experiments and imported data are stored
    license key file to that location. For TAR SH installs, the
    placed in /etc/systemd/system.

The package installs the
instructions. Installing the Azure Instance
1. Log in to your Azure portal at https://portal.azure.com, and click
    the CREATE A RESOURCE button. 2. Search for and select H2O DRIVERLESSAI in the Marketplace. 3. Click CREATE. This launches the H2O DriverlessAI Virtual Machine
    creation process. 4. On the BASICS tab:
5. On the SIZE tab, select your virtual machine size. Specify the HDD
    disk type and select a configuration. We recommend using an N-Series
    type, which comes with a GPU.

We recommend a minimum of 30 GB of disk
    space. Click OK when you are done. 6. On the SETTINGS tab, select or create the Virtual Network and Subnet
    where the VM is going to be located and then click OK.
7. The SUMMARY tab performs a validation on the specified settings and
    will report back any errors. When the validation passes
    successfully, click CREATE to create the VM. 8. After the VM is created, it will be available under the list of
    Virtual Machines. Select this Driverless AI VM to view the IP
    address of your newly created machine. 9. Connect to Driverless AI with your browser using the IP address
    retrieved in the previous step. Stopping the Azure Instance
The Azure instance will continue to run even when you close the Azure
portal. To stop the instance:
1. Click the VIRTUAL MACHINES left menu item. 2. Select the checkbox beside your DriverlessAI virtual machine. 3. On the right side of the row, click the ... button, then select
    STOP.

[image]

Upgrading the Driverless AI Image
WARNINGS:

DRIVERLESS AI INSTALLATION AND UPGRADE

The following sections describe how to install and upgrade Driverless
AI.

NOTE: Driverless AI is available as part of the H2O AI Cloud (HAIC)
platform or as a standalone offering. For information on HAIC, see the
official documentation.

supported-environments installing-before-you-begin docker native cloud

GPUS IN DRIVERLESS AI

architectures. Ampere-based NVIDIA GPUs are also supported on x86
machines (requires NVIDIA CUDA Driver 11.2 or later). Driverless AI ships with NVIDIA CUDA 11.2.2 and cuDNN. Image <image-processing-in-dai> and NLP <nlp-in-dai> use cases in
Driverless AI benefit significantly from GPU usage. Model building algorithms, namely, XGBOOST (GBM/DART/RF/GLM), LIGHTGBM
(GBM/DART/RF), PYTORCH (BERT models) and TENSORFLOW (CNN/BiGRU/ImageNet)
models utilize GPU.

Shapley calculation on GPUs is coming soon. MOJO
scoring for productionizing models on GPUs can be enabled for some uses
cases. See tensorflow_nlp_have_gpus_in_production in
config.toml <sample-configtoml>. Driverless AI Tensorflow, BERT and
Image models support C++ MOJO <cpp_scoring_pipeline> scoring for
production. Feature engineering <feature_engineering> transformers such as
CLUSTERDIST cuML Transformer, TRUNCSVDNUM cuML Transformer, DBSCAN cuML
Transformer run on GPUs. With Driverless AI Dask multinode <dask-multinode-training> setup, GPUs
can be used for extensive MODEL HYPERPARAMENTER SEARCH. For details see -
DRIVERLESS AI & NVIDIA CUDNN
NVIDIA CUDNN is a library for deep neural nets built using CUDA and
optimized for GPUs. For NLP <nlp-in-dai> data modeling and feature
engineering , Driverless AI uses CUDNN PYTORCH (BERT MODELS) and
TENSORFLOW NLP recipe based on CNN and BiGRU (RNN) deep learning models.

DRIVERLESS AI & NVIDIA RAPIDS
NVIDIA RAPIDS provides PyData APIs that are GPU-accelerated.Driverless
AI integrates RAPIDS CUML (scikit-learn)
transformers <numeric_transformers> namely ClusterDist cuML Transformer,
TruncSVDNum cuML Transformer, DBSCAN cuML Transformer for feature
engineering and RAPIDS CUDF extension to
XGBoost GBM / DART <enable_xgboost_rapids> for building machine learning
models on GPUs. DRIVERLESS AI & DASK + RAPIDS
DASK is a Python library that enables out of core parallelism and
distribution of some popular Python libraries. Combining DASK with
RAPIDS supports a multinode multigpu framework. Driverless supports
DASK multinode cluster <dask-multinode-training> and offers DASK CUDF
multiGPU XGBoost GBM / DART <enable_xgboost_gbm_dask> for modeling and
Dask multinode / multiGPU hyperparameter search <enable_hyperopt_dask>
for model parameter selection.

INSTALL THE GOOGLE CLOUD PLATFORM OFFERING

This section describes how to install and start Driverless AI in a
Google Compute environment using the GCP Marketplace. This assumes that
you already have a Google Cloud Platform account. If you don't have an
account, go to https://console.cloud.google.com/getting-started to
create one. Before You Begin
account, check your Google Compute Engine (GCE) resource quota limits. By default, GCP allocates a maximum of 8 CPUs and no GPUs. Our default
recommendation for launching Driverless AI is 32 CPUs, 120 GB RAM, and 2
P100 NVIDIA GPUs. You can change these settings to match your quota
limit, or you can request more resources from GCP. Refer to
https://cloud.google.com/compute/quotas for more information, including
information on how to check your quota and request additional quota. Installation Procedure
1.

2. In the left navigation panel, select MARKETPLACE. 3. On the Marketplace page, search for DRIVERLESS and select the H2O.ai
    Driverless AI offering. The following page will display. 4. Click LAUNCH ON COMPUTE ENGINE. (If necessary, refer to Google
    Compute Instance Types for information about machine and GPU types.) 5. A summary page displays when the compute engine is successfully
    deployed. This page includes the instance ID and the username
    (always h2oai) and password that will be required when starting
    Driverless AI. Click on the Instance link to retrieve the external
    IP address for starting Driverless AI. 6. In your browser, go to https://%5BExternal_IP%5D:12345 to start
    Driverless AI. 7. Agree to the Terms and Conditions. 8. Log in to Driverless AI using your user name and password. 9. Optionally enable GCS and Big Query access.

LINUX TAR SH

The Driverless AI software is available for use in pure user-mode
environments as a self-extracting TAR SH archive. This form of
installation does not require a privileged user to install or to run. This artifact has the same compatibility matrix as the RPM and DEB
packages (combined), it just comes packaged slightly differently. See
those sections for a full list of supported environments. The installation steps assume that you have a valid license key for
Driverless AI. For information on how to obtain a license key for
Driverless AI, visit https://www.h2o.ai/products/h2o-driverless-ai/. Once obtained, you will be prompted to paste the license key into the
Driverless AI UI when you first log in. Note
To ensure that AutoDoc <autodoc> pipeline visualizations are generated
correctly on native installations, installing fontconfig is recommended.

-   Driverless AI TAR SH, available from https://www.h2o.ai/download/
on GPUs) are included in the Driverless AI package. Installing OpenCL
Note
config.toml setting.

# Install Driverless AI. chmod 755 |VERSION-tar-lin|
    ./|VERSION-tar-lin|
You may now cd to the unpacked directory and optionally make changes to
CONFIG.TOML. Starting Driverless AI
    # Start Driverless AI. ./run-dai.sh

Install OpenCL
OpenCL is required in order to run LightGBM on GPUs. Run the following
for Centos7/RH7 based systems using yum and x86.

./kill-dai.sh

To uninstall Driverless AI, just remove the directory created by the
unpacking process. By default, all files for Driverless AI are contained
within this directory. Upgrading Driverless AI
WARNINGS:
upgrade.

DRIVERLESS AI OPENID CONNECT AUTHENTICATION

This page describes how to set up OPENID CONNECT (OIDC) authentication
in DRIVERLESS AI (DAI). -   oidc_setup
-   oidc_understanding

Setting up OIDC authentication
To set up OIDC authentication locally (or in production), the following
config.toml options must be specified:
1.  authentication_method = "oidc" - Specifies OIDC as the
    authentication method
2.  auth_oidc_issuer_url = "https://login.microsoftonline.com/<client_id>/v2.0"
    - Specifies the URL of the Identity Provider (IDP), which is also
    used for automatic provider discovery
3.  auth_oidc_identity_source = "id_token" - Specifies whether user
    identity is retrieved from _ID Token_ or the _UserInfo_.

However, there are additional configuration options that can be
specified, which are needed, for example, when using MLOps / Remote
Storage or if specific OIDC scopes are needed.

The following sections describe how OIDC authentication is implemented
in DAI. Note
DAI only supports the Authorization Code Flow. As stated on the OpenID
website, the Authorization Code Flow returns an Authorization Code to
the Client, which can then exchange it for an ID Token and an Access
Token directly. Note
DAI mainly supports the client_secret_basic authentication method. Identity sources
The DAI OIDC authentication mechanism allows two different methods of
retrieving a user identity from IDP. Note
For both of the following methods, the user must specify the
auth_oidc_username_claim config.toml option, which controls which claim
is used as a username in DAI. -   userinfo: Makes a UserInfo endpoint request, which in response
    returns a set of claims that should contain the preferred username,
    which will be used as the DAI username.

Identity Validation
Driverless AI allows two different methods of evaluating whether user
(identity) has required privileges to access the DAI application. The
validation step is performed after DAI retrieves the user identity from
_Identity Sources_ mentioned above. -   If auth_openid_use_objectpath_match is enabled, then the user _must_
    specify auth_openid_use_objectpath_expression, which evaluates
    ObjectPath against identity (_UserInfo_ response or _ID Token_)
-   If auth_openid_use_objectpath_match is disabled, then the user may
    specify auth_openid_userinfo_auth_key and
    auth_openid_userinfo_auth_value to compare value with given key in
    identity against the configured value. Logging in using OIDC
The following steps describe the procedure of logging in using OIDC:
1. The OIDC Client is initialized at server startup and performs
    _Provider Discovery_, which discovers all the _Identity Provider
    (IDP)_ endpoints. 2.

auth_oidc_post_logout_url`` needs to be specified in the config.toml
file, which by design should point to the absolute DAI login URL.

WIDE DATASETS IN DRIVERLESS AI

A wide dataset with many features comes with its own challenges for
feature engineering and model building. In Driverless AI, datasets where number of columns > number of rows are
considered as wide. When running experiments on such datasets,
Driverless AI automatically enables wide rules <enable_wide_rules> that
extend the limits on the maximum number of allowed features (that can be
selected for feature evolution and selection) to a large number,
disables certain checks like data leakage and shift detection,
monotonicity constraints, AutoDoc and pipeline visualization creation. It also enables XGBoost random forest model for modeling, which helps to
avoid overfitting on wide datasets with few rows. See
enable_wide_rules <enable_wide_rules>. A big-wide dataset can result in large models that can run out of memory
on GPUs.

A final model is then built on the important features
that fit on the GPU. See
allow_reduce_features_when_failure <allow_reduce_features_when_failure>
for details. Here is an example of config.toml settings for a QUICK MODEL RUN ON A
WIDE DATASET. This _disables genetic algorithm/tuning/evolution_ to get a _quick final
model_. It also uses (XGBoost) random forest that is best to avoid
overfit on wide data with few rows. The following config settings can be
copy/pasted in the expert settings GUI TOML to run this model.

DRIVERLESS AI EXPERIMENT SETUP WIZARD

The Driverless AI Experiment Setup Wizard makes it simple for you to set
up a Driverless AI experiment and ensure that the experiment's settings
are optimally configured for your specific use case. The Experiment
Setup Wizard helps you learn about your data and lets you provide
information about your use case that is used to determine the
experiment's settings. This Wizard covers topics such as data leakage,
NLP handling, validation method, model reproducibility, and model
deployment. NOTES:
-   This feature is currently in an experimental state. -   A Dataset Join Wizard that makes it simple for you to join two
    datasets together is also available in Driverless AI. For more
    information, see join_dataset_wizard. The following sections describe how to access and use the Driverless AI
Wizard.

-   On the EXPERIMENTS page, click the NEW EXPERIMENT button and select
    WIZARD SETUP. If this method is used, then the Driverless AI Wizard
    prompts you to select a dataset to use for the experiment. []

Driverless AI Wizard sample walkthrough
The following example walks through the Driverless AI Wizard. Note that
this walkthrough does not contain every possible step that the wizard
offers. 1. Select the option that best describes your role and specify how many
    years of experience you have with machine learning and data science. In this example, the options DATA SCIENTIST and <1 YEAR are
    selected. Click CONTINUE to proceed. 2. SELECT A DATASET. Select a tabular dataset with training data. Each
    row in the dataset must contain predictor variables (features) that
    can be used to predict the target column. In this example, the Rain
    in Australia dataset is selected. 3. SELECT A PROBLEM TYPE AND TARGET COLUMN.

Note that you can select a
    target column for only one of the available problem types. The goal
    in this example is to use the Rain in Australia dataset to predict
    next-day rain by training classification models, so RainTomorrow is
    specified as the target column in the BINARY CLASSIFICATION section. Click CONTINUE to proceed. 4. TARGET COLUMN ANALYSIS. The Driverless AI Wizard provides
    information about the selected target column and prompts you to
    confirm that the target column looks as expected. Click YES to
    proceed, or click NO to return to the previous page and select a
    different column. 5. EXCLUDE COLUMNS. The Driverless AI Wizard prompts you to check for
    columns to drop from the experiment. Dropped columns are not used as
    predictors for the target column. If you already know which
    column(s) you want to drop, then you can click the YES, I WANT TO
    HAVE A LOOK button to select the column(s) you want to drop.

6. MODEL DEPLOYMENT. The Driverless AI Wizard prompts you to specify
    how you plan to use the model. In this example, the I'M NOT READY
    FOR PRODUCTION option is selected. 7. IMPORTANCE OF TIME ORDER. If your dataset contains at least one date
    or datetime column that doesn't contain missing values, the
    Driverless AI Wizard prompts you to specify how important time order
    is to the experiment. In this example, the TIME ORDER DOESN'T MATTER
    option is selected. 8. PROVIDE A TEST SET. Specify a test set to use for the experiment. You can select an existing test set, create a test set from the
    training data, or skip this step entirely. To refresh the list of
    available datasets, click the REFRESH DATASET LIST button. In this
    example, the CREATE TEST SET FROM TRAINING DATA option is selected. 9. SPLIT THE TRAINING DATA. Use the slider to specify what fraction of
    the training dataset you want to use for testing.

In this example, 15 percent of the training
    dataset is used for testing. Click SPLIT MY TRAINING DATA to
    proceed. 10. CONFIRM THE TRAIN / TEST SPLIT. The Driverless AI Wizard lists the
    following information for both the training and testing data based
    on the percentage specified in the preceding step:
    -   The size of each dataset. -   The number of rows and columns in each dataset. -   Whether either dataset has any temporal order. 11. SELECT A MODEL TYPE. Specify a model type based on settings for
    Accuracy, Time, and Interpretability, as well as training time and
    deployment size. You can also optionally specify whether you have
    strict runtime limits or if you want to limit the complexity of the
    model. In this example, the KEEP IT SIMPLE option is selected. Click
    CONTINUE to proceed. 12. SELECT A SCORER. Specify a scorer to optimize. In this example, AREA
    UNDER ROC CURVE (AUC) is selected. Click CONTINUE to proceed. 13.

TIPS 'N TRICKS

This section includes Arno’s tips for running Driverless AI. Pipeline Tips
Given training data and a target column to predict, H2O Driverless AI
produces an end-to-end pipeline tuned for high predictive performance
(and/or high interpretability) for general classification and regression
tasks. The pipeline has only one purpose: to take a test set, row by
row, and turn its feature values into predictions. A typical pipeline creates dozens or even hundreds of derived features
from the user-given dataset. Those transformations are often based on
precomputed lookup tables and parameterized mathematical operations that
were selected and optimized during training. It then feeds all these
derived features to one or several machine learning algorithms such as
linear models, deep learning models, or gradient boosting models (and
several more derived models).

The pipeline is a directed acyclic
graph. It is important to note that the training dataset is processed as a
whole for better results (e.g., aggregate statistics). For scoring,
however, every row of the test dataset must be processed independently
to mimic the actual production scenario. To facilitate deployment to various production environments, there are
multiple ways to obtain predictions from a completed Driverless AI
experiment, either from the GUI, from the R or Python client API, or
from a standalone pipeline. GUI
-   SCORE ON ANOTHER DATASET - Convenient, parallelized, ideal for
    imported data
-   DOWNLOAD PREDICTIONS - Available if a test set was provided during
    training
-   DEPLOY - Creates an Amazon Lambda endpoint (more endpoints coming
    soon)
-   DIAGNOSTICS - Useful if the test set includes a target column
CLIENT APIS
-   PYTHON CLIENT - Use the make_prediction_sync() method.

(Pass pred_contribs=True.) -   R CLIENT - Use the predict() method. An optional argument can be
    used to get per-row and per-feature 'Shapley' prediction
    contributions. (Pass pred_contribs=True.) STANDALONE PIPELINES
-   PYTHON - Supports all models and transformers, and supports
    'Shapley' prediction contributions and MLI reason codes
-   JAVA - Most portable, low latency, supports all models and
    transformers that are enabled by default (except TensorFlow NLP
    transformers), can be used in Spark/H2O-3/SparklingWater for scale
-   C++ - Highly portable, low latency, standalone runtime with a
    convenient Python and R wrapper

Time Series Tips
H2O Driverless AI handles time-series forecasting problems out of the
box. All you need to do when starting a time-series experiment is to provide
a regular columnar dataset containing your features.

If you have a test set for which you want predictions for
every record, make sure to provide future time stamps and features as
well. In most cases, that's it. You can launch the experiment and let
Driverless AI do the rest. It will even auto-detect multiple time series
in the same dataset for different groups such as weekly sales for stores
and departments (by finding the columns that identify stores and
departments to group by). Driverless AI will also auto-detect the time
period including potential gaps during weekends, as well as the forecast
horizon, a possible time gap between training and testing time periods
(to optimize for deployment delay) and even keeps track of holiday
calendars.

-   If you find that the automatic lag-based time-series recipe isn't
    performing well for your dataset, we recommend that you try to
    disable the creation of lag-based features by disabling "Time-series
    lag-based recipe" in the expert settings. This will lead to regular
    feature engineering but with time-based causal validation splits. Especially for small datasets and short forecast periods, this can
    lead to better results. -   If the target column is present in the test set and has partially
    filled information (non-missing values), then Driverless AI will
    automatically augment the model with those future target values to
    make better predictions. This can be used to extend the usable
    lifetime of the model into the future without the need for
    retraining by providing past known outcomes. Contact us if you're
    interested in learning more about test-time augmentation.

-   For datasets that are non-stationary in time, create a test set from
    the last temporal portion of data, and create time-based features. This allows the model to be optimized for your production scenario. -   We are working on further improving many aspects of our time-series
    recipe. For example, we will add support to automatically generate
    lags for features that are only available in the training set, but
    not in the test set, such as environmental or economic factors. We'll also improve the performance of back-testing using rolling
    windows. Scorer Tips
A core capability of H2O Driverless AI is the creation of automatic
machine learning modeling pipelines for supervised problems. In addition
to the data and the target column to be predicted, the user can pick a
scorer. A scorer is a function that takes actual and predicted values
for a dataset and returns a number.

There are more detailed ways to
estimate the performance of a machine learning model such as residual
plots (available on the Diagnostics page in Driverless AI), but we will
focus on scorers here. For a given scorer, Driverless AI optimizes the pipeline to end up with
the best possible score for this scorer. The default scorer for
regression problems is RMSE (root mean squared error), where 0 is the
best possible value. For example, for a dataset containing 4 rows, if
actual target values are [1, 1, 10, 0], but predictions are [2, 3, 4,
-1], then the RMSE is sqrt((1+4+36+1)/4) and the largest misprediction
dominates the overall score (quadratically). Driverless AI will focus on
improving the predictions for the third data point, which can be very
difficult when hard-to-predict outliers are present in the data. If
outliers are not that important to get right, a metric like the MAE
(mean absolute error) can lead to better results.

Another scorer that is robust to outliers is RMSLE
(root mean square logarithmic error), which is like RMSE but after
taking the logarithm of actual and predicted values - however, it is
restricted to positive values. For price predictions, scorers such as
MAPE (mean absolute percentage error) or MER (median absolute percentage
error) are useful, but have problems with zero or small positive values. SMAPE (symmetric mean absolute percentage error) is designed to improve
upon that. For classification problems, the default scorer is either the AUC (area
under the receiver operating characteristic curve) or LOGLOSS
(logarithmic loss) for imbalanced problems. LOGLOSS focuses on getting
the probabilities right (strongly penalizes wrong probabilities), while
AUC is designed for ranking problems. Gini is similar to the AUC, but
measures the quality of ranking (inequality) for regression problems.

We highly suggest experimenting with different scorers and to study
their impact on the resulting models. Using the Diagnostics page in
Driverless AI, all applicable scores can be computed for any given
model, no matter which scorer was used during training. Knob Settings Tips
H2O Driverless AI lets you customize every experiment in great detail
via the expert settings. The most important controls however are the
three knobs for accuracy, time and interpretability. A higher accuracy
setting results in a better estimate of the model generalization
performance, usually through using more data, more holdout sets, more
parameter tuning rounds and other advanced techniques. Higher time
settings means the experiment is given more time to converge to an
optimal solution. Higher interpretability settings reduces the model's
complexity through less feature engineering and using simpler models.

Generally, it
is sufficient to use settings of 7/5/5 or similar, and we recommend to
start with the default settings. We highly recommend studying the
experiment preview on the left-hand side of the GUI before each
experiment - it can help you fine-tune the settings and save time
overall. Note that you can always finish an experiment early, either by clicking
'Finish' to get the deployable final pipeline out, or by clicking
'Abort' to instantly terminate the experiment. In either case, the
experiment can be continued seamlessly at a later time with 'Restart
from last Checkpoint' or 'Retrain Final Pipeline', and you can always
turn the knobs (or modify the expert settings) to adapt to your
requirements. Tips for Running an Experiment
H2O Driverless AI is an automatic machine learning platform designed to
create highly accurate modeling pipelines from tabular training data.

During an experiment, Driverless AI
automatically tunes these parameters by scoring candidate pipelines on
held out ("validation") data. This important validation data is either
provided by the user (for experts) or automatically created (random,
time-based or fold-based) by Driverless AI. Once a final pipeline has
been created, it should be scored on yet another held out dataset ("test
data") to estimate its generalization performance. Understanding the
origin of the training, validation and test datasets ("the validation
scheme") is critical for success with machine learning, and we welcome
your feedback and suggestions to help us create the right validation
schemes for your use cases. Expert Settings Tips
H2O Driverless AI offers a range of 'Expert Settings' that let you
customize each experiment. For example, you can limit the amount of
feature engineering by reducing the value for 'Feature engineering
effort' or 'Max.

You can also select the model types to be used for training
on the engineered features (such as XGBoost, LightGBM, GLM, TensorFlow,
FTRL, or RuleFit). For time-series problems where the selected
time_column leads to an error message (this can currently happen if the
the time structure is not regular enough - we are working on an improved
version), you can disable the 'Time-series lag-based recipe' and
Driverless AI will create train/validation splits based on the time
order instead, which can increase the model's performance if the time
column is important. Checkpointing Tips
Driverless AI provides the option to checkpoint experiments to speed up
feature engineering and model tuning when running multiple experiments
on the same dataset. By default, H2O Driverless AI automatically scans
all prior experiments (including aborted ones) for an optimal checkpoint
to restart from.

TIME SERIES BEST PRACTICES

This document describes best practices for running time series
experiments in Driverless AI. Preparing Your Data
The goal for a time series use case is to use historical data to
forecast. The manner in which the data for forecasting is formatted
depends on what we want to do with this forecast. To format your data
for forecasting, aggregate the data for each group you are interested in
for a specific period of time. The following are three use cases in which the volume of stocks sold in
the S&P 500 is predicted. Each use case provides a unique scenario that
determines how the data is formatted. Our raw data looks like this:
[]
-   USE CASE 1: Forecast the total volume for a stock tomorrow. -   USE CASE 2: Forecast the total volume for a stock next month. -   USE CASE 3: Forecast the total volume of all S&P 500 stocks next
    year. Experiment Setup
Once your data is formatted to match your use case, you can begin
setting up your experiment.

[]
-   TIME GROUP COLUMNS: The columns that separate the data into
    individual time series. In this example, there is one time series
    per stock (column: Name ), so Name is selected as the time group
    column. -   UNAVAILABLE COLUMNS AT PREDICTION TIME: The columns that are not
    known at time of prediction. In the S&P 500 data example, the
    independent variables are open, high, low, and close. Any variables
    that are not known in advance must be marked as columns that are
    unavailable at prediction time. Driverless AI only uses historical
    values for the independent variables that are marked. -   FORECAST HORIZON: How far in advance you want to forecast. -   GAP: Specify whether there is any gap between the training data and
    when you want to start forecasting.

In Driverless AI, validation data is automatically created
by default, and this data is used to evaluate the performance of each
model. The test data is an optional dataset that is provided by the user. It is
not used by Driverless AI until after the final model has already been
chosen to prevent any accidental overfitting on the test data. VALIDATION DATA
Validation data is automatically generated by Driverless AI using a
rolling window approach. The number of time units contained in the
validation data matches the forecast horizon and gap configurations. If
you want to forecast the next day, the validation data must consist of
one day's worth of data. If you want to forecast the next five days, the
validation data must consist of five days' worth of data. In the first
use case, Driverless AI internally creates splits where the validation
data always consists of one day of data.

For these cases, it is important to make
sure that the validation data is large enough to prevent overfitting. There are generally two ways to do this: increase the number of
validation splits done by Driverless AI, or increase the number of Time
Group Columns in the dataset. You can increase the number of validation
splits performed by Driverless AI by going to the Expert Settings under
the Time Series tab:
[]
By default, Driverless AI automatically determines the number of
validation splits based on the Accuracy setting (higher accuracy leads
to more validation splits). You can override this to a larger number if
you know that the number of rows for each validation split will be small
(that is, a small number of Time Group Columns and/or a small Forecast
Horizon). If you override this, you can see the change reflected in the experiment
preview. In the following experiment, the number of validation splits
has been increased to 20 in the expert settings panel.

[]
Another way to prevent small validation data is to consider including
more Time Group Columns. Instead of forecasting the volume for each
stock separately, you can combine the data for all stocks and run a
single experiment. TEST DATA
The test data is an optional dataset provided by the user. Driverless AI
automatically calculates the performance of the final model on this
dataset but does not use it for model selection. The test dataset can be
larger than the Forecast Horizon. The first use case involves
forecasting the next day's stock volume. You can, however, provide
Driverless AI with one month of test data. In this scenario, Driverless
AI evaluates how the model does at forecasting the next day's stock
volume over the one month period. Scorers
The scorer determines how Driverless AI evaluates the success of each
model. []
The following is a list of popular scorers with information about which
use cases they excel in.

The first graph in the Model Interpretability module shows the error for
each date in the validation and test data:
[]
You can also see groups with very high error and very low error:
[]
You can search for a specific group to see the actual time series vs
predicted:
[]
By clicking on a specific forecasted point, you can see the Shapley
contributions for that point. The Shapley contributions tell you the
effect each predictor has on the prediction for that point. []
The Shapley contributions also show the strength and direction of each
predictor for the selected date. Scoring
Because Driverless AI is building a traditional machine learning model
(such as GLM, GBM, Random Forest), it requires a record to score on to
generate a prediction.

This means that if you provide Driverless AI with training data
up to 2018-02-07 and ask it to build a model to predict tomorrow's
volume, the MOJO can only be used to score for 2018-02-08. The MOJO is stateless. It takes a single record and provides a
prediction. Because it is stateless, it is only able to use information
(like past behavior) that is saved from the training data. If a
Driverless AI model shows that the previous day's stock volume is very
important, then once the MOJO is used to start scoring past 2018-02-08,
it no longer has information about the previous day's stock volume. PREDICTING WITHIN FORECAST HORIZON
If you want to predict within the Forecast Horizon, you can provide
Driverless AI, the Python Scoring pipeline, or the MOJO scoring pipeline
with the record that you want to predict for. Consider the following
example:
The training data ends on Friday 2018-01-05 and you want to forecast the
next business day's stock volume.

To predict the Stock volume for Stock: AAL on
2018-01-08, provide any scoring method with the following data. []
The output is the volume prediction. NOTE: Because open, high, low, and close are not known at the time of
prediction, these are filled in with NAs. PREDICTING OUTSIDE FORECAST HORIZON
If you now want to use the model to predict past 2018-01-08, then you
can only use Driverless AI or the Python scoring pipeline to score
because the MOJO is stateless and cannot be used outside of the Forecast
Horizon. In order to forecast outside of the Forecast Horizon, you need to
provide the model with any information that occurred after the training
data ended up until the date you want to forecast for. In the case where
you want to forecast for 2018-01-09, you must tell the model what
happened on 2018-01-08 (this date was not in the training data, so
Driverless AI does not know what ended up happening on that date).

[]
The model now returns two predictions: one for 2018-01-08 and one for
2018-01-09 (the prediction of interest). Other Approaches
Using the IID Recipe
Sometimes it can be helpful to try building an experiment without the
Time Series recipe even if you have a forecasting use case. The Time
Series recipe relies heavily on lagging the data, which means that it is
most helpful for cases where the past behavior is predictive. If you
have a use case where there is no strong temporal trend, then it may be
helpful to use Driverless AI without the Time Series recipe turned on. You can do this by simply not providing a Time Column when setting up
the experiment. NOTES:
-   If you decide to try the model without Time Series turned on, make
    sure to provide a test dataset that is out of time. -   By default, Driverless AI performs random cross validation that can
    lead to overfitting if you have a forecasting use case.

This
    means that you must lag your data in advance. LAGGING DATA IN ADVANCE
For this use case, you do not know the open, close, low, and high
predictors in advance. If you want to use Driverless AI without the Time
Series recipe turned on, you must manually lag these variables so that
they consist of the previous day's information. _Original Data_:
[]
_Lagged for Driverless AI IID_
[]
You can do this by using the Data Recipe option in Driverless AI. To do
this, click on the Details of the Dataset and select Modify by Recipe:
[]
You can upload a modification recipe or use Live Code.

LAUNCHING DRIVERLESS AI

Driverless AI is tested on Chrome and Firefox but is supported on all
major browsers. For the best user experience, we recommend using Chrome. 1. After Driverless AI is installed and started, open a browser and
    navigate to <server>:12345. 2. The first time you log in to Driverless AI, you will be prompted to
    read and accept the Evaluation Agreement. You must accept the terms
    before continuing. Review the agreement, then click I AGREE TO THESE
    TERMS to continue. 3. Log in by entering unique credentials. For example:
      USERNAME: h2oai PASSWORD: h2oai
4. As with accepting the Evaluation Agreement, the first time you log
    in, you will be prompted to enter your License Key. Click the ENTER
    LICENSE button, then paste the License Key into the LICENSE KEY
    entry field. Click SAVE to continue. This license key will be saved
    in the host machine's /LICENSE folder.

[]

Server startup files
The following is a list of files written by Driverless AI (DAI) on
server startup:
-   PID FILES: These files are used to instruct later DAI stop Bash
    scripts on which processes to stop. -   STANDARD OUTPUT (STDOUT) LOG FILES: These log files are the standard
    output for different servers (given as prefix). -   STANDARD ERROR (STDERR) LOG FILES: These log files are standard
    error for different servers (given as prefix). -   TMPDIR DIRECTORIES: These are temporary directories used by various
    packages or servers. -   uploads DIRECTORY: This directory is where files are uploaded by the
    web server. -   funnels DIRECTORY: This directory is where certain forked processes
    store stderr or stdout files. -   sys DIRECTORY: This directory is used by the system to perform
    various generic tasks. -   startup_job_user DIRECTORY: This directory is used by the system to
    perform various startup tasks.

Resources
[]
The RESOURCES drop-down menu lets you view system information, download
DAI clients, and view DAI-related tutorials and guides. -   System Info: View information relating to hardware utilization and
    worker activity. Note that the worker activity tab is only available
    if Driverless AI was started in multinode mode. For more information
    on multinode training in DAI, see multinode-training. -   Python Client: Download the Driverless AI Python client. For more
    information, see python_client. -   R Client: Download the Driverless AI R client. For more information,
    see r_client. -   MOJO Java Runtime: Download the MOJO Java Runtime. For more
    information, see Mojo_Pipeline. -   MOJO Py Runtime: Download the MOJO Python Runtime. For more
    information, see cpp_scoring_pipeline. -   MOJO R Runtime: Download the MOJO R Runtime. For more information,
    see cpp_scoring_pipeline. -   Documentation: View the DAI documentation.

MONOTONICITY CONSTRAINTS

Monotonicity can be enforced for the feature engineering pipeline, the
fitted model(s), or the entire modeling pipeline. Monotonicity constraints enforce a monotonic relationship between a
specified feature and the target prediction. For example, given a model
trained to predict housing prices, you may want to enforce that the
model predicts higher housing prices with increasing lot size and lower
housing prices with increasing neighborhood crime rate. When monotonicity constraints are enabled, Driverless AI automatically
determines if monotonicity is present and then enforces it through all
or part of the modeling pipelines. Depending on the level of correlation
between a feature and the target, Driverless AI assigns positive,
negative, or no monotonicity constraints. Specifically, monotonicity is
enforced if the absolute correlation is greater than a specific
threshold (default 0.1).

Driverless AI will
automatically adjust multiple settings so that monotonicity is enforced
on the feature engineering and model building level. For details see
Monotonic GBM <pipeline-building-recipe> in pipeline building recipe
under experiment expert settings. For more granular control, over thresholds, manual override of
monotonicity constraints etc, refer to
these settings <enable-constraints> under feature settings of the expert
panel of an experiment. To build monotonic fitted models, ensure that:
-   The Interpretability setting for the experiment must be greater than
    or equal to the
    monotonicity_constraints_interpretability_switch <enable-constraints>,
    that has a default value of 7). So Interpretability setting for the
    experiment and/or monotonicity_constraints_interpretability_switch
    can be toggled to achieve this.

These can be set to 'ON'
    from the Model settings of the expert panel. The ensemble level can
    be toggled by setting fixed_ensemble_level <fixed_ensemble_level>
    level. For no ensemble model, set it to 0. -   Drop features with low correlation to the target. See
    monotonicity constraints drop low correlation features <monotonicity-constraints-drop-low-correlation-features>. -   For regression case, make sure the
    target_transformer <target_transformer> is monotonic like 'identity'
    or 'identity_noclip'. This can be toggled under experiment settings
    of the expert panel. and for monotonic feature engineering:
-   Disable features engineered from multi-feature interaction i.e set
    max_feature_interaction_depth <max-feature-interaction-depth> to 1
    in feature settings under expert settings panel. -   Disable numerical to categorical feature transformations i.e set
    num_as_cat <num_as_cat> to False in the feature settings under
    expert settings panel.