AutoML Climate Change Use Case

How AutoML Can Help

As an example, we are using a dataset that focuses around simulation crashes that occur during uncertainty quantification (UQ) ensembles for climate models. This dataset contains records from simulations performed in LLNL's UQ Pipeline software system utilizing a Latin hypercube sampling technique. The dataset includes of three unique 180-member Latin hypercube ensembles, for a total of 540 simulations. Of these, 46 simulations were unsuccessful because of numerical problems resulting from certain combinations of parameters. Our goal is to create classification models that, given the values of input parameters, can forecast if a simulation will succeed or fail. We can speed up the process of developing and assessing predictive models by utilizing MLJAR AutoML, which will enable us to thoroughly examine the elements that lead to simulation failures. The automation features of MLJAR AutoML greatly increase this analysis's efficiency and accuracy, promoting better climate model reliability and more knowledgeable climate research decision-making.

Business Value

30%
Faster

Up to 30% quicker than using conventional techniques, MLJAR AutoML can process huge climate datasets. This acceleration improves the responsiveness of climate research and decision-making by enabling faster identification of trends and anomalies.

25%
More Accurate

Climate model forecasting accuracy can be increased by up to 25% using MLJAR AutoML, which automates the building and optimization of models. Forecasts become more accurate as a result, and policy and mitigation plans become more knowledgeable.

35%
Larger Data Capacity

Compared to traditional methods, MLJAR AutoML effectively handles and analyzes up to 35% more data. This capacity facilitates deeper comprehension of intricate climatic systems and more thorough climate simulations.

20%
Deeper Insights

Automation with MLJAR AutoML can increase the detection of important elements impacting climate change by up to 20% by highlighting patterns and linkages in climate data that were previously overlooked.

40%
Quicker Model Development

The time needed to create and implement predictive models can be reduced by up to 40% using MLJAR AutoML. The cycle for converting data insights into workable climate strategies is shortened by this reduction.

AutoML Report

MLJAR AutoML generates comprehensive reports that are packed with valuable information, providing a deep insight of model performance, data analysis, and assessment measures. Here are a few illustrations.

Leaderboard

To evaluate the effectiveness of trained models, AutoML has used logloss as its performance measure. As can be seen in the table and graph below, Ensemble was consequently chosen as the best model.

Performance

Spearman Correlation of Models

Based on their rank-order correlation, the plot shows the relationships between various models. The strength of the models' performance rankings is demonstrated by the Spearman correlation, which measures how well a monotonic function may describe the relationship between two variables. Stronger associations, when models perform similarly across various measures or datasets, are indicated by higher correlation values.

Feature Importance

The graphic illustrates how important different features are in relation to different models. Each cell in this heatmap shows the significance of a single characteristic for a certain model, and the color intensity corresponds to the degree of significance. Lighter colors suggest lesser importance, while darker or more vivid colors indicate higher importance. By comparing the contributions of various models, this visualization aids in identifying which elements regularly contribute significantly to prediction performance and which have less of an impact.

Install and import necessary packages

Install the packages with the command:

pip install mljar-supervised, scikit-learn

Import the packages into your code:

# import packages
from supervised import AutoML
from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

Load data

We will read data from an OpenML dataset.

# load dataset
data = fetch_openml(data_id=1467, as_frame=True)
X = data.data
y = data.target
# display data shape
print(f"Loaded X shape {X.shape}")
print(f"Loaded y shape {y.shape}")
# display first rows
X.head()

Loaded X shape (540, 20)
Loaded y shape (540,)

Split dataframe to train/test

To split a dataframe into train and test sets, we divide the data to create separate datasets for training and evaluating a model. This ensures we can assess the model's performance on unseen data.

# split data
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.90, shuffle=True, stratify=y, random_state=42)
# display data shapes
print(f"X_train shape {X_train.shape}")
print(f"X_test shape {X_test.shape}")
print(f"y_train shape {y_train.shape}")
print(f"y_test shape {y_test.shape}")

X_train shape (486, 20)
X_test shape (54, 20)
y_train shape (486,)
y_test shape (54,)

Fit AutoML

We need to train a model for our dataset. The fit() method will handle the model training and optimization automatically.

# create automl object
automl = AutoML(total_time_limit=300, mode="Explain")
# train automl
automl.fit(X_train, y_train)

Predict

Generate predictions using the trained AutoML model on test data.

# predict with AutoML
predictions = automl.predict(X_test)
# predicted values
print(predictions)

['2' '1' '1' '1' '1' '1' '1' '1' '1' '1' '2' '1' '1' '2' '1' '1' '1' '1'
 '1' '2' '2' '2' '1' '1' '1' '2' '2' '1' '1' '2' '1' '1' '1' '2' '2' '1'
 '2' '1' '1' '1' '1' '1' '2' '1' '1' '1' '1' '1' '1' '1' '1' '2' '1' '1'
 '2' '1' '1' '1' '2' '1' '2' '2' '2' '1' '1' '2' '1' '2' '2' '1' '1' '1'
 '1' '1' '2' '1' '1' '1' '1' '1' '1' '1' '1' '2' '1' '2' '1' '1' '1' '2'
 '1' '2' '1' '2' '1' '1' '1' '1' '1' '2' '1' '1' '2' '1' '1' '1']

Compute accuracy

We are computing the accuracy score and valid values (y_test) with our predictions.

# compute metric
metric_accuracy = accuracy_score(y_test, predictions)
print(f"Accuracy: {metric_accuracy}")

Accuracy: 0.9074074074074074

Conclusions

When applied to climate model simulations, MLJAR AutoML provides a number of benefits for enhancing model dependability and comprehending accidents in the simulation. MLJAR AutoML improves the precision of simulation results prediction and failure root cause analysis by automating the intricate process of data analysis. Due to this advanced automation, researchers can handle large amounts of simulation data quickly and effectively, identifying trends and other elements that they might have missed using more conventional techniques. The increasing assimilation of this technology has potential to propel our comprehension of climate processes and augment the accuracy of climate forecasts.

See you soon👋.