Snowflake Native App and Snowflake Series — How to build an XGboost Forecasting App

If you love to build a highly secured applications in production, then our Native Application is the way forward. Please refer to the Snowflake doc or my previous blog for info on Native application.

With the advent of competition, hourly electricity prices are being determined by a variety of market mechanisms, rather than cost-based engineering calculations. As a result, electric utilities, generators, and traders face a new set of short-term forecasting problems. These problems are unlike those in other industries since electricity must be produced at the same time that it is consumed. As a result, prices are determined hourly, 24 hours a day, 7 days a week.

Historically, price forecasting has been performed with least-cost optimization models. These models compute marginal cost based on assumptions about system loads, power plant availability, and fuel prices. These models do not explain price variations related to market strategy and to buyer and seller behavior in a market system. Statistical models, which reflect actual market outcomes, are better suited to short-term forecasting in this dynamic environment.

With this context, let us build a time-series model forecasting which is widely used in this space. This app can perform Feature selection (from the data from Marketplace), Hyper-parameter tuning, Model training and building, Backtesting, Monitor performance, and Deploying the model for Energy price forecasting use-case. This demo was built using the support of @JenPrusa who built the DS components and Titiaan Palazzi who is the brain behind this.

Energy Price Forecasting Application

Components of Native Application

To begin with, we should understand what are the components of Native Application. There are 3 major components of the Native Application object. They are as follows.

Components of Native Application

Prerequisites:

1. Snowflake trial account. Signup here. Role with access to install NativeApps from a marketplace like ACCOUNTADMIN
2. Agree to Anaconda Python Packages using ORGADMIN role under the “Billing & Terms” section (if not enabled by default).
3. You must have conda/miniconda installed. If not, please install following the doc.
4. Any environment to run notebooks like Jupyterlab, Jupyter Notebook, etc.,

Step by Step Instructions to build the App:

1. Bringing in Data:

The data we use is available in the Marketplace for free. Thanks to Yes Energy and Samuel Lockshin for making the sample data available for building the demo.

1. Please navigate to the Marketplace tab under Data Products
2. Ensure you are in the “ACCOUNTADMIN” role
3. Type in “Yes Energy” in the search box
4. Select the provider “Yes Energy”

1. Select the “Yes Energy — Sample Data”
2. Click “Get”
3. Click “Get” again on the next screen
4. Click “Done” on the final screen.

That’s it, within a few clicks the whole 3 years’ worth of data is available for you to play around. Isn’t it amazing!!! Thanks for the data-sharing feature of Snowflake.

2. Replicating the Algorithm in your local:

All our code is in Python, hence we use Snowpark to run them within Snowflake.

The link to the gist is available here. Please download the gist to your local and follow the folder structure.

Local folder structure

NativeApp Folders:

Now, please create a folder named “energy_price_forecasting_nativeapp” and create subfolders “python”, “scripts” and “streamlit” as per the below screenshot. The final structure should be as below now. Copy the respective files from the gist here.

Native App folder structure

3. Building Native Application

To build the application, just run all the cells in the Setup_Energy_Price_Forecasting.ipynb and the app gets deployed in your Snowflake account.

Currently, Streamlit is the only supported UI in the Native Application framework, with SPCS in the future this will change. Hence our app also uses Streamit to build the front-end UI to our Energy Price Forecasting Native application.

Forecast Price and deploy via the Native App

Follow the below steps to forecast and deploy the model in Snowflake.

1. Feature Selection:

Please select the right view, target column, and all the features in the dropdowns and click “SELECT FEATURES” as shown in the picture below.

Feature Selection

2. Exploration:

Based on the features selected, the exploration part changes dynamically. With all the features selected, it should be as below.

Data Exploration and Correlation

3. Hyperparameter optimization:

Please click the “Run hyperparameter optimization” to run the proc sproc_optuna_optimized_model. The parameter results will vary for each person, so do not worry if your results are different.

4. Build model and run backtest:

Once the hyperparameter optimization is complete, please click “Run Backtest” which will build the forecasting model, upload the .sav file to the snowflake stage, and then run a backtest using the latest model file.

Backtest

4. View model performance:

Now, we can see some performance metrics of the model that we built in the next block.

Performance metrics

5. Deploy Model:

Once you are happy with the model, you can deploy the model by clicking “Deploy Model” which will run inferencing using the selected model and display the predicted results alongside actual data.

Note: Dynamic table is a new feature that will simplify deployment. This feature will aid in saving maintenance overhead. Another feature called Model registry is coming up as well which will avoid holding the model files in stages and gives you a registry to maintain model versions and its associated metadata.

Model deployment

You can repeat the same process for spike forecasting as well if you are interested.

Thank you!

Glad you all reached here! Yes, now you have built your first Native Application in Snowflake for free which is highly secured, highly scalable, and sharable. Secure, because the data never leaves Snowflake, bringing the processing as close as possible to the data improves the data security and the performance. Scalable, the storage and computing are independent of each other, hence our application is highly scalable. This is sharable because you can share this application with another Snowflake account in the same region (this needs Enterprise+ Snowflake edition), so you can build an app and share it with your vendors or customers in the same region securely. If you or your organization already have a product you can very well build your application and try to monetize the app by listing them in our marketplace too.

Summary:

Hope you enjoyed building the app along with me! We will build a few more applications together in this series. So keep building!!!

Make it work, make it secure, make it fast, make it grow!

Opinions expressed in this post are solely my own and do not represent the views or opinions of any of my employer.