Installing DBT-Snowflake and creating a DBT project in local storage using a Virtual Environment

Dbt is a transformation workflow that enables users to complete more work faster and with better quality. Users can use dbt Core or dbt Cloud to access dbt. The quickest and most reliable way to deploy dbt is through dbt Cloud. In a single web-based UI, users can develop, test, schedule, and examine data models. Data teams may transform data using best practices for analytics engineering thanks to the open-source application dbt Core. On the command line, dbt Core can be installed and used.

This blog is about how to install dbt Core in local storage utilizing a virtual environment. Make sure Python is installed on the computer before begin installing dbt Core.

Note:- The command prompt executes the entire code below.

Step 1: Installing Virtual Environment

By using pip install, the user can set up a virtual environment.

python -m pip install virtualenv

or

python -m pip install venv

Create an environment using the installed virtual environment

python -m virtualenv env_name

or

python -m venv env_name

Step 2: Activating the Environment

The folder with the environment name will be generated in the chosen path once the environment has been created. Open Scripts in the environment folder, then use the “activate” command to turn on the environment.

........\env_name\Scripts> activate

Step 3: Installing dbt-snowflake in the virtual environment

The following command can be used to install dbt-core or dbt-snowflake once the virtual environment has been activated.

pip install dbt-snowflake

We may create our project in the required location once the dbt has been installed. For best practice, clone the git repository and create the project

git clone https://github.com/username/repositoryName.git
cd repositoryName

dbt init projectName

The adapter that DBT supports must be configured when building a DBT project. It includes

• biAlloyDB
• Azure Synapse
• BigQuery
• Databricks
• Dremio
• Postgres
• Redshift
• Snowflake
• Spark
• Starburst & Trino

Since we installed dbt-snowflake and we need snowflake to be configured in our project,

Which database would you like to use?
[1] snowflake
(Don't see the one you want? https://docs.getdbt.com/docs/available-adapters)
Enter a number: 1
account (https://<this_value>.snowflakecomputing.com): account (e.g.: ab01234.country-zone.azure)
user (dev username): username
[1] password
[2] keypair
[3] sso
Desired authentication type option (enter a number): 1
password (dev password): password
role (dev role): role (e.g.: DBT_DEV_ROLE)
warehouse (warehouse name): warehouse (e.g.: DBT_DEV_WH)
database (default database that dbt will build objects in): database (e.g.: DBT_DEV_DB)
schema (default schema that dbt will build objects in): schema (e.g.: DBT_DEV_SCH)
threads (1 or more) [1]: 4

If the user wants to use SSO (Okta or External Authenticator), the user can use the following configuration.

Which database would you like to use?
[1] snowflake

(Don't see the one you want? https://docs.getdbt.com/docs/available-adapters)

Enter a number: 1
account (https://<this_value>.snowflakecomputing.com): account 
(e.g.: ab01234.country-zone.azure)
user (dev username): username
[1] password
[2] keypair
[3] sso
Desired authentication type option (enter a number): 3
authenticator ('externalbrowser' or a valid Okta URL) [externalbrowser]:
externalbrowser or your okta url (e.g: https://abc.oktapreview.com)
role (dev role): role (e.g.: DBT_DEV_ROLE)
warehouse (warehouse name): warehouse (e.g.: DBT_DEV_WH)
database (default database that dbt will build objects in): database (e.g.: DBT_DEV_DB)
schema (default schema that dbt will build objects in): schema (e.g.: DBT_DEV_SCH)
threads (1 or more) [1]: 4

Important note: - It would not prompt for a password while creating a project. If using okta, it must be configured in the “profiles.yml” file as follows

project_name:
  outputs:
    dev:
      account: ab01234.country-zone.azure
      authenticator: your okta Url (https://abc.oktapreview.com)
      warehouse: DBT_DEV_WH
      database: DBT_DEV_DB
      schema: DBT_DEV_SCH
      role: DBT_DEV_ROLE
      user: okta username
      password: okta password
      type: snowflake
      threads: 4
  target: dev

The “profiles.yml” file, which is located in the “.dbt” folder, can always be changed if the setup is incomplete or done incorrectly. Make sure that the profile name in “profiles.yml” and the project name in “dbt project.yml” are identical.

profiles.yml

dbt_project.yml

Note:- project_name in “profiles.yml” should match the profile name in “dbt_project.yml”

The project will be created in the local storage as

created dbt project

Before running the code, make sure your environment is activated.

Once the project was created, the user can build a dbt project in a code editor preferably using VSCode or Atom, and run the project from the command line.

About Us

Bi3 has been recognized for being one of the fastest-growing companies in Australia. Our team has delivered substantial and complex projects for some of the largest organizations around the globe and we’re quickly building a brand that is well-known for superior delivery.

Website: https://bi3technologies.com/

Follow us on,
LinkedIn: https://www.linkedin.com/company/bi3technologies
Instagram: https://www.instagram.com/bi3technologies/
Twitter: https://twitter.com/Bi3Technologies