Unveiling Snowflake’s Stealthy Upgrade : A Deep Dive into Enhanced Worker Node Specs

In my recent investigation into Snowflake’s worker nodes, I discovered several significant updates that appear to have been made without formal announcements.

• The local disk capacity has increased from 200GB/node to 400GB/node
• The CPU architecture has changed from x86_64 to aarch64.

A ninja polar bear hides among database servers.

Current Specifications of Worker Nodes

Current specifications of worker nodes.

The queries are now executed on the following machine specifications per node:

• Instance Type: C6g.2xlarge
• Platform: Elastic Kubernetes Service
• Operating System: Amazon Linux 2 (Linux Kernel 5.4.181)
• CPU Cores: 8 cores/node
• CPU Architecture: aarch64 (arm64)
• Disk Capacity: 400GB (392156184576 bytes)/node

In support of this, I found a statement from last May’s financial results where the CFO mentioned that they had already migrated to Graviton 2.

We’ve fully migrated all of our customers in AWS to Graviton 2. [1]

Specifications of Previous Worker Nodes

According to the report from Select Corporation, the c5d.2xlarge was being used as of November 2022.

it’s fairly well known that on AWS each node is a c5d.2xlarge EC2 instance, with 16GB of RAM and a 200GB SSD. [2]

To better understand these changes, I compared the single node specifications from November 2022 and January 2024

Comparing Single Node Specifications: November 2022 vs. January 2024

Information Gathering Procedure

Python UDF

Using Python’s platform module, I was able to output the information of the nodes.

import platform

system_info= {
    'system': platform.system(),
    'release': platform.release(),
    'version': platform.version(),
    'architecture': platform.architecture(),
    'machine': platform.machine(),
    'processor': platform.processor(),
    'python_version': platform.python_version(),
    'python_compiler': platform.python_compiler(),
    'python_build': platform.python_build(),
    'os_version': platform.uname(),
    'disk_usage': shutil.disk_usage("/"),
    'environment_variables': dict(os.environ),
}

Executed Result :

{
    "system": "Linux",
    "release": "5.4.181-99.354.amzn2.aarch64",
    "version": "#1 SMP Wed Mar 2 18:50:47 UTC 2022",
    "architecture": [
        "64bit",
        ""
    ],
    "machine": "aarch64",
    "processor": "",
    "python_version": "3.11.5",
    "python_compiler": "GCC 11.2.0",
    "python_build": [
        "main",
        "Sep 11 2023 13:15:12"
    ],
    "os_version": [
        "Linux",
        "PYTHON-UDF",
        "5.4.181-99.354.amzn2.aarch64",
        "#1 SMP Wed Mar 2 18:50:47 UTC 2022",
        "aarch64",
        ""
    ],
   "disk_usage":[
      392156184576,
      6392868864,
      385763315712
   ],
   "environment_variables":{
      "USER":"udf",
      "LOGNAME":"udf",
      "HOME":"/home/udf",
      "TZ":"America/Los_Angeles",
      "OMP_NUM_THREADS":"1",
      "OPENBLAS_NUM_THREADS":"1",
      "MKL_NUM_THREADS":"1",
      "BLIS_NUM_THREADS":"1",
      "VECLIB_MAXIMUM_THREADS":"1",
      "NUMBA_NUM_THREADS":"1",
      "NUMEXPR_NUM_THREADS":"1",
      "CONDA_PREFIX":"/usr/lib/python_udf/xxxxxxxxxxxxxxxxx",
      "PROJ_LIB":"/usr/lib/python_udf/xxxxxxxxxxxxxxxxx/share/proj",
      "GDAL_DATA":"/usr/lib/python_udf/xxxxxxxxxxxxxxxxx/share/gdal",
      "GDAL_DRIVER_PATH":"/usr/lib/python_udf/xxxxxxxxxxxxxxxxx/lib/gdalplugins",
      "GEOTIFF_CSV":"/usr/lib/python_udf/xxxxxxxxxxxxxxxxx/share/epsg_csv"
   },
}

The Python runtime is conda. The default timezone seems to be Los Angeles. The processor name appears to be hidden. Each worker node is limited to a single thread.

Investigating /proc

I wanted to know the processor name, so I investigated /proc.

In Linux, you can collect system information by looking under the /proc folder. The information about the CPU is stored in /proc/cpuinfo, but I didn’t have read permission.

However, I did manage to look at the following files, which I had read access to:

• /proc/self/exe
• /proc/self/stat
• /proc/self/maps

/proc/self/exe indicates the process name of the UDF, which was /bin/python_udf_server_3.11.

/proc/self/maps is a memory map file. It’s too long to detail here, but it’s interesting because it shows the loaded Python standard libraries, among other things.

/proc/self/stat shows the state of the process itself, looking like this:

4858 (python_udf_serv) S 4857 4858 4788 0 -1 4194560 5530 601 0 0 9 1 0 0 25 5 2 0 9957 57290752 6351 18446744073709551615 2097152 16947760 281473996413216 0 0 0 0 17305601 17640 0 0 0 17 7 0 0 0 0 0 17475264 17517840 318431232 281473996417949 281473996423137 281473996423137 281473996423137 0

Of interest was the CPU number. The “7” in “17 7 0” indicates the number of the CPU executing the process, starting from 0.

I ran the UDF multiple times and it was always between 0 to 7, leading me to deduce it’s an 8-core CPU.

Conclusion

Snowflake has improved their infrastructure continuously.

The doubling of local disk space at no extra cost is a welcome improvement !