Announcing FiftyOne 0.24 with 3D Meshes and Custom Workspaces

I’m thrilled to announce the general availability of FiftyOne 0.24, an exciting update that brings significant upgrades to the representational power and extensibility of open source FiftyOne, including:

• 3D Meshes: FiftyOne now natively supports full 3D scenes, including meshes (GLTF, OBJ, PLY, STL, FBX), point clouds, and arbitrary 3D geometries composed of boxes, spheres, cylinders, and planes.
• Custom Workspaces: this all-new feature allows you to create custom panel layouts in the FiftyOne App and persist them to your dataset so that you can load your favorite workspaces later with a single click or function call. For example, you can configure an embeddings visualization workspace, a model evaluation workspace, or build a completely custom workspace composed of panels from your own plugins!

At Voxel51, our goal is to simplify and automate how you explore, visualize, and curate visual data. FiftyOne is the place where you can test and refine your models alongside the data they’re trained on, enabling you to build higher quality datasets, iterate more quickly, integrate flexibly into the way you work, and ultimately get your visual AI models into production.

With this release, we’re advancing our mission by expanding our commitment to multimodal AI and unlocking new ways to standardize key parts of your model development workflow.

You can give the latest release a test drive by installing FiftyOne locally:

pip install --upgrade fiftyone

Check out the release notes for a full rundown of additional enhancements and bugfixes in FiftyOne 0.24.

Okay, now let’s dive into some highlights from the release.

3D Meshes

Ever since we added support for 3D point clouds last year, one of the most popular feature requests we’ve heard has been adding full support for 3D, including meshes and other 3D geometries. Well, as of FiftyOne 0.24, 3D scenes are now fully supported!

3D datasets in FiftyOne are composed of samples whose filepaths have the .fo3d extension. Each FO3D file encapsulates a 3D scene, which is internally represented as an n-ary tree of 3D objects that have one of the following types:

Scenes may also be given additional attributes, such as camera, lights, and background.

3D scenes can be conveniently constructed using the Scene class, which provides methods to add, remove, and manipulate 3D objects in a scene. For example:

import fiftyone as fo

scene = fo.Scene()
scene.camera = fo.PerspectiveCamera(up="Z")

mesh = fo.GltfMesh("mesh", "mesh.glb")
mesh.rotation = fo.Euler(90, 0, 0, degrees=True)

sphere1 = fo.SphereGeometry("sphere1", radius=2.0)
sphere1.position = [-1, 0, 0]
sphere1.default_material.color = "red"

sphere2 = fo.SphereGeometry("sphere2", radius=2.0)
sphere2.position = [-1, 0, 0]
sphere2.default_material.color = "blue"

scene.add(mesh, sphere1, sphere2)

scene.write("/path/to/scene.fo3d")

Creating 3D datasets in FiftyOne is as simple as creating samples whose filepaths point to an FO3D file:

dataset = fo.Dataset()


sample = fo.Sample(filepath="/path/to/scene.fo3d")
dataset.add_sample(sample)

print(dataset.media_type)  # 3d

3D samples may contain any type and number of custom fields, including 3D detections and 3D polylines, which are natively visualizable by the App’s 3D visualizer.

detection = fo.Detection(
    label="vehicle",
    location=[0.47, 1.49, 69.44],
    dimensions=[2.85, 2.63, 12.34],
    rotation=[0, -1.56, 0],
)

sample = fo.Sample(
    filepath="/path/to/scene.fo3d",
    ground_truth=fo.Detections(detections=[detection]),
)

To get started exploring 3D datasets, try loading the quickstart-3d dataset from the FiftyOne Dataset Zoo:

import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart-3d")
session = fo.launch_app(dataset)

The FiftyOne App’s 3D Visualizer provides a rich heads-up display that offers full control over the 3D scene, including:

• Full 3D orbit controls
• Graph view of the 3D objects in the scene
• Visibility toggles for each scene element
• Sliders to adjust the rendering of each object
• Controls for lights, gridlines, and more

FiftyOne also supports grouped datasets, which contain multiple slices of samples of possibly different modalities. Grouped datasets can be used to represent multiview scenes, where data for multiple perspectives of the same scene can be stored, visualized, and queried in ways that respect the relationships between the slices of data.

Grouped datasets can contain image, video, and-now as of FiftyOne 0.24–3D slices!

To get started with 3D grouped datasets in FiftyOne, check out the quickstart-groups dataset, which contains a point cloud slice:

import fiftyone as fo
import fiftyone.utils.utils3d as fou3d
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart-groups")

# Populate orthographic projections
fou3d.compute_orthographic_projection_images(dataset, (-1, 512), "/tmp/proj")

dataset.group_slice = "pcd"
session = fo.launch_app(dataset)

Custom Workspaces

The FiftyOne App has a powerful Spaces feature that provides a customizable framework for organizing interactive panels of information:

FiftyOne natively includes the following panel types:

• Samples panel: the media grid that loads by default when you launch the App
• Histograms panel: a dashboard of histograms for the fields of your dataset
• Embeddings panel: a canvas for working with embeddings visualizations
• Map panel: visualizes the geolocation data of datasets that have a GeoLocation field

Remember that you can also build and use custom Panels via plugins!

Now in FiftyOne 0.24, if you find yourself frequently using or recreating a certain layout, you can save it as a workspace with a name of your choice and then load it later via the App or programmatically.

Once you’ve configured a layout of interest, click the “Unsaved workspace” icon in the upper right corner to open the workspaces menu and save your current workspace with a name and optional description and color of your choice:

You can load or edit saved workspaces at any time later via this same menu.

Saved workspaces include all aspects of your current layout, including panel types, positions, sizes, and even the current state of each panel!

You can also programmatically create and manage saved workspaces in Python:

import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")

samples_panel = fo.Panel(type="Samples", pinned=True)

histograms_panel = fo.Panel(
    type="Histograms",
    state=dict(plot="Labels"),
)

embeddings_panel = fo.Panel(
    type="Embeddings",
    state=dict(brainResult="img_viz", colorByField="metadata.size_bytes"),
)

workspace = fo.Space(
    children=[
        fo.Space(
            children=[
                fo.Space(children=[samples_panel]),
                fo.Space(children=[histograms_panel]),
            ],
            orientation="horizontal",
        ),
        fo.Space(children=[embeddings_panel]),
    ],
    orientation="vertical",
)

dataset.save_workspace(
    "my-workspace",
    workspace,
    description="Samples, embeddings, histograms, oh my!",
    color="#FF6D04",
)

With saved workspaces, you have infinite flexibility to create custom App layouts that can be persisted to your dataset and loaded later with a single click or function call.

For example, you could configure workspaces optimized for different parts of your model development lifecycle, such as:

• Embeddings visualization
• Finding image quality issues
• Reverse image search
• Zero shot prediction
• Model evaluation
• Active learning

You can also build a completely custom workspace composed partly or entirely of panels from your own plugins.

At Voxel51, we’re just getting started tapping into the potential of custom panels and workspaces in FiftyOne. We’ve got some exciting new features under development that will make it easier than ever to add new (and custom) panels to FiftyOne that deepen its value as the center of your data + model development. So stay tuned!

Community Updates

Shoutout to the following community members who contributed to FiftyOne 0.24:

• Rohith Srinivasan contributed #4296 Adding DeepSort object tracking utility
• Lucas Antunes contributed #4304 Add support for loading FiftyOneDataset exports from different operating systems
• Theo Patron contributed #4309 Add support for non-sequential COCO category IDs
• Fatih Akyon contributed #4398 Add Ultralytics YOLOv8 models pretrained on Open Images V7
• Rohith Srinivasan contributed #4425 Update mask_path argument description for Segmentation class

The FiftyOne community continues to grow!

• 2,700+ FiftyOne Slack members
• 6,800+ stars on GitHub
• 23,000+ Meetup members
• 100+ contributors

What’s Next?

• If you like what you see on GitHub, give the project a star.
• Get started! We’ve made it easy to get up and running in a few minutes.
• Join the FiftyOne Slack community, we’re always happy to help.

Originally published at https://voxel51.com on May 30, 2024.