Introducing Apache Pinot 0.5.0
We are excited to announce that Apache Pinot 0.5.0 has been released in September 2020. Apache Pinot is a real-time distributed datastore, designed to answer OLAP queries with low latency. In this release, we continue to add new query and ingestion capacity and at the same time make Pinot more performant, easy to manage and reliable. This post highlights a few features in this release. The full release notes can be found here.
New Cluster Manager UI and Query Console
Pinot 0.5.0 replaces the existing consoles with an app developed using React + TypeScript + Material UI. The new consoles have a modern Material UI styling (see samples below). There are more status pages added to the cluster manager UI including table details page, segment details pages, zookeeper data exploration page, query response stats pages and so on. Hope you will enjoy the new looks of Pinot!
Groovy Transform UDF in Pinot Query
Backed in 0.4.0, Pinot added the support for transformation functions during ingestion using Apache Groovy, which allowed users to define derived fields directly in the schema. In 0.5.0, the Groovy transformation functions are also added in Pinot queries. Pinot queries now are able to run a Groovy script (e.g., string concatenation) in queries as a UDF. For example, you can now run the following Pinot query to concatenate the return values of columnA and columnB:
SELECT GROOVY(// Function result type
‘{“returnType”: “STRING”, “isSingleValue”: true}’, // Function definition
‘arg0 + “ “ + arg1’,// Arguments
columnA, columnB) FROM myTable
This feature makes transformation of query results much easier and it is all done by Pinot directly.
JSON Object Conversion During Ingestion
Before 0.5.0, map type complex data in Pinot are converted to string. However, the format of the stored string is not parsable using common jsonExtract functions. Consequently, the fields in the complex object can not be retrieved easily.
Pinot 0.5.0 rewrites the JSON objects conversion during ingestion. With that, complex map objects with nested structure are converted into proper JSON strings. More important, these stored JSON objects can later be queried using jsonExtractScalar() functions. For example, now you can add the following to your PQL query:
SELECT JSONEXTRACTSCALAR(profile_json_str, // jsonField‘$.name’, // jsonPath to extract the scalar‘STRING’ // ‘resultType’) FROM myTable
The complete list of JSON functions and its syntax can be found here.
Support for JDBC clients
Pinot 0.5.0 continues to expand its supports for popular client applications and input formats. It adds support for JDBC driver so that native Java JDBC clients can execute queries on Pinot. Please check out this doc for examples and the complete documentations.
Support for Protocol Buffer Input Format
Pinot 0.5.0 continues to expand its supports for popular input formats. It now supports to ingestion of data inputs in the popular Proto-buf format.
Allow Star-Tree Creation During Segment Load
Star-Tree index is an index for speeding up aggregation queries (e.g., SUM, COUNT and so on). It provides excellent query latency improvement with moderate space overhead. Previously if one wants to add star-tree indexes to an existing table, he/she has to re-create the entire table. Pinot 0.5 allows users to modify table configs to add Star-Tree index and then reload the table segments to add the indexes on the fly.
Add Geo Support for Pinot Queries
Many business operations need to query geo spatial data. Pinot 0.5.0 adds native supports for geo-spatial data models including geometry and geography. Pinot users now can add common geo relationship function like ST_CONTAINS and ST_EQUALS to their Pinot queries. The complete list of supported function can be found in this PR. This is the beginning of a bigger project to provide a complete and efficient solution for Geo data query in Pinot. Stay tuned!
Download page: https://pinot.apache.org/download/
Getting started: https://docs.pinot.apache.org/getting-started
Slack channel: https://communityinviter.com/apps/apache-pinot/apache-pinot
Special thanks
We would like to take a moment to thank the Pinot community for supporting our Product. We have seen a healthy increase of commits to Pinot code base since September 2019 (see the graph below). Here we also like to thank the committers and contributors for their contribution to this release.
