Dune Engine V2 is Dune's new query engine that brings a new level of performance, scalability and functionality to Dune's core set of tools that enables wizards to query, extract, and visualize the vast amounts of data on the blockchain.
It leverages Apache Spark to enable increased performance of complex queries, handle data scale, and enable cross-chain queries all within the same UI.
All of the data sources contained in this section are available for querying with the new query engine today. Currently we have the following data available to query:
DuneV2 changes our entire database architecture. We are transitioning away from a PostgresQL database to an Instance of Apache Spark hosted on Databricks. The difference between the two systems can be summarized as follows:
Instead of PostgresQL, we will now use DatabricksSQL. The change in SQL keywords is minimal but might be relevant for some of your querying habits.
Spark is a column oriented database in contrast to PostgresQL’s row oriented approach.
traditional indexes are replaced by column chunk level min/max values
Abstraction in DuneV2 will run on dbt (data build tool). dbt enables analytics engineers to transform data in their warehouses by simply writing select statements. dbt handles turning these select statements into tables and views.
This will make abstractions more robust, scaleable and easier to work with.
One final note, as the query engine is still in in beta you may run into bugs or have feedback on how it can be improved, feel free to share it with us on Discord and Canny.