Azure Data Explorer gets new engine, numerous enhancements and Synapse integration

3.png

An Azure Data Explorer dashboard


Credit: Microsoft

Today, at a dedicated online event for Azure Data Explorer (ADX), Microsoft is announcing numerous enhancements to the service, including a next-gen release of the underlying engine and an array of integration points that should make it more accessible, more enticing and more useful. ADX, which is most often used for telemetry data lake workloads and analytical solutions as a service, will now run even faster overall than it had, will have numerous optimizations and will connect with a variety of other data services, streaming data sources and data visualization solutions. This will help a service that’s been very successful but not especially well-known, even among Azure analytics experts, achieve more mainstream appeal.

Performance gains galore

What new features are coming to ADX? To start with, Microsoft’s introducing a new version of the core engine (in preview, with GA expected in February), which takes a wholly different strategy to querying data. The Kusto v3 engine will generate multiple versions of the desired query, use the fastest, and compile it to native code before executing, so that it runs at maximum speed. The indexing layer in the v3 engine has also been rewritten. As a result of these changes, Microsoft says queries will run between 2x and 30x faster.

And beyond this raw performance gain, ADX will now offer self-refreshing materialized views, query result set caching and configurable sharding/partitioning. Near real time scoring with machine learning models — including those hosted on Azure Machine Learning as well as those from other platforms, packaged in ONNX format — is being added as well. Fast Fourier Transforms, geospatial joins and polynomial regression are onboarding too. ADX is also getting row-level security capabilities that will make it more appealing to customers who want to support a wide