Support for reference data up to 5GB and new improvements in Azure Stream Analytics now generally available

The latest update of Azure Stream Analytics, now deployed in every supported region, now supports larger reference data as well as other improvements. Here is the list of significant changes.

Larger reference data

Reference data is a key feature of Stream Analytics, enabling to augment the stream with static or slow changing data coming from Blob storage, Data Lake or SQL Databases. Reference data is versioned, enabling to always get the same results, even when we “replay” the stream. It is also pre-loaded by the Stream Analytics job in order to provide fast lookups.

Previously reference data size was limited to 300 MB, with this new version customers would be able to use up to 5 GB of reference data. More info here. Together with this change, we improved the way reference data is loaded to minimize latency when reference data is updated.

Other product improvements

This update includes hundreds of performance and functional improvements. While we are not releasing full release notes, but we wanted to call out the following ones:

• Composite keys are now supported for blob output partitioning. Previously we did not support the use of characters such a ‘/’ and ‘.’ in the column used for the output path. Now, we support a wider range of characters, allowing to define hierarchy of files directly in the column used for the output path. More information is available here.
• Customer logging when using JavaScript user-defined functions (UDF) or user-defined aggregates (UDA). Users can now emit their own log from UDF and UDA code, enabling better debuggability. More info here.
• Previously, at the start of the job, older events were not processed until the end of the late arrival window. Now, they are not delayed.

Learn more about Azure Stream Analytics