Azure Data Lake Storage Gen2

Export telemetry directly to Azure Data Lake Storage Gen2 (ADLS Gen2). This connector is optimized for high-throughput scenarios, supporting efficient batching and popular big-data formats like Apache Parquet.

Azure Setup Guide

To use this connector, you need to configure Azure resources to allow Proxus to write data.

1. Create a Storage Account

1. Go to the Azure Portal.
2. Create a new Storage Account.
3. Under the Advanced tab, enable Hierarchical namespace. This is required for ADLS Gen2.
4. Note down the Account Name.

2. Create a Container

1. Navigate to your new Storage Account.
2. Go to Data storage > Containers.
3. Create a new container (e.g., telemetry). This will be your FileSystem parameter.

3. Register an Application (Service Principal)

1. Search for Microsoft Entra ID (formerly Azure AD).
2. Go to App registrations > New registration.
3. Name it (e.g., "Proxus-Integration") and register.
4. From the Overview page, copy the Application (client) ID and Directory (tenant) ID.

4. Create a Client Secret

1. In your App Registration, go to Certificates & secrets.
2. Create a New client secret.
3. Copy the Value immediately (you won't be able to see it later). This is your ClientSecret.

5. Assign Permissions (RBAC)

1. Go back to your Storage Account.
2. Select Access Control (IAM) > Add > Add role assignment.
3. Select the role Storage Blob Data Contributor (Note: "Contributor" or "Owner" is NOT sufficient for data access).
4. Assign access to User, group, or service principal.
5. Select the App you registered in Step 3.

Configuration Parameters

To configure the connector in the Proxus Management Console, navigate to Integrations > Outbound Connectors and create a new Azure Data Lake target.

Click the Edit Parameters button to open the configuration dialog.

Authentication

Authentication is handled via a Service Principal.

Storage Settings

Batching & Performance

These settings control when data is flushed from the internal buffer to Azure.

Performance Considerations

Avoiding the "Small File Problem"

In big data ecosystems, having millions of small files (e.g., a few KBs or MBs each) significantly degrades the performance of analytics engines like Apache Spark, Azure Databricks, or Azure Synapse. This is known as the "Small File Problem".

To avoid this, we recommend:

• Increase Batching Thresholds: Adjust MaxBatchBytes and MaxBatchCount to create larger files. While Parquet files are ideally 128MB - 512MB for big data tools, even reaching 32MB - 64MB per file is a major improvement over frequent small flushes.
• Tune Wait Times: Increase MaxWaitSeconds (e.g., to 1800 for 30 minutes) if your data volume is low, allowing Proxus to collect more data before creating a new file.
• Optimized Path Patterns: Avoid using too many high-cardinality dynamic tokens (like {guid} or {deviceId}) in the middle of your directory structure if it causes too much partitioning.

Path Pattern Tokens

The PathPattern parameter supports dynamic tokens that are replaced at runtime:

Default Pattern: {region}/{subject}/{yyyy}/{MM}/{dd}/{HH}/telemetry-{yyyyMMddHH}-{deviceName}-{guid}

Internal Behavior

• Reliability: Uses a durable outbox mechanism. Messages are acknowledged (ACK) only after a successful upload to Azure. If an upload fails, messages are negatively acknowledged (NAK) and retried.
• Memory Management: Utilizes advanced memory pooling (RecyclableMemoryStream) to prevent Large Object Heap (LOH) fragmentation during high-volume exports.
• Parquet Schema: When using parquet format, files are written with a strictly typed schema including deviceId, deviceName, topic, measureName, measureValueType, and values for double, long, bool, and string types along with a timestamp.