Skip to content
  • There are no suggestions because the search field is empty.

Best Practices for Mass Data Uploads in Proqio

Large data imports can significantly impact system performance if they are uploaded in bulk. When millions of records—each containing multiple variables—are sent at the same time, Proqio must process them sequentially for each instrument. This creates a long processing queue that delays the handling of live instrument data. As a result, real-time updates may be slowed or temporarily blocked.
 
To ensure smooth system performance, especially for projects that require timely processing of live measurements, consider the following best practices detailed in this article.
 

Summary

Mass data uploads can disrupt Proqio’s processing pipeline when sent as a single large batch. The most effective way to prevent delays is to:
  1. Upload data continuously throughout the day (best option). 
  2. Schedule large uploads outside times when live data is required. 
  3. Aggregate or reduce dataset size before uploading. 
Following these guidelines ensures reliable platform performance and timely processing of both historical and live instrument data.

1. Upload Data Continuously Instead of in Bulk (Recommended)

The ideal approach is to send records as they are generated, rather than accumulating them and sending them all at once.
Benefits:
  • Prevents large processing spikes.
  • Ensures live instrument data continues to be processed without delays.
  • Reduces strain on the processing queue by distributing load throughout the day.
If your workflow allows for it, this is the most effective and scalable solution.
 
 

2. If Continuous Upload Is Not Possible, Schedule Bulk Uploads Outside Peak Times

If you must upload large batches, plan them for times when:
  • No live data is being transmitted, or 
  • The project does not require immediate instrument processing.
This avoids competing with time-critical live data and helps maintain platform responsiveness for active monitoring tasks.

 

3. Aggregate or Downsample Data Before Sending

When neither continuous uploads nor off-peak scheduling is viable, consider reducing the size of the dataset prior to upload.
Possible approaches:
  • Aggregate multiple readings into summary values.
  • Downsample at intervals where full resolution is not needed.
  • Compress high-frequency data into grouped batches.
This reduces the total data volume and minimizes ingestion load.