Editar

Compartir a través de

Loop and page over data returned from the Azure SDK for JavaScript

When working with Azure services, you often need to process large sets of data. The Azure SDK for JavaScript provides async iterators to help manage this task efficiently. This article explains what async iterators are, how to use them, and provides examples for key Azure services.

What are async Iterators?

Async iterators are a feature in modern JavaScript that allow you to consume data asynchronously. They're useful for handling paginated data from APIs. Async iterators use the for-await-of loop to iterate over data, fetching it as needed.

Using async iterators provides several advantages:

  • Simplified Syntax: The for-await-of loop makes consuming async iterators straightforward.
  • On-Demand Data Fetching: Fetch only the data you need, reducing memory usage and load on the backend.
  • Future Compatibility: Async iterators are a standard feature in JavaScript, ensuring compatibility with future updates and libraries.

If you're new to async iterators, the following concepts help to understand how paging works in Azure SDKs for JavaScript.

  • Async Functions: Functions that return a Promise.
  • Generators: Functions that can be paused and resumed, yielding multiple values.
  • Async Generators: Combine the features of async functions and generators to produce async iterators.

Azure client libraries use async iterators to handle potentially large collections of data. Below are examples of how to use async iterators with various Azure services.

Loop over a few items

If you result set is only a few items, you can loop through that small list. The following code loops through a small set of containers in Azure Storage:

for await (const container of blobServiceClient.listContainers()) {
  console.log(`Container: ${container.name}`);
}

Loop over data by page

If your data set is larger, you may want to return the data in pages, then iterate over items in each page. The following code loops through a data by page, then each item.

const firstPage = await blobServiceClient.listContainers().byPage().next();

const continuationToken = firstPage.value.continuationToken;

// The iterator also supports iteration by page.
for await (const page of blobServiceClient
  .listContainers()
  .byPage({ continuationToken })) {
  if (page.containerItems) {
    for (const container of page.containerItems) {
      console.log(`Container: ${container.name}`);
    }
  }
}

Continue looping

If you need to have more control over the loop, including resuming the loop, use a continuation token. The paged iterator also supports resuming from a continuation token. In the following example, we use the continuation token from the first iteration to resume iteration at the second page.

console.log('Starting to process pages...');

let processingComplete = false;
let pageNumber = 1;

try {
  let page = undefined;
  let continuationToken = undefined;

  do {
    // Get a page of results
    page = await blobServiceClient.listContainers().byPage().next();

    // Get the continuation token from the current page
    continuationToken = page?.value?.continuationToken;

    console.log(
      `Processing page ${pageNumber}, items ${page.value.containerItems?.length || 0} with continuation token: ${continuationToken || 'none'}`
    );
    console.log(page.value);

    // Scenario to continue paging:
    // Perform an operation on the current page results
    // if the operation returns true, stop processing
    processingComplete = await fakeProcessor(page.value);
    if (processingComplete) {
      console.log('Stopping processing.');
      break;
    }
    console.log(
      `Processing complete for page ${pageNumber++}: get next page if a continuation token exists`
    );
  } while (continuationToken && !processingComplete);

  console.log(
    `Finished processing. Total pages processed: ${pageNumber - 1}`
  );

Additional resources