How to load more than 100 stories with the JS client?

If you need to retrieve a list of stories or, in general, a list of elements managed by Storyblok (links, assets, etc.) you need to consider managing the pagination in case you will retrieve a huge number of elements.

The suggestion is to use the Open Source Storyblok JS client that provides you with some helpers and methods to manage the integration with the Storyblok APIs (Management API and Content Delivery API).

You can use the getAll() method to retrieve all the stories. The getAll() method is very convenient because it automatically manages the pagination mechanism and allows you to use all the parameters for filtering the items. A practical example of how to use the getAll() method is in the article Generating an RSS Feed. If you want to manage the pagination or understand how it works under the hood, you can use the get() method. Calling the get() method, you can set the parameters per_page and page. With the per_page parameter, you can define the page size in terms of the number of items (in this case, the number of stories) for each page. With the page parameter, you can set the number of the page. When you use the pagination with these two parameters, you will receive one important information in the HTTP headers: the total header. The total value represents the total number of items (across all the pages). With the total parameter, you can calculate how many pages you need to retrieve, so how many API calls you need to call.

So, the strategy is that you can perform the first API call for the first page for retrieving the first set of elements and the total information. With the total header, you can calculate if, within the first response, you have all the stories or if you need to perform other API calls incrementing the page parameter. At the end, you need to merge all the responses you collected across all the paginated API calls

The first step is to be sure you are using the Storyblok JS Client. You can install manually, or if you are using one of the Storyblok Frontend SDKs for example, for React, Nuxt, Vue, SvelteKit the Storyblok JS Client is a dependency of the SDKs.

// 001 Import the Storyblok JS Client
import StoryblokClient from "storyblok-js-client";

// 002 init the Stroyblok client with access token
const Storyblok = new StoryblokClient({
  accessToken: "YOUR_PREVIEW_TOKEN",
  cache: {
    clear: "auto",
    type: "memory",
  },
});

// 003 setting some parameters, 25 as per_page parameter
const perPage = 25;
let params = {
  version: "draft",
  per_page: perPage,
  page: 1,
};

// 004 retrieving the first page
let firstResponse = await Storyblok.get("cdn/stories", params);

// 005 calculating the needed pages via total information
let lastPage = firstResponse.total
  ? Math.ceil(firstResponse.total / perPage)
  : 1;

// 006 retrieving the other pages (accessing to the data.stories)
let otherStories = [];
for (let currentPage = 2; currentPage <= lastPage; currentPage++) {
  params.page = currentPage;
  otherStories.push((await Storyblok.get("cdn/stories", params)).data.stories);
}

// 007 merging all the arrays
let allStories = [firstResponse.data.stories, ...otherStories].flat();
console.log(allStories);
console.log(allStories.length);
console.log(perPage);
console.log(firstResponse.total);

The code snippet above highlights the steps needed to retrieve multiple pages of data. Certainly, the javascript code can be optimized, but let's follow the basic steps needed:

001 importing the Storyblok JS Client StoryblokClient;
002 initializing the Stroyblok client with the access token;
003 setting parameters, the page and the per_page parameters for the pagination;
004 retrieving the first page to retrieve the first chunk of data and the total information in the HTTP headers;
005 calculating how many pages are needed via total information;
006 retrieving the other pages (accessing the data.stories JSON attribute);
007 merging all the arrays, obtaining one array with all the stories.

Some practical scenarios

Managing the rate limits

According to the limits defined here https://www.storyblok.com/docs/api/content-delivery/v2#topics/rate-limit you can perform some fine-tuning within the rateLimits parameter. For example, if you want to increase the rate limits, in the StoryblokClient initialization, you can set the rateLimit parameter as shown in the following snippet.

const Storyblok = new StoryblokClient({
  accessToken: "YOUR_PREVIEW_TOKEN",
  cache: {
    clear: "auto",
    type: "memory",
  },
  rateLimit: 20,
});

Retrieving the stories published in the current month

This example allows you to understand how to combine the getAll() method and the filtering on the published date. You can customize the code according to your needs:

const date = new Date();
let month = date.getMonth() + 1; // retrieving the current month (months are 0 index)
let year = date.getFullYear();
let stories = await Storyblok.getAll("cdn/stories", {
  version: "published",
  per_page: 25,
  cv: "undefined",
  published_at_gt: `${year}-${month}-20 00:00`,
});

As you can see, we are using the published_at_gt parameter that acts on the published_at attribute with the gt operator (greater than).

You can see all the query parameters you can use, like starts_with, search_term, content_type, and published_at_lt in the Content Delivery API documentation. You can also filter for the content with the filter_query parameter here.

Recap

Retrieving multiple stories or all the stories can be useful in multiple contexts when you need to retrieve multiple stories like:

building your frontend with the SSG approach, and you need to access all the stories (maybe filtered via the published date);
building your RSS feed, you can read specific suggestions in the How to generate RSS feed article;
building your sitemap and the information retrieved (or the filter you want to apply) from the links endpoint is not enough for your case. The article for building the sitemap using the links endpoint is here;
integrating with an external system, and you need to retrieve the content from Storyblok.