API Design for a React Tree Desk

A latest React freelance challenge of mine provided me a difficult job: The shopper needed to have a tree desk element in React. The position mannequin for this was MacOS’s Finder and its tree view; and as a cherry on high: it ought to be capable to fetch asynchronously as much as 100.000 objects in chunks as paginated and nested lists.

Over the subsequent months, I constructed this element for my freelance shopper. Earlier than I began the implementation, I wrote down all of the challenges I might face alongside the way in which and the way I might clear up them. Right here I wish to offer you a walkthrough of my thought course of, how I designed the API necessities, and the way I applied this tree desk element in React ultimately.

The main focus for this text is on the API design choices. In hindsight, beginning with the distant information API specification first was the very best determination I made for this challenge. Solely when you have got a well-designed API to your desk necessities, you possibly can construct the frontend and backend correctly.

For what it is price: Not one of the present React desk libraries have been adequate to satisfy all the necessities. Thus I needed to constructed a customized answer which is now obtainable as open supply library for React.

React Tree Desk: Record Construction

First, we now have to outline what sort of information we have to visualize a tree desk in React. Since it is a desk and never solely a listing, we would want a couple of property to indicate up for every row. So a simple checklist of things could be:

const information = [
  {
    id: '1',
    name: 'profile image1.png',
    size: 234561,
    dateModified: '01-01-2021'
  },
  {
    id: '2',
    name: 'profile image2.png',
    size: 346221,
    dateModified: '02-01-2021'
  },
  {
    id: '3',
    name: 'profile image3.png',
    size: 124112,
    dateModified: '01-02-2021'
  },
];
In a checklist element, we’d show every merchandise as a row with its properties identify, measurement, and dateModified as cells. If we’d remodel the checklist to a desk element, it could have a column for every property.
As a way to hold the next examples extra light-weight, I’ll omit the measurement and dateModified properties, as a result of they do not immediately impression the implementation particulars of the tree desk.
const information = [
  {
    id: '1',
    name: 'profile image1.png',
  },
  {
    id: '2',
    name: 'profile image2.png',
  },
  {
    id: '3',
    name: 'profile image3.png',
  },
];
Nonetheless, in a tree element the information ought to comply with a tree construction as a substitute of a listing construction. Subsequently, we adapt the earlier checklist with objects to a tree with nodes:
const information = [
  { id: '0', name: 'profile image.png' },
  { id: '51', name: 'Thumbnails', nodes: [] },
  {
    id: '52',
    identify: 'Excessive Decision',
    nodes: [
      { id: '1', name: 'profile image1.png' },
      { id: '2', name: 'profile image2.png' },
      { id: '3', name: 'profile image3.png' },
      { id: '4', name: 'image4.png' },
      { id: '5', name: 'image5.png' },
    ]
  },
];
We are able to see how this tree construction would unfold as a hierarchy with folders and recordsdata in a MacOS Finder element. Whereas recordsdata should not have a nodes property, folders have both empty or crammed nodes. The previous could be an empty folder.
By having the nodes property at our fingers, we will distinguish every node within the tree as one in all three choices:

nodes: undefined | null -> file
nodes: [] -> empty folder
nodes: [{ ... }] -> crammed folder

As various, one might declare a isFolder boolean as property for every node, nevertheless, this would not hold it DRY — as a substitute it could introduce redundancy immediately, inconsistency ultimately, and would bloat our information contemplating that we wish to switch hundreds of nodes over the wire.
Final, this tree construction permits us to introduce nested bushes too:
const information = [
  { id: '0', name: 'profile image.png' },
  { id: '51', name: 'Thumbnails', nodes: [] },
  {
    id: '52',
    identify: 'Excessive Decision',
    nodes: [
      {
        id: '53',
        name: 'Favorites',
        nodes: [
          { id: '4', name: 'image4.png' },
          { id: '5', name: 'image5.png' },
        ]
      },
      { id: '1', identify: 'profile image1.png' },
      { id: '2', identify: 'profile image2.png' },
      { id: '3', identify: 'profile image3.png' },
    ]
  },
];
Whether or not we now have a folder or a file first within the information construction doesn’t matter. If we’d wish to render this tree construction client-side, we might type all nodes as lists primarily based on the situation of their entry’s obtainable nodes property to indicate both folders or recordsdata first. Similar goes after we ship the information from the server, we’d let the server determine during which order the information arrives on the shopper if no server-side type function is current.
React Tree Desk: Paginated Record
After we now have finalized the information construction for our tree desk, we have to conceptualize how we wish to chunk the information into smaller items and how one can request these items from a distant API. For the time being, our request would seem like the next to fetch all the information:
const request = {
  path: '/nodes',
  physique: {},
};
The response could be a tree information construction which we now have outlined earlier than. Nonetheless, as talked about earlier, for this job we might be coping with hundreds of things in a listing (and its nested lists), so it’s a necessity to separate up the information. That is generally achieved with pagination and paginated lists. Thus, the request wants to simply accept an offset and a restrict argument:
const request = {
  path: '/nodes',
  physique: {
    offset: quantity,
    restrict: quantity,
  },
};
Whereas the offset dictates at which index we wish to begin the paginated checklist, the restrict dictates what number of objects needs to be included. The next instance will illustrate it:
const checklist = [
  { id: '34151', name: 'a' },
  { id: '23114', name: 'b' },
  { id: '23171', name: 'c' },
  { id: '46733', name: 'd' },
];
const paginatedList = extractPaginatedList(
  checklist,
  {
    offset: 1,
    restrict: 2,
  }
);
console.log(paginatedList);
A response to our offset-based pagination request might have the next information construction:
const outcome = {
  nodes: [node],
  pageInfo:  null,
  ,
};
A substitute for an offset-based pagination — which merely takes the index of a listing and due to this fact could possibly be fragile when CRUD operations are utilized between requests — could be utilizing a cursor-based pagination. The next instance will illustrate it:
const checklist = [
  { id: '34151', name: 'a' },
  { id: '23114', name: 'b' },
  { id: '23171', name: 'c' },
  { id: '46733', name: 'd' },
];
const paginatedList = extractPaginatedList(
  checklist,
  {
    cursor: 23114,
    restrict: 2,
  }
);
console.log(paginatedList);
As a way to hold issues easy, we’ll keep on with the offset-based pagination although.
Let’s stroll by means of a state of affairs primarily based on our beforehand outlined information construction the place a request could possibly be the next:
const request = {
  path: '/nodes',
  physique: {
    offset: 0,
    restrict: 2,
  },
};
If we extrapolate this onto our information from earlier than, the response — now a paginated checklist — might seem like the next:
const outcome = {
  nodes: [
    { id: '0', name: 'profile image.png' },
    { id: '51', name: 'Thumbnails', nodes: [] },
  ],
  pageInfo: {
    complete: 3,
    nextOffset: 2,
  }
};
As a result of the nextOffset is given and never null, we might fetch one other paginated checklist. From a UI/UX perspective, this could possibly be carried out with a “Load Extra” button on the finish of our checklist (handbook execution) or with infinite scrolling (computerized execution). The following request then would seem like the next:
const request = {
  path: '/nodes',
  physique: {
    offset: 2,
    restrict: 2,
  },
};
The returned outcome could be a paginated checklist with just one merchandise, as a result of our supply information has solely three objects within the checklist. Since we already fetched two objects earlier than, what’s left is just one merchandise. Subsequently, the subsequent offset is null and we can not fetch extra pages afterward:
const outcome = {
  nodes: [
    { id: '52', name: 'High Resolution', nodes: [] },
  ],
  pageInfo: {
    complete: 3,
    nextOffset: null,
  }
};
Discover how we’re in a position to fetch pages (paginated lists) of our supply checklist with solely utilizing offset and restrict. Through the use of this method, we will request all of the top-level nodes. With each further request, the frontend can merge the outcomes by concatenating the nodes and changing the pageInfo with the newest one:
const outcome = {
  nodes: [
    { id: '0', name: 'profile image.png' },
    { id: '51', name: 'Thumbnails', nodes: [] },
    { id: '52', identify: 'Excessive Decision', nodes: [] },
  ],
  pageInfo: {
    complete: 3,
    nextOffset: null,
  }
};
Now what concerning the tree construction? You’ll have seen that the final node which we now have fetched has an empty nodes property regardless that it is not empty in our supply information. That is by selection, as a result of when coping with a number of information the nodes property could possibly be full of hundreds of entries. Then, regardless that we now have our pagination function in place now, we would not get any benefits from it and would get a efficiency hit.
React Tree Desk: Nested Record
The earlier part was about splitting up lists into paginated lists (pages) whereas retaining the checklist itself shallow by not populating the nodes property. This part is about populating the nodes property asynchronously.
Thus far, we now have carried out requests for paginated information, not for nested information. If a person needs to navigate right into a tree by increasing a node within the UI, we will fetch its content material (right here nodes). Subsequently, by extending the earlier request with an id argument, we will specify what node’s content material we wish to request:
const request = {
  path: '/nodes',
  physique:  null ,
};
Since id may be null or undefined, our earlier requests for the top-level checklist are nonetheless legitimate. After fetching the top-level pages, the person sees that the displayed node with the id 52 is a folder which might have content material. Now the request for this folder’s content material might seem like the next:
const request = {
  path: '/nodes',
  physique: {
    id: '52',
    offset: 0,
    restrict: 2,
  },
};
Whereas we will use the id to request a node’s content material, we will nonetheless apply our offset and restrict arguments to fetch solely a fraction of it as we will see within the following outcome:
const outcome = {
  nodes: [
    { id: '53', name: 'Favorites', nodes: [] },
    { id: '1', identify: 'profile image1.png' },
  ]
  pageInfo: {
    complete: 4,
    nextOffset: 2,
  }
};
The frontend merges the outcomes by inserting nodes and pageInfo into the earlier outcome:
const outcome = {
  nodes: [
    { id: '0', name: 'profile image.png' },
    { id: '51', name: 'Thumbnails', nodes: [] },
    {
      id: '52',
      identify: 'Excessive Decision',
      nodes: [
        { id: '53', name: 'Favorites', nodes: [] },
        { id: '1', identify: 'profile image1.png' },
      ],
      pageInfo: {
        complete: 4,
        nextOffset: 2,
      }
    },
  ],
  pageInfo: {
    complete: 3,
    nextOffset: null,
  }
};
From there, a person can additional increase the tree by clicking the folder with the id 53 (request for nested information) or load extra information beneath the entry with the id 1 (request for paginated information).
There are a couple of extra issues to notice right here:
First, all nodes with an empty nodes property might have potential content material. For the time being, each time a person expands a tree node there could be a request which returns an empty checklist of nodes. We experimented with a hasNodes boolean flag per node which might forestall the information fetching on the client-side if there isn’t a content material. Ultimately we eliminated it although, as a result of it made retaining server-side information and client-side state in sync extra complicated when a number of customers interacted (e.g. person A creates a file in an empty folder, person B doesn’t load content material as a result of their property nonetheless says no content material) with the applying.
Second, regardless that we created an API which makes it attainable to request structured tree information in smaller chunks, we have to deal with a number of this information as state on the client-side. We have to deal with the merging of a number of outcomes into one state object, but in addition have to deal with retaining this state in sync with the distant information for multi-user collaboration.
React Tree Desk: Sync
If all information could be fetched without delay, a easy refetch of all this information could be adequate to maintain the information in sync between frontend and backend. Nonetheless, since we’re requesting paginated and nested lists, one in all these states (paginated/nested pages) might go stale in a multi-user software, and thus refetching this one state will get extra complicated.
In our state of affairs, we had no sources to implement net sockets for real-time notifications of desk adjustments, so we needed to go along with HTTP lengthy polling and optimistic updates.
What’s wanted is a brand new request which fetches particular paginated and nested pages on demand to replace the merged outcomes from the earlier requests:
const request = {
  path: '/nodes-sync',
  physique: {
    pages: [ null ],
  },
};
So if we return and examine what information we fetched to date, we will iterate over all pageInfo properties from the client-side state and due to this fact would want the next request to get an up to date model of all pages:
const request = {
  path: '/nodes-sync',
  physique: {
    pages: [
      {
        id: null,
        offset: 0,
        limit: 3,
      },
      {
        id: '52',
        offset: 0,
        limit: 2,
      },
    ],
  },
};
You see, regardless that we made three requests earlier than, we solely have two pageInfo properties in our client-side state, as a result of one in all them has been overridden earlier by a subsequent pageInfo property. Thus we will request the replace for less than two pages.
With this new API, we acquire full management of how we wish to refetch this information: We are able to use the pageInfo from the client-side state (as seen within the final instance) or do one thing fully completely different.
The outcome from the earlier request would seem like the next:
const outcome = {
  pages: [
    {
      nodes: [
        { id: '0', name: 'profile image.png' },
        { id: '51', name: 'Thumbnails', nodes: [] },
        { id: '52', identify: 'Excessive Decision', nodes: [] },
      ],
      pageInfo: {
        complete: 3,
        nextOffset: null,
      }
    },
    {
      nodes: [
        { id: '53', name: 'Favorites', nodes: [] },
        { id: '1', identify: 'profile image1.png' },
      ],
      pageInfo: {
        complete: 4,
        nextOffset: 2,
      }
    }
  ],
};
Why is the outcome a listing of pages? As a substitute of returning a listing of pages, we might return a hierarchy. Nonetheless, we realized, in our case, that by returning a listing, the shopper will get full management over what pages to refetch (e.g. pages that needn’t share the identical hierarchy). As well as, the shopper can simply undergo its state and carry out for each web page within the outcome a substitute operation on its state.
Now we now have this new API for retaining distant server information and shopper state in sync. So when will we execute it? There are two choices how one can execute it: manually or robotically.


Manually: Should you select to execute it manually, you would want to present your customers a button subsequent to every folder which provides them the choice to refresh the folder’s content material. That is a great way to present the person extra management, nevertheless, feels in our fashionable net world a bit outdated.


Robotically: Since we should not have net sockets, we will use the API for lengthy polling. Concerning the interval it is as much as you what number of occasions you wish to set off the refetch behind the scenes to your customers.


In spite of everything, if this desk with hundreds of things needs to be utilized in collaboration by a number of customers, an internet socket connection could be the very best case state of affairs. Should you can not set up this, your finest wager could be utilizing lengthy polling prefer it’s proposed with this API.
React Tree Desk: CRUD
Thus far, we now have solely fetched chunks of paginated and nested information for our tree desk. These have been solely learn operations and with none write operations you would not want the sync API from the earlier part within the first place. Nonetheless, most information tables include write operations too.
Conserving it quick, each write CRUD operation (Create, Replace, Delete) wants a standalone API endpoint. All of those operations would have an effect on the customers information desk (and different customers — if they’re working with the information desk).
There are two methods of dealing with it for the person performing the write operation: carry out a pressured refetch of all (or particular) pages from the server-side that are affected by the write operation or carry out an optimistic UI client-side modification of the state (e.g. delete operation results in eradicating a node from nodes).
Each methods have their drawbacks, so let me clarify them within the case of making a brand new node.
Optimistic UI
If we replace the UI optimistically, we have to take into account that we now have a protracted polling replace operating within the background which overrides the information desk periodically. There are a number of points that are partly attributable to this race situation:


Placement Concern: The optimistic UI operation inserts the brand new node initially or finish of our nodes checklist. However that is not in sync with the implementation particulars of the backend (e.g. which inserts the node sorted by its identify into the opposite nodes). When the lengthy polling refetch executes ultimately, the optimistically inserted node will bounce to a unique place.


Fragmentation Concern: The optimistic UI operation inserts the brand new node, however the lengthy polling refetch — which refetches solely a subset (web page) of the whole checklist — doesn’t embody this new node, as a result of it is not a part of this specific subset. Thus the optimistically inserted node would possibly simply disappear once more for the person after the lengthy polling refetch executes.


Timing Concern: Generally it might occur that the lengthy polling request is executed proper after the write operation. Now, if the lengthy polling request is resolved first, it’ll substitute the client-side state with its information which incorporates the brand new node. Nonetheless, as soon as the write operation resolves, the optimistic UI will insert the node a second time.


All these consistency issues could possibly be mitigated in some way, however in our case we realized that this method — regardless that it ought to enhance the UX — comes with a number of prices. Which leads us to the pressured refetch.
Pressured Refetch
A pressured refetch would occur for each write operation and the nodes that are affected by it. So if I create a node within the nodes property of a node with a particular id, I might use the brand new synchronization API to refetch this node’s content material. This comes with fewer (and extra unlikely) points:


Fragmentation Concern: Much like the optimistic UI, the refetch doesn’t want to incorporate the brand new node, as a result of the checklist is fragmented into paginated lists (pages) and there’s no assure that the brand new node is a part of the already fetched pages. Thus the person creates a brand new node however doesn’t see it.


Timing Concern: Extra unlikely to occur is the timing problem from the optimistic UI try, however there’s a probability that it might occur. If there’s a race situation between lengthy polling (a number of information) and compelled refetch (little information), it might occur that the lengthy polling resolves after the pressured fetch and due to this fact doesn’t embody but the brand new node.


As you possibly can see, with solely utilizing a pressured refetch we find yourself with comparable points, regardless that they aren’t as impactful as if we’d use solely an optimistic UI. Nonetheless, the optimistic UI presents nonetheless higher UX. So which one to make use of?
Hybrid
What we ended up with is a hybrid method of utilizing optimistic UI and compelled refetch on a case by case foundation. For instance, after we create a node, we’re utilizing an optimistic UI after which a pressured refetch. The previous offers the person an important UX whereas the latter makes certain that there aren’t any inconsistency points. In distinction, after we replace (e.g. a node’s identify) or delete a node, we’re solely performing the optimistic UI technique. Once we transfer nodes with our transfer operation, we carry out only a pressured refetch.
We additionally realized that we now have to contemplate two issues:


Queue: All API operations for the desk are pushed right into a queue and are carried out sequentially. This mitigates the danger of the sooner talked about race circumstances (Timing Concern). For instance, if there’s a interval synchronization refetch, then a CRUD operation, after which one other synchronization refetch, they’re all carried out one after one other.


Order: With out taking a sorting function into consideration for the sake of retaining it easy, newly created nodes will at all times be positioned on the high of the checklist by the database (order by dateCreated). This fashion, we mitigate the danger of Placement Points and Fragmentation Points, as a result of if we insert a node and place if with an optimistic UI on the high of the checklist, the pressured refetch will place it there as nicely.



A lot of work goes right into a desk with hundreds of tree structured nodes. The preliminary fetching may be cut up up into smaller chunks by utilizing paginated and nested lists. This solely covers the learn operations although. If a person writes to the desk, the implementation must deal with the person (and different customers). In a finest case state of affairs, we’d use net sockets for this sort of real-time updates. Nonetheless, if that is not obtainable, you possibly can obtain your targets with lengthy polling too.
A desk comes with greater than learn and write operations although. Within the following bonus part, I wish to undergo our implementation of a Search and Filter function and the way we designed the API for it. This could present how a lot work goes into element when creating such an API and element by simply going by means of one superior function.
React Tree Desk: Search and Filter
A server-side search function could possibly be fairly straight ahead. Within the request which fetches the checklist, one might embody a search argument which is used on the server-side to return the searched checklist. Nonetheless, with our model of paginated and nested fetches, it will get extra sophisticated. However let’s discover this drawback step-by-step.
We figured it could be finest to increase our earlier API for fetching pages:
const request = {
  path: '/nodes',
  physique:  null ,
};
Now, with this elective extension of the request in place, we will carry out the identical requests as earlier than however with a situation. With out trying on the request’s physique at a full extent (no restrict, offset, id), an instance request could possibly be the next:
const request = {
  path: '/nodes',
  physique: {
    search: 'picture',
  },
};
The outcome for this search could be not a flat checklist this time, however a hierarchical tree construction:
const outcome = [
  { id: '0', name: 'profile image.png' },
  {
    id: '52',
    name: 'High Resolution',
    nodes: [
      {
        id: '53',
        name: 'Favorites',
        nodes: [
          { id: '4', name: 'image4.png' },
          { id: '5', name: 'image5.png' },
        ]
      },
      { id: '1', identify: 'profile image1.png' },
      { id: '2', identify: 'profile image2.png' },
      { id: '3', identify: 'profile image3.png' },
    ]
  },
];
Within the case of search, the father or mother nodes of the matching nodes are returned too. That is as a result of we do not wish to present the search outcome as a flat checklist, however nonetheless of their hierarchical context. What could be returned if we’d seek for “Favorites” as a substitute?
const outcome = [
  {
    id: '52',
    name: 'High Resolution',
    nodes: [
      { id: '53', name: 'Favorites', nodes: [] },
    ]
  },
];
The matched node is retrieved inside its context once more, however solely with its higher (father or mother nodes, e.g. “Excessive Decision”) and never with its decrease (baby nodes) context. That is how we determined it for our implementation, nevertheless, it is also legitimate to return baby nodes too; with a purpose to give the person the total higher and decrease context boundaries.
UI sensible it helps to spotlight the matching nodes within the Desk (instance), as a result of when they’re proven in a hierarchy, it isn’t at all times simple for the person to identify the matching nodes.
Paginated and Nested Search
The earlier examples have proven how we will return searched nodes of their hierarchy from the backend. Nonetheless, we did not combine this into our paginated/nested lists but. Within the state of affairs of getting hundreds of matching search outcomes, we nonetheless wish to hold the chunking function from earlier than.
Let’s have a look at how this seems if we hold the unique arguments (restrict, offset, id) for the request and alter the search time period to one thing completely different:
const request = {
  path: '/nodes',
  physique: {
    id: null,
    offset: 0,
    restrict: 1,
    search: 'profile',
  },
};
The outcome could be a nested paginated checklist:
const outcome = {
  nodes: [
    { id: '0', name: 'profile image.png' },
  ],
  pageInfo: {
    complete: 2,
    nextOffset: 1
  },
};
If there could be no search, the top-level checklist would have a complete of three. Now discover how the overall quantity of things for this search result’s 2 although. For the reason that backend can iterate over all top-level nodes, it is aware of that solely two of the nodes are both themselves matching nodes or have matching baby nodes.
Observe: I can’t go into the efficiency hits that the backend has to endure resulting from this new search function. Primarily the backend must iterate by means of the entire tree to find out the matching nodes. This places stress on the database and on the backend itself.
Now we all know that there’s extra matching information for the search question, as a result of we now have a nextOffset as outcome. Let’s fetch it with one other request:
const request = {
  path: '/nodes',
  physique: {
    id: null,
    offset: 1,
    restrict: 1,
    search: 'profile',
  },
};
This time the result’s a hierarchical match, as a result of not the top-level node matches, however its baby nodes:
const outcome = [
  nodes: [
    {
      id: '52',
      name: 'High Resolution',
      nodes: [
        { id: '1', name: 'profile image1.png' },
        { id: '2', name: 'profile image2.png' },
      ],
      pageInfo: {
        complete: 3,
        nextOffset: 2
      },
    },
  ],
  pageInfo: {
    complete: 2,
    nextOffset: null
  },
];
It is vital to notice that the node with the id of 1 is returned too, regardless that it isn’t within the offset-limit-threshold. For nested nodes this can be a mandatory conduct, as a result of in any other case we’d by no means retrieve this node both with an offset of 0 or offset of 1.
Ultimately, the frontend provides each outcomes into one once more, by utilizing the newest pageInfo objects and concatenating lists:
const outcome = [
  nodes: [
    { id: '0', name: 'profile image.png' },
    {
      id: '52',
      name: 'High Resolution',
      nodes: [
        { id: '1', name: 'profile image1.png' },
        { id: '2', name: 'profile image2.png' },
      ],
      pageInfo: {
        complete: 3,
        nextOffset: 2
      },
    },
  ],
  pageInfo: {
    complete: 2,
    nextOffset: null
  },
];
When performing a paginated/nested search, the person will get introduced a hierarchical outcome. That is completely different from what we had earlier than when utilizing solely paginated and nested requests. Nonetheless, the UI stays the identical: Inside the displayed hierarchical tree view, the person can set off extra paginated and nested fetches.

I need to say that this challenge was difficult, however I realized a number of issues alongside the way in which. It is not as straight ahead as one would possibly suppose to create an API for an asynchronous tree desk which must deal with hundreds of entries. If it could be solely learn operations, it could be okay by simply utilizing paginated and nested requests, nevertheless, the write operations make this endeavour more difficult, as a result of one has to maintain the information in sync between frontend and backend.
As well as, a desk does not come solely with learn and write operations, but in addition with options like looking out, filtering, focusing right into a folder, sorting and so forth. Placing all of these items collectively, in hindsight it was an important determination to first work on the API necessities after which on the backend/frontend implementation.
Ultimately, with the API design necessities in place to attach frontend and backend, a brand new React Desk Library was born to implement all of it on the client-side. One of many essential motivations behind it was utilizing server-side operations as first-class residents; which allow one to implement options like type, search, pagination not solely client-side, however with a server which presents these options as API.