Hello everybody! 👋 I’ve been engaged on a sports activities analytics startup and the entire front-end is made utilizing React. I haven’t needed to work on such an enormous React mission earlier than this and so it introduced alongside some distinctive challenges. Up to now, search engine marketing hasn’t been an enormous challenge for the net tasks I’ve labored on. I might all the time simply add related knowledge to the meta tags, outline the navigation utilizing semantic HTML tags and Google would present the web site appropriately on its consequence pages. Nonetheless, this time, the entire web site was being generated utilizing client-side Javascript. On preliminary load, there may be near nothing on the internet web page and the navigation turns into obtainable solely when JavaScript is executed.
Despite the fact that Google crawler can execute Javascript, it’s a higher search engine marketing follow to serve it a rendered utility as an alternative of counting on it to render the appliance by itself. Furthermore, if the rendering takes longer than some set time, the crawler would merely transfer on. On this article, I’ll present you methods to get the prerender service up and operating in your server and serve it through NGINX to completely different bots and increase your search engine marketing.
Searching for an answer
I searched round and located a number of options to this drawback. Sure node scripts could possibly be run as a post-processing step and they might output a rendered React app. I attempted utilizing them however none labored flawlessly with out requiring me to tweak the app code considerably. It’s simpler to include such instruments at the beginning of a mission. Including them later simply results in plenty of ache. I used to be on the lookout for a plug-and-play resolution so I saved trying.
I quickly got here throughout prerender. This service acts as a middleware between your utility and the end-user. It makes use of Headless Chrome to render HTML. If the end-user is a human, the server will return HTML + JS and the React code might be rendered on the client-side. Nonetheless, if the end-user is a bot then the server will ship the HTML + JS to the prerender service and return the rendered static HTML web page to the bot. There are a number of related companies. I might have simply as simply used renderton however prerender simply appeared extra widespread so I went forward with that.
Observe: It goes with out saying that this may not be essentially the most environment friendly resolution but it surely works for my use case. Your mileage might differ so attempt at your personal danger 😄
That is how a typical circulation with the prerender service appears to be like like:
The prerender service will cache the static HTML for some time earlier than requesting a brand new copy from the server. This methodology didn’t require me to alter something on my React app and solely required me to make some changes to the NGINX server configuration.
That is how my utility was being served earlier than prerender got here into the combo:
After including prerender, the circulation regarded one thing like this:
Organising prerender service
There are two methods you may make use of the prerender service. You’ll be able to both make use of the paid hosted service (accommodates a free plan as nicely) or you may run the open supply service in your server and interface with that. I made a decision to go for the latter. Step one was to put in prerender on the server. It was pretty simple to take action utilizing npm:
$ npm set up prerender
The subsequent step was to create a brand new node script that ran the service. These 3 traces are sufficient to get the service up and operating:
const prerender = require('prerender');
const server = prerender();
server.begin();
Save this code to a server.js file and run it utilizing node:
$ node server.js
At this level you may go forward and take a look at whether or not the prerender service is working appropriately or not by opening http://localhost:3000/render/?url=https://google.com/. This could show the Google homepage. If the pictures don’t present up appropriately, don’t fear. The problem might be fastened after we serve the prerender service through NGINX.
Run prerender on system begin
The subsequent step is to run prerender on system begin and ensure it retains operating in case of crash or system restart. We are going to make this occur by making a systemd service file. I’m assuming that you simply saved the server.js file in your house folder. My house folder is yasoob however yours could be completely different so be sure you edit the WorkingDirectory path. Create a prerender.service file in /and so forth/systemd/system folder with the next contents:
[Unit]
Description=Prerender server for bot crawling
After=community.goal
[Service]
Consumer=yasoob
WorkingDirectory=/house/yasoob/
ExecStart=/usr/bin/node server.js
Restart=all the time
[Install]
WantedBy=multi-user.goal
On this file, we’re telling systemd to start out server.js when the community is up and operating. The service would launch underneath the consumer yasoob and the command it must run is /usr/bin/node server.js. The script would additionally robotically restart in case of a crash or system restart.
After saving this file, let’s be sure systemd can acknowledge it:
$ sudo service prerender standing
● prerender.service - Prerender server for bot crawling
Loaded: loaded (/and so forth/systemd/system/prerender.service; disabled; vendor preset: enabled)
Energetic: inactive (lifeless)
Excellent! Now let’s begin the service after which test the standing:
$ sudo service prerender begin
$ sudo service prerender standing
● prerender.service - Prerender server for bot crawling
Loaded: loaded (/and so forth/systemd/system/prerender.service; disabled; vendor preset: enabled)
Energetic: energetic (operating) since Thu 2021-01-07 19:59:46 UTC; 2s in the past
Fundamental PID: 589168 (node)
Duties: 7 (restrict: 1137)
Reminiscence: 41.8M
CGroup: /system.slice/prerender.service
└─589168 /usr/bin/node server.js
Jan 07 19:59:46 systemd[1]: Began Prerender server for bot crawling.
Integrating prerender with NGINX
Now that our prerender service is operating, we are able to go forward and combine it with NGINX. What we need to do is that the traditional consumer ought to be despatched the traditional HTML + JS response however a bot ought to be despatched a response by the prerender service.
The unique NGINX configuration file for my React app regarded like this:
server {
server_name instance.com;
root /house/yasoob/instance/construct;
index index.html;
location / {
try_files $uri /index.html;
add_header Cache-Management "no-cache";
}
location /static {
expires 1y;
add_header Cache-Management "public";
}
location /api {
embrace proxy_params;
proxy_pass http://localhost:5000;
}
}
It is a pretty generic configuration file. We add some cache headers to sure path responses and cross the /api route visitors to the gunicorn server operating on port 5000. Now we simply have to guarantee that all requests made by a bot are responded to by the prerender service that’s operating on port 3000. The template file for these modifications is conveniently offered by the prerender of us. I took the identical file and tweaked it a bit to make it work for my setup. The key factor I modified within the template was to edit the prerender service URL and take away the proxy header half. As I’m utilizing a self-hosted service, I changed service.prerender.io with 127.0.0.1:3000 and since that is our service, we don’t have to cross any authentication headers.
The ensuing NGINX configuration file appears to be like like this:
server {
server_name shotquality.com;
root /house/yasoob/shotqenterprise/REACT/construct;
index index.html;
location / {
try_files $uri @prerender;
add_header Cache-Management "no-cache";
}
location /static {
expires 1y;
add_header Cache-Management "public";
}
location /api {
embrace proxy_params;
proxy_pass http://localhost:5000;
}
location @prerender {
set $prerender 0;
if ($http_user_agent ~* "googlebot|bingbot|yandex|baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora hyperlink preview|showyoubot|outbrain|pinterest/0.|pinterestbot|slackbot|vkShare|W3C_Validator|whatsapp") {
set $prerender 1;
}
if ($args ~ "_escaped_fragment_") {
set $prerender 1;
}
if ($http_user_agent ~ "Prerender") {
set $prerender 0;
}
if ($uri ~* ".(js|css|xml|much less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff|svg|eot)") {
set $prerender 0;
}
if ($prerender = 1) {
set $prerender "127.0.0.1:3000";
rewrite .* /$scheme://$host$request_uri? break;
proxy_pass http://$prerender;
}
if ($prerender = 0) {
rewrite .* /index.html break;
}
}
}
I eliminated the SSL assist and redirection from this configuration file for the sake of simplicity.
Testing NGINX configuration
With the intention to take a look at whether or not our NGINX config modifications didn’t break something, we are able to run:
$ sudo nginx -t
If every part appears right, we are able to restart NGINX:
$ sudo service nginx restart
To check whether or not our service is working the way in which it’s alleged to, we are able to run the next CURL command:
$ curl -A googlebot https://instance.com
Exchange instance.com together with your React-based app URL and see if the output of this command is completely different from if you happen to run curl with out -A googlebot.
In case you have reached this step then likelihood is that your prerender service is working wonderful. In case there are errors, please write about them within the feedback beneath and I’ll attempt to assist.
You all have an exquisite day and a satisfying new 12 months 🙂
