Friday, March 17, 2023
HomeGolangTesting the resumable file add server utilizing curl and dd

Testing the resumable file add server utilizing curl and dd

Welcome to tutorial no. 4 in our Resumable file uploader collection.

Within the earlier tutorials, we coded our resumable tus server utilizing Go. On this tutorial we’ll use curl and dd instructions to check the tus server.


We’ve the resumable file add tus server prepared however now we have not examined it but. We’d like a tus shopper to check the tus server. We are going to create the Go shopper within the upcoming tutorials. For now, we’ll use the curl command to check the tus server.

Let’s run the server first. Run the next instructions within the terminal to fetch the code from github after which to run it.

go get  
go set up  

After working the above instructions, the server will likely be up and working.

2019/03/30 18:01:41 Connection established efficiently  
2019/03/30 18:01:41 TUS Server began  
2019/03/30 18:01:41 Listing created efficiently  
2019/03/30 18:01:41 desk create efficiently  

We’d like a file to check the tus server. I’ve made a collage video of my pet and it’s accessible at Please be happy to make use of it :). I’ve downloaded the video to my ~/Downloads listing.

Let’s ship a publish request and create a brand new file. We have to specify the Add-Size of all the file within the publish request. That is nothing however the measurement of the file. We will use the ls command to seek out the scale of the file

ls -al ~/Downloads/  

The above command returns the next output.

-rw-rw-r-- 1 naveen naveen 11743398 Mar 31 11:11 /dwelling/naveen/Downloads/

11743398 is the scale of the file. Now that we all know the Add-Size, let’s create the file by sending a publish request.

curl --request POST  localhost:8080/recordsdata --header "Add-Size: 11743398" -i  

The above command creates the file. The -i argument ultimately is used to show the response headers. The above command will return the next outcome.

HTTP/1.1 201 Created  
Location: localhost:8080/recordsdata/1  
Date: Solar, 31 Mar 2019 07:47:33 GMT  
Content material-Size: 0  

The file creation has been created efficiently.

Now comes the tough half. How will we take a look at the tus server by simulating a community disconnection? If we ship a patch request to the file URL utilizing curl, the request will likely be accomplished instantly because the server is working regionally and we will be unable to check whether or not the server is ready to deal with resumable uploads.

That is the place the --limit-rate argument of curl helps us. This argument can be utilized to fee restrict the patch file request.

curl --request PATCH --data-binary "@/dwelling/naveen/Downloads/" localhost:8080/recordsdata/1 --header "Add-Offset: 0" --header "Count on:" -i --limit-rate 200K  

Within the curl request above, we’re sending a patch request to the file at location localhost:8080/recordsdata/1 and Add-Offset: 0 and we’re rate-limiting the request to 200KB/Sec. The contents of is added to the request physique. The --header "Count on:" header is required to forestall curl from sending Count on: 100-continue header. Please learn to know why that is wanted.

After issuing the above patch request, the file will likely be transferred at 200KB/S. Let the request run for a number of seconds, say 10 seconds. After roughly 10 seconds, please cease the request by urgent ctrl + c. Now now we have terminated the patch request within the center. The server ought to have saved the bytes transferred until now. Let’s test whether or not it has achieved it.

Transfer to the server logs and it is possible for you to to see the next within the log,

2019/03/31 13:36:00 Obtained file partially sudden EOF  
2019/03/31 13:36:00 Dimension of acquired file  1589248  
2019/03/31 13:36:00 variety of bytes written  1589248  

The scale of the acquired file could also be completely different for you. The above is my output.

hmm seems prefer it has saved the bytes acquired until now. However how will we confirm it? Properly, let’s test the scale of the uploaded file.

ls -al ~/fileserver/1  

Operating the above command outputs

-rw-r--r-- 1 naveen naveen 1589248 Mar 31 13:36 /dwelling/naveen/fileserver/1

The scale of the file matches the server output. Now we could be 100% positive that the server has saved the bytes it has acquired. In the event you attempt to play the video now, it will not play because the file continues to be not fully uploaded but.

The subsequent step is to proceed the patch request from the place it stopped. We first wanted to know the Add-Offset in order that we are able to concern the subsequent patch request. That is the place the pinnacle request turns out to be useful.

curl --head localhost:8080/recordsdata/1 -i  

The above curl command will return the Add-Offset

HTTP/1.1 200 OK  
Add-Offset: 1589248  
Date: Solar, 31 Mar 2019 08:17:28 GMT  

Be aware that the offset matches the server logs and the file measurement.

Now we have to ship a PATCH request with the above add offset. Yet one more concern is we have to ship the file knowledge(bytes of the file) from this offset solely, not all the file.

That is the place the dd command helps us.

dd if=/dwelling/naveen/Downloads/ skip=1589248 bs=1 | curl --request PATCH --data-binary @- localhost:8080/recordsdata/1 --header "Add-Offset: 1589248" --header "Count on:" -i  

Within the above command, we use if to specify the enter file and skip is used to skip 1589248 bytes. 1589248 is our Add-Offset. bs specifies that we learn one byte at a time. We pipe the output of dd to curl command. After working the above command, we’ll get the output

HTTP/1.1 204 No Content material  
Add-Offset: 11743398  
Date: Solar, 31 Mar 2019 08:25:10 GMT  

A 204 No Content material signifies that the patch was profitable. To know whether or not the file add is full, we are able to once more concern a head request and the add offset ought to match the add size(measurement) of the file.

curl --head localhost:8080/recordsdata/1 -i  

The above command will output

HTTP/1.1 200 OK  
Add-Offset: 11743398  
Date: Solar, 31 Mar 2019 08:30:54 GMT  

The add offset matches the add size and we’re positive that the file has been uploaded fully. Now when you attempt to concern a patch request once more, the server will complain saying that the add is already full.

Let’s once more test the file measurement now.

ls -al ~/fileserver/1  

Operating the above command outputs

-rw-r--r-- 1 naveen naveen 11743398 Mar 31 13:55 /dwelling/naveen/fileserver/1

The file measurement within the output matches the add size and this confirms that the file has been uploaded fully. You possibly can go forward and play the video and it’ll play now 🙂

Our resumable tus server is prepared 🙂


Though the file uploader works, this code must refactored additional. It at the moment would not deal with concurrency. As an example, we would hit a race situation when a number of purchasers ship a concurrent patch request for a similar file and on the similar offset.

This code additionally would not deal with DB transactions effectively. There’s a likelihood {that a} POST request to create a file may find yourself creating the file within the DB however not creating the precise file within the file system. For instance, what occurs when there isn’t a house left within the file system.

All of the code is at the moment current in the primary package deal in a single file and this method isn’t extensible. The code needs to be refactored into usable packages. One method to construction code in Go is utilizing Area Pushed Design.

We additionally do not have a tus shopper prepared but :).

All these will likely be addressed within the upcoming tutorials. I hope you loved studying. Please depart your feedback. Have day.

Like my tutorials? Please present your help by donating. Your donations will assist me create extra superior tutorials.



Please enter your comment!
Please enter your name here

Most Popular

Recent Comments