Welcome to tutorial no. 4 in our Resumable file uploader collection.
Within the earlier tutorials, we coded our resumable tus server utilizing Go. On this tutorial we’ll use curl and dd instructions to check the tus server.
Testing
We’ve the resumable file add tus server prepared however now we have not examined it but. We’d like a tus shopper to check the tus server. We are going to create the Go shopper within the upcoming tutorials. For now, we’ll use the curl command to check the tus server.
Let’s run the server first. Run the next instructions within the terminal to fetch the code from github after which to run it.
go get github.com/golangbot/tusserver
go set up github.com/golangbot/tusserver
tusserver
After working the above instructions, the server will likely be up and working.
2019/03/30 18:01:41 Connection established efficiently
2019/03/30 18:01:41 TUS Server began
2019/03/30 18:01:41 Listing created efficiently
2019/03/30 18:01:41 desk create efficiently
We’d like a file to check the tus server. I’ve made a collage video of my pet and it’s accessible at https://www.dropbox.com/s/evchz5hsuvtrvuu/mypet.mov?dl=0. Please be happy to make use of it :). I’ve downloaded the video to my ~/Downloads listing.
Let’s ship a publish request and create a brand new file. We have to specify the Add-Size of all the file within the publish request. That is nothing however the measurement of the file. We will use the ls command to seek out the scale of the file
ls -al ~/Downloads/mypet.mov
The above command returns the next output.
-rw-rw-r-- 1 naveen naveen 11743398 Mar 31 11:11 /dwelling/naveen/Downloads/mypet.mov
11743398 is the scale of the file. Now that we all know the Add-Size, let’s create the file by sending a publish request.
curl --request POST localhost:8080/recordsdata --header "Add-Size: 11743398" -i
The above command creates the file. The -i argument ultimately is used to show the response headers. The above command will return the next outcome.
HTTP/1.1 201 Created
Location: localhost:8080/recordsdata/1
Date: Solar, 31 Mar 2019 07:47:33 GMT
Content material-Size: 0
The file creation has been created efficiently.
Now comes the tough half. How will we take a look at the tus server by simulating a community disconnection? If we ship a patch request to the file URL utilizing curl, the request will likely be accomplished instantly because the server is working regionally and we will be unable to check whether or not the server is ready to deal with resumable uploads.
That is the place the --limit-rate argument of curl helps us. This argument can be utilized to fee restrict the patch file request.
curl --request PATCH --data-binary "@/dwelling/naveen/Downloads/mypet.mov" localhost:8080/recordsdata/1 --header "Add-Offset: 0" --header "Count on:" -i --limit-rate 200K
Within the curl request above, we’re sending a patch request to the file at location localhost:8080/recordsdata/1 and Add-Offset: 0 and we’re rate-limiting the request to 200KB/Sec. The contents of mypet.mov is added to the request physique. The --header "Count on:" header is required to forestall curl from sending Count on: 100-continue header. Please learn https://gms.tf/when-curl-sends-100-continue.html to know why that is wanted.
After issuing the above patch request, the file will likely be transferred at 200KB/S. Let the request run for a number of seconds, say 10 seconds. After roughly 10 seconds, please cease the request by urgent ctrl + c. Now now we have terminated the patch request within the center. The server ought to have saved the bytes transferred until now. Let’s test whether or not it has achieved it.
Transfer to the server logs and it is possible for you to to see the next within the log,
2019/03/31 13:36:00 Obtained file partially sudden EOF
2019/03/31 13:36:00 Dimension of acquired file 1589248
2019/03/31 13:36:00 variety of bytes written 1589248
The scale of the acquired file could also be completely different for you. The above is my output.
hmm seems prefer it has saved the bytes acquired until now. However how will we confirm it? Properly, let’s test the scale of the uploaded file.
ls -al ~/fileserver/1
Operating the above command outputs
-rw-r--r-- 1 naveen naveen 1589248 Mar 31 13:36 /dwelling/naveen/fileserver/1
The scale of the file matches the server output. Now we could be 100% positive that the server has saved the bytes it has acquired. In the event you attempt to play the video now, it will not play because the file continues to be not fully uploaded but.
The subsequent step is to proceed the patch request from the place it stopped. We first wanted to know the Add-Offset in order that we are able to concern the subsequent patch request. That is the place the pinnacle request turns out to be useful.
curl --head localhost:8080/recordsdata/1 -i
The above curl command will return the Add-Offset
HTTP/1.1 200 OK
Add-Offset: 1589248
Date: Solar, 31 Mar 2019 08:17:28 GMT
Be aware that the offset matches the server logs and the file measurement.
Now we have to ship a PATCH request with the above add offset. Yet one more concern is we have to ship the file knowledge(bytes of the file) from this offset solely, not all the file.
That is the place the dd command helps us.
dd if=/dwelling/naveen/Downloads/mypet.mov skip=1589248 bs=1 | curl --request PATCH --data-binary @- localhost:8080/recordsdata/1 --header "Add-Offset: 1589248" --header "Count on:" -i
Within the above command, we use if to specify the enter file and skip is used to skip 1589248 bytes. 1589248 is our Add-Offset. bs specifies that we learn one byte at a time. We pipe the output of dd to curl command. After working the above command, we’ll get the output
HTTP/1.1 204 No Content material
Add-Offset: 11743398
Date: Solar, 31 Mar 2019 08:25:10 GMT
A 204 No Content material signifies that the patch was profitable. To know whether or not the file add is full, we are able to once more concern a head request and the add offset ought to match the add size(measurement) of the file.
curl --head localhost:8080/recordsdata/1 -i
The above command will output
HTTP/1.1 200 OK
Add-Offset: 11743398
Date: Solar, 31 Mar 2019 08:30:54 GMT
The add offset matches the add size and we’re positive that the file has been uploaded fully. Now when you attempt to concern a patch request once more, the server will complain saying that the add is already full.
Let’s once more test the file measurement now.
ls -al ~/fileserver/1
Operating the above command outputs
-rw-r--r-- 1 naveen naveen 11743398 Mar 31 13:55 /dwelling/naveen/fileserver/1
The file measurement within the output matches the add size and this confirms that the file has been uploaded fully. You possibly can go forward and play the video and it’ll play now 🙂
Our resumable tus server is prepared 🙂
Enhancements
Though the file uploader works, this code must refactored additional. It at the moment would not deal with concurrency. As an example, we would hit a race situation when a number of purchasers ship a concurrent patch request for a similar file and on the similar offset.
This code additionally would not deal with DB transactions effectively. There’s a likelihood {that a} POST request to create a file may find yourself creating the file within the DB however not creating the precise file within the file system. For instance, what occurs when there isn’t a house left within the file system.
All of the code is at the moment current in the primary package deal in a single file and this method isn’t extensible. The code needs to be refactored into usable packages. One method to construction code in Go is utilizing Area Pushed Design.
We additionally do not have a tus shopper prepared but :).
All these will likely be addressed within the upcoming tutorials. I hope you loved studying. Please depart your feedback. Have day.
Like my tutorials? Please present your help by donating. Your donations will assist me create extra superior tutorials.
