This article describes how to use Node file uploads to power direct upload functionality to a Backblaze B2 bucket. No need for files to touch your server! Just direct Node upload file to S3.This is important for deploying to a PaaS such as Heroku due to the ephemeral file system.
B2 is an S3-compatible object storage provider, which means that the approach described in this article can largely be applied to other S3-compatible services too, such as Amazon S3, Linode Object Storage, and DigitalOcean Spaces.
Object storage often considered a good solution for storing user-uploaded content as it is cost-effective, scalable, robust, and can be used to store a wide range of file types.
In particular, Backblaze’s pricing model makes B2 stand out amongst its competitors.
Some apps accept file uploads through a standard POST request as part of a form submission. In these cases, the server-side code must receive the file, process it, and then proceed to persist it somehow. This may be by writing the contents to a local filesystem, or passing on the object to some other storage mechanism.
Whilst this approach may offer some advantages - such as the ability to run on-the-fly processing, image resizing, watermarking, etc. - these can be largely outweighed by the fact that the server must consume more resources (both CPU and RAM), which means that the approach becomes less scalable.
Furthermore, not all application deployments allow for local filesystem storage, and processing large files server-side may cause requests to timeout. These factors are particularly relevant to apps deployed on Heroku or on a serverless architecture.
As such, this article covers the concept of direct uploading to B2. This means that the browser does the uploading directly to your B2 bucket without files having to go through your server.
The rest of this article covers the process of building in direct-to-B2 uploading in a Node application. Although it’ll focus on a barebones Node application, the process can be applied to Node running in any environment - including on Heroku or in a serverless deployment.
The article uses the Express framework for easing the generation of a simple webserver. However, the Node-specific code can easily be adapted into any other framework.
Firstly, ensure your system’s Node and NPM installations are up-to-date.
Then, in your terminal, create a new directory for your Express project and change into it:
.png)
Use the `express-generator` tool to scaffold your app:
.2.png)
A number of files and directories will be created. You can now install the required dependencies:
.png)
Finally, you can test your app by running it:
.png)
Navigate to localhost:3000 in your web browser and you should see a page that looks like the following.
You can now add some code to form a basic interface to enable a user to select a file to upload.
The Express generator created some "view" files for you, which can be found in the `views/` directory. The default setup uses the Jade templating engine to render HTML, so you'll notice files with the `.jade` extension. If your setup involves writing HTML directly, or uses another templating engine, then the following should be translatable for your particular case.
Rather than creating a whole new view, you can just modify the existing primary index route. To do so, open up `views/index.jade` and add the following to the bottom of the file (inside the `content` block):
.png)
This code renders a basic file input, and the script then attaches an `onchange` listener for the input.
The listener function accesses the first file selected in the input and then queries a `/signed-request` route on the Express application. This route hasn't been implemented yet, but you'll do so in Step 3. Once implemented, the route will generate and return a signed `PUT` request that can then be used to upload the file directly to B2.
That's it for the client-side code. Next you'll write the logic for generating the signed request.
In order for the browser to upload directly to B2, it needs to use signed requests. For security purposes, these are generated server-side, as otherwise the client would have access to your B2 secret keys.
To generate the signed request you can make use of AWS' SDK for working with S3-compatible object storage providers. Add the required dependency to your project:
.png)
Next, open up the file `routes/index.js` in your project to add a new route definition. This route will accept information about the file to be uploaded and return a signed request that will enable the browser to complete the upload. Near the top of the file (next to the other imports) import the package we installed earlier:
.png)
Then create the route at the bottom of the same file:
.png)
Broadly, the code above performs the following:
A number of environment variables have been referenced (such as `B2_ENDPOINT_URL` and `B2_BUCKET`). These will be configured during Step 7.
This concludes the changes required to the server-side code. Next, you'll cover creating and configuring your bucket on B2.
Firstly, login to your Backblaze B2 account console by visiting backblaze.com. If you do not yet have a Backblaze account, then you'll need to create an account here.
In the "Buckets" tab of the console, select "Create a Bucket".
In the dialog that opens, enter a name for your bucket and enable any of the other settings you may need. Then click "Create a Bucket" to complete the process. Please note that bucket names must be unique. Make a note of your bucket name, as you'll need this later.
Once the bucket is created, it will be added to the list of buckets on your account. At this stage, make a note of the "Endpoint" displayed for the bucket, as you'll need this later.
You'll need some credentials in order to give your app the access needed to write objects to your bucket.
In the Backblaze console, navigate to the "App Keys" tab. Here, we will create two sets of keys: one that can be used to manage your B2 account, and another that is specific to this particular app.
Firstly, create the "management" key. To do so, click the "Add a New Application Key" button on this page.
On the dialog that pops up, enter a name for your key and ensure the key has read/write access to all buckets.
Once created, the console will display the `keyID` and `applicationKey` for your new "management" key. Make a note of these as you'll need them later and they only get displayed once.
Next, create the key for your application (we'll call this the "application" key). Go through the "Add a New Application Key" process again, but this time restrict the key's access to just the bucket you created earlier.
Make a note of the `keyId` and `applicationKey` for this "application" key as above.
Cross-origin resource sharing (CORS) needs to be configured on your bucket in order to allow the browser to send `PUT` requests directly to it. This is because web-browsers perform pre-flight checks against target cross-domain servers to ensure the relevant security provisions are in place.
The CORS configuration you need on this bucket needs to be managed using the B2 API, and for this we recommend making use of the official `b2` tool.
To begin, download the B2 tool for your system from the Backblaze website. You can also use Homebrew on Mac:brew install b2-tools Rename the downloaded file to b2 and move it to a sensible place. You'll need to mark it as executable (if you use MacOS or Linux) by running chmod +x /path/to/b2 on the downloaded file. For the rest of this article we'll assume that the b2 tool is available in your path and can be invoked simply using the b2 command.
Login to your B2 account by running the following:
.png)
You will be prompted for a key ID and an application key. For these use the details from the "management" key you created in Step 5.
Once you've authenticated the tool you can use it to set the CORS settings for the bucket. To do so, issue the following command:
.png)
You'll need to change `<bucket name>` to the name of the bucket you created in Step 4. You may also wish to change the other attributes of the CORS rules, such as the allowed origins or operations, depending on your required level of security and the scope of usage for this bucket.
The final step in the setup involves configuring the environment variables accessed by the Node app in Step 3. At this stage, you should have noted:
Create a new file named `envfile` in the root of your Node Express project and add these contents, replacing the values with those noted above:
.png)
If you manage your environment variables differently in your application, then instead simply drop them into your existing setup. Either way, we recommend not including your environment variables in your source control system since each of your environments (development, production, etc.) will likely have different variables.
You're now ready to run the complete application!
If your terminal is still running your application, then exit it first by pressing ctrl-C.
Next, load the environment variables you created earlier:
.png)
Finally, you can run the application again:
Navigate to localhost:3000 again to view the new application version, which should look something like the following image.
Use the file input to select a file, and then browse the bucket contents on the B2 console and you should see the file listed.
If the file doesn't upload, then we recommend checking your browser console or the terminal window running your Node app. One or both of these may print warnings or errors that can help you find the issue.
In this article we have discussed an approach to achieving basic direct-to-B2 uploads with direct Node file uploads. No need for files to touch your server! Just direct Node upload file to S3.
Whilst the finished product does not feature additional useful UX enhancements (such as loading indicators or upload previews and confirmations), the technical process for implementing an upload system should provide enough guidance for including such a mechanism in your own applications.
While this is an effective solution to Node file uploads on Heroku, it does not provide a visual drag-and-drop interface, client side image resizing, or automatic backups. This is why we created Simple File Upload, a widget that allows you to integrate file uploads into your website in minutes and handles all these things! No heavy lifting is required. Try it free for 7 days!
