Object Storage on TrueNAS

 

I wanted to cover the many ways TrueNAS can be used to create and manage object storage. TrueNAS includes Minio S3 object storage out of the box, and the two work very well together. TrueNAS is a scalable, easy way to run and manage ZFS, and Minio is the de-facto way to self host S3 objects. 

Using the standard S3 service in TrueNAS is the quickest way to get setup and running with S3 objects. However, user management, if one wanted to open up remote access to the Minio web console, is a bit more difficult. 

What the video proposes is a couple of ways to silo off and breakdown the S3 service either with a web front end, or simply by making use of jails to host multiple, separate instances of the S3 object storage. 

Here, we'll look more at the details for each option. 

Create a jail to run Minio:

Go to the Jails section on TrueNAS. Create a new jail, give it a name, and set the network as desired - the video showed with DHCP, but static IP addresses are available as well. 

Once created Start the jail. Enter the shell. In the shell type 

pkg update

Accept yes to install.

pkg search minio

pkg install minio-0420....(whatever the current version is provided)

Once installed, make a directory. Could be anywhere, I chose in the /mnt directory of the jail.

mkdir /mnt/miniodata

Start the Minio server with the following

minio server /mnt/miniodata

That will start the service, but if you close the console/terminal screen the service will also terminate. To make this a bit more robust we can run the service with cron. 

Type crontab -e

Insert (defaul is vi, type 'i' to insert, esc to stop inserting, :w to write, :q to exit)

@reboot minio server /mnt/miniodata --console-address=":9090"

Now the service for Minio will start on each boot of the jail, with the console dedicated to run on port 9090 of the jails unique IP address.

Create a web server to run a frontend in a jail: 

To build out a  LAMP (technically a FAMP - FreeBSD, Apache, MySQL, PHP) I followed this excellent guide on Digital Ocean. I did not need the database portion, so that was skipped, though I did install the php-mysqli packages just in case I wanted it in the future. 

Digital Ocean Guide 

Install steps in the jail terminal/shell. 

pkg update -y

pkg install apache24

sysrc apache24_enable="YES"

service apache24 start

Navigate to the jail IP address and check to see if "It works!" appears.

pkg install php81 php81-mysqli mod-php81 php81-simplexml

The php81-simplexml package doesn't come down with the meta package for php81, and this threw me for a couple of hours because it is needed for the AWS PHP S3 plugin we install. 

Initiate PHP with specific settings:

cp /usr/local/etc/php.ini-production /usr/local/etc/php.ini 

Initiate PHP in Apache (you can use either vi or ee as the text editor):

ee /usr/local/etc/apache24/modules.d/001_mod-php.conf

Copy this in the file:

<IfModule dir_module>
    DirectoryIndex index.php index.html
    <FilesMatch "\.php$">
        SetHandler application/x-httpd-php
    </FilesMatch>
    <FilesMatch "\.phps$">
        SetHandler application/x-httpd-php-source
    </FilesMatch>
</IfModule>

Install PHP Composer for the S3 support.

pkg search composer

pkg install php81-composer 

composer require --working-dir=/usr/local/www/apache24 aws/aws-sdk-php

Now all this was to support the POC file I have over on Github. If you want to use it copy it to the jail, or install git in the jail and run a git clone command. Put everything in the /work/src/ folder in the /usr/local/www/apache24/data directory in the jail. Also be sure to modify the S3 endpoint, credentials, bucket and host address to those of the jail's IP and relevant credentials. 

I hope this gives a better overview of using object storage on TrueNAS. It is a really flexible feature, can be plugged into a lot of other environments, or even self-hosted on the TrueNAS itself using jails.

PHP with S3

 

This blog and video runs through getting started with PHP and S3. The subsequent code is all open source in the GitHub page located at the below link.

https://github.com/JoeMrCoffee/yourS3objects

Previously we looked at MinIO using it's console and UI, as well as quick set up in docker. This is a slightly more advanced look at interacting with object storage from a programmatic standpoint. 

PHP has an open SDK to tie in with S3 object storage that is provided by AWS. The video, as well as the source code in GitHub installs the SDK as part of the Docker bring up and build. This is done in the Dockerfile, and is very similar to installing the MongoDB packs required to make a PHP connection to a MongoDB database. 

From the code all the connection information to the S3 object store is in the 'header.php' file. If one wanted to use this site, or just its source code, they could also adjust the endpoint to the appropriate values. The 'header.php' file gets included in all the subsequent pages using the PHP include command, so everything else will follow the values in the header.php file.

The video talks a bit about the usefulness of the site. I developed it mostly as a reference for how the SDK could be used, but the actual site might be useful as a quick way for teams to just store and manage files in a particular bucket that only the admin or host could adjust.

Some more useful reference is below:

• MinIO documentation
• Linode AWS SDK for PHP
• PHP manual around file uploads - not about S3, but reinforces how the file upload process is handled with PHP

Intro to Minio S3 Object Storage

 

Minio  S3 object storage is powerful and easily run in a Docker, LXD, Podman or other container environment. Super easy to setup, super easy to understand, and the latest console has a ton of features to improve the intuitiveness of the platform. 

The command run in the video is the same as in the previous blog:

sudo docker run -it --name=miniotest -p 9000:9000 -p 9001:9001 minio/minio server /data --console-address ":9001"

To map a volume to the container one can create a directory on the local machine and add the -v <localvolume>:/data to the above command. That should help keep the buckets and related files available if the container is ever removed.

Example below from the container created in this test.

More information is below:

https://docs.min.io/docs/minio-docker-quickstart-guide.html

https://www.truenas.com/docs/core/services/s3/

https://s3browser.com/

https://s3tools.org/s3cmd

Happy to field any questions, just let me know. 

Nextcloud External Storage and Apps

 

This post and video go through how to add external storage in Nextcloud and introduces the wide number of applications that can be used to tailor the functionality of Nextcloud. 

For the external storage, the example is S3 object storage from a Minio container. Minio is a fantastic project that allows for locally hosted S3 API equipped storage. It's also nice because it is quite easy to start and get running, particularly on Docker. 

The command I used to make the Minio test container is below:
sudo docker run -it --name minios3 -p 9000:9000 -p 9001:9001 minio/minio server /data

*Update: Minio's latest image (tested 4/2022) needs to define the console port or it tries to auto find a port that was a bit hit or miss for me.
Revised:

sudo docker run -it --name=miniotest -p 9000:9000 -p 9001:9001 minio/minio server /data --console-address ":9001"

I did need to make some cuts in the video which is why the mouse jumps in a couple of places. Most notably, I initially had the wrong IP address of the Minio instance. This was essentially because in the test environment both containers were running on the same system and couldn't connect using the host IP. External systems wouldn't have had that issue. Around minute 7:25 the change will show moving from the host IP to the actual IP of the container. 

In more detail:

Docker, Podman, and other container management tools assign IP addresses to each container service. When running as a group, say if using a pod and Kubernetes, or running the containers together with Docker Compose, the containers are part of a single network and can identify each other by the service name. 

In this example, the containers were created separately, and being on the same host where unable to reach each other using the address I provided. Changing to the Minio specific IP was all that was needed, but as this was out of scope for the video - and honestly not something that would normally come up, so I chose to omit the debugging.

Some more information and resources about both Nextcloud and Minio are below.

Nextcloud Docker

Minio

Minio Quickstart

Minio Docker