PHP with S3

 

This blog and video runs through getting started with PHP and S3. The subsequent code is all open source in the GitHub page located at the below link.

https://github.com/JoeMrCoffee/yourS3objects

Previously we looked at MinIO using it's console and UI, as well as quick set up in docker. This is a slightly more advanced look at interacting with object storage from a programmatic standpoint. 

PHP has an open SDK to tie in with S3 object storage that is provided by AWS. The video, as well as the source code in GitHub installs the SDK as part of the Docker bring up and build. This is done in the Dockerfile, and is very similar to installing the MongoDB packs required to make a PHP connection to a MongoDB database. 

From the code all the connection information to the S3 object store is in the 'header.php' file. If one wanted to use this site, or just its source code, they could also adjust the endpoint to the appropriate values. The 'header.php' file gets included in all the subsequent pages using the PHP include command, so everything else will follow the values in the header.php file.

The video talks a bit about the usefulness of the site. I developed it mostly as a reference for how the SDK could be used, but the actual site might be useful as a quick way for teams to just store and manage files in a particular bucket that only the admin or host could adjust.

Some more useful reference is below:

• MinIO documentation
• Linode AWS SDK for PHP
• PHP manual around file uploads - not about S3, but reinforces how the file upload process is handled with PHP

The 'sed' command in Linux

So I am building a new project involving PHP and S3 object storage, and as part of the solution I am looking at adjusting some of the defaults in the PHP configuration. This is not too hard to do with interactive commands, but a bit more complex when scripting the solution in a Dockerfile for automated install.

First hurdle: The docker image doesn't have a text editor. No nano, no vi, no nothing. This is actually a security measure. The general rule with containers is to make them as small as possible to keep the images small, but also improve security - the less on the server, the less can go wrong or get used the wrong way.

Second hurdle: Replacing a couple of lines of text in the php.ini file. 

Here is where 'sed' comes in.

When learning how to pipe or append data to a file I've often used | tee, or > or >> to send data to a file. 

Example:

$echo "some random text" > test.txt

That would create a file called test.txt with the 'some random text' as the content of the file. 

To append you can use the >> flag. Building on the previous example:

$echo "some more random text" >> test.txt

This adds to the file like below. 

some random text
some more random text

That is all fine, and works in the PHP containers from Docker Hub. However, if one uses the example php.ini files - php.ini-development or php.ini-production some of the values are written already. 

Here is where the 'sed' command can be employed. 

We have a very long config file with a bunch of comments and several settings that PHP can use to overwrite or adjust the defaults. In my case I want to change the default upload file size from 2M to something larger. With a text editor it is pretty easy to just search for that line and change it. To do this automatically, 'sed' is perfect. 

The syntax for substituting a line of text with sed is as below:

sed 's/<original text>/<replacement text>/' <file to change>

An example, if a file has the phrase 'some random text' we can change the sed output to 'some more text' with the below command.

sed 's/random/more/' filename

The output would be 'some more text'. 

IMPORTANT: 'sed' just changes the output, but it does not change the information in the file. To insert the changes to the file use the '-i' flag.

For example, if I wish to permanently replace the word 'random' with the word 'more', I would change the previous command to something like the following:

sed -i 's/random/more/' filename

This is a super useful command that honestly doesn't get a lot of attention, at least not in my decade plus of putzing around with Linux. Again really useful for scripting and I should have an example build in another couple of weeks.

A look at Snaps

 

Snaps are another way that developers package applications to make them installable in Linux.  Simlar to Flatpaks they are sandboxed and typically - though they don't have to be - used to run desktop applications on the Linux desktop. 

Snaps leverage the SquashFS file system which compresses the software for smaller footprints when installed on devices. When run the packages are uncompressed and mounted - shown in the video when running the 'df -alh' command. Because of the decompression needed when starting the application, sometimes the start up is a tad slower which is also shown in the video.

Snaps have had some controversy because the Snap store is hosted by Canonical and mirrors are not allowed. This organization is done for security, but there are members in the open source community who fundamentally resist this dynamic. Flatpaks would be the major alternative, though similarly, in practice, most flatpaks are hosted in a single instance in Flathub. 

More information about Snaps and contributing located below.

https://snapcraft.io/

https://en.wikipedia.org/wiki/Snap_(software)

Package managers: Flatpak - Linux Desktop Applications

 

Flatpak is quickly becoming the best way to get Linux desktop applications up and running on a Linux PC. Flatpak applications operate in a sandbox with their own images and environments that download and mirror the existing system in an isolated environment. Once installed, the applications themselves offer a responsive, essentially native experience.

Case in point, all my work requires me to use Google Chrome, Slack, and occasionally Signal on my Linux desktop. All of these can run as Flatpak applications, and in some cases perform even better or more reliably as a Flatpak install vs a native desktop client. 

When first installing Flatpaks, there is some overhead. When installing, if there is a specific environment - say a graphics driver / library - required, than the application will list it as an additional download. This can eat away at disk space versus a native application, but often it is trivial in the long run given the size of modern disk drives, and the fact that most of these environments will be re-used by other applications overtime. 

While not terribly in depth, the above video seeks to demonstrate quickly the different ways for getting started with Flatpaks, how to use the command line interface, and hopefully how to better use Flatpaks to get up an running with Linux on the desktop (i.e. a computer with point and click, not just a terminal).

More information:

https://flatpak.org/

https://docs.flatpak.org/en/latest/basic-concepts.html

Good info from System76

More specific permission settings that may be needed with certain Flatpaks

Turn down gamma in Linux under X11

 I purchased a second hand computer recently, an old Thinkpad X280, which is perfectly fine for most all of my needs. Only problem is the screen is awful. 

In an effort to slightly modify the contrast I found that there is a command in Linux under X11 (x-server is the traditional image renderer in Linux, so this may not work if your distro is using the new Wayland compositor). The command is xgamma. 

Example:

~$xgamma -gamma 0.7 

The above will reduce the gamma of the screen by 30% according to the X11 compositor. To make the changes stick, one can update the hidden file .profile in his/her home directory with the following:

xgamma -gamma 0.7 &> /dev/null

The &> /dev/null basically captures any would be output that results from the setting and throws it to /dev/null - basically throws it away. This is useful because otherwise there is an annoying "Okay to proceed" notice when first booting the system. 

Just another find which is available if not obvious in Linux! 

Reference:

https://linux.die.net/man/1/xgamma

https://forums.linuxmint.com/viewtopic.php?t=334249

 

Package managers: APT vs DNF

 

Here we are looking at the package managers 'apt' and 'dnf'. Both are modern and powerful tools for searching, installing, and managing the applications and packages installed in their respective Linux derivatives. Both share a lot of common commands as well, so it is very familiar to work with either. 

Not mentioned in the video, but a wonderful feature of open source software, is both work in most distributions and are not 100% relegated to those specific distros that ship with each respectively. That said, the package manager is a powerful component of a distribution, and I feel it is worth trying to learn the default tooling for each distribution one really wants to spend time with.  

This effort is part of a series I have rolling around in my noggin around package managers and their importance to the adoption of Linux by most people. I strongly feel that while they predate, and perhaps even inspired the concepts of App stores or Google Play used by mobile devices, not a lot of credit is really given to them. What makes package managers and the relevant repositories even more powerful than app stores is, because they run as commands, they can be scripted. Scripting allows more ease in management by sysadmins and others who need to work with a large fleet of devices. 

Before App stores, Linux was always safer then Windows primarily because software was curated for the distribution of choice. Installed packages, so long as they were coming from the package manager and the associated, known-good repos, had, and continue to have, way less risk than just going to websites and downloading software to run on a Windows PC. Package managers are faster, more reliable, and safer than what many people use to install and manage software on their computing devices, making learning them very important. 

Hopefully with just a few simple commands like 'apt update', 'apt upgrade', 'dnf update', 'apt install <whatever package you want in the repo>', and 'apt remove <installed package you do not want>', etc. new users can better take advantage of Linux and open source tools. 

More reference:

https://docs.fedoraproject.org/en-US/quick-docs/dnf-vs-apt/

https://en.wikipedia.org/wiki/APT_(software)

https://en.wikipedia.org/wiki/DNF_(software)

Minio in an Alpine Linux VM

 

Alpine is a really nice, really light Linux distribution that is often use as the base for a lot of containerized applications. This video demonstrates how to create an appliance-like VM with a minimal Alpine install to run Minio from boot. 

All of the steps aside from running the start up command using cron are in the documentation below. It is truly very simple to get working in Alpine if you just need something simple to act as an S3 object store with HTTP/HTTPS file sharing capabilities. 

A couple of mistakes in the videos are there because by copying the commands from the documentation I forgot to adjust the start up and cron commands to the directory we gave it. This is why at minute 5:50 I am checking the wrong directory and the new directory is shown (miniodata vs minio). Overall it is pretty simple and perhaps the mkdir command can be skipped depending on your preference.

References:

For the basic setup on Linux:
https://min.io/docs/minio/linux/index.html

For Debian/Ubuntu or RedHat/CentOS/Fedora installs:
https://min.io/docs/minio/linux/operations/install-deploy-manage/deploy-minio-single-node-single-drive.html 

If there are any questions, please let me know in the comments. 

Upgrading Nextcloud in Docker

 

This is a quick post on updating and maintaining a Docker install of Nextcloud. I have had my install pretty much untouched since I first put it up about a year or two ago. The whole time it's been fine, but I heard about performance improvements in later versions and have begun the process of upgrading the service. 

Here are some steps to check for. 

Based on my previous restore experience, I know that going from one version of docker to the next can sometimes be version dependent. What has worked in managing the upgrades is to upgrade one version at a time. 

To do this, first check what Nextcloud describes as the newest version. Enter the system as the 'admin' user, go to 'Settings' -> 'Overview'. A screen like below should show.

Nextcloud will give a few suggestions of areas to make your instance more secure or highlight any issues (this is a private instance without HTTPS, as noted, and without a lot of the Calendar and other apps), and it will display what the next version is. 

That next version will be the upgrade target for the Docker instance.

In your host, you can first pull the new Docker image of Nextcloud by running:

sudo docker image pull nextcloud:<the target version from the UI>

This will save some time in the upgrade.

Next make sure everything is saved and no one is using the containers. Stop the containers with:

sudo docker stop <nextcloud container name> <nextcloud DB container name>

In my sample docker-compose reference the command would look like:

sudo docker stop nxtcloud nxtclouddb

Then modify the docker-compose.yml file with the new image specified just downloaded. For example 25.0.3 shown below. 

app:

image: nextcloud:25.0.3

container_name: nxtcloud

restart: unless-stopped

  ports: 

  .....

Once changed bring up the containers again.

sudo docker-compose up

Here I omit the -d for running in the background. This offers a terminal view of the upgrade similar to below.

Give that a few minutes, and then the Nextcloud should be upgraded to the new version. 

To ensure the containers can be run in the background without an active terminal, stop the containers again, and re-run the sudo docker-compose up -d with the -d flag.

Some more information about Nextcloud is throughout the blog - just click on the Overview page or search by the Nextcloud tags on the right-hand side. 

For more information about the reference docker-compose to get started with it and a MariaDB instance check out the source in GitHub.

Save USB ports using Logitech Devices in Linux

A new find. I was watching a tech reviewer video of some Logitech stuff. I am a huge fan of Logitech especially on Linux as seems to have the best driver support for both there unifying receiver and bluetooth. In the review, they mentioned that the support software for Logitech can allow a single unifying receiver connect multiple devices. OMG!

Googling quickly about how to accomplish the same task on Linux, which Logitech doesn't officially support with their software, again the community came to the rescue. I stumbled upon Solaar!

Solaar allows for device management of Logitech stuff with a simple GUI and easy install. In PopOS - I imagine Ubuntu as well, likely Debian too - you can simply run 'apt search solaar', and then install with a 'sudo apt install solaar'. 

Launch the application from the application center - in my case hit the Window key and just type 'solaar'. Like any Linux software with a GUI it should appear.

To combine the devices connected on a single receiver you can simply start a search. There will be a warning that if the devices are connected you need to unplug them for them to appear. Unplug, re-plug the 2 USB receivers. 

The devices and their respective receivers should appear. 

To pair a new device click on a single receiver. Click 'Pair new device'. 

Unplug the mouse receiver or keyboard. 

The device will start broadcasting, and the other receiver will pick it up. If the mouse is removed, there is a confirmation button that needs to be clicked. Tab to the 'Confirm' button from the keyboard and press 'Enter'. 

At this point you should have 2 Logitech wireless devices on a single receiver, thus saving a USB port in Linux, or perhaps saving the need to get a new receiver in case one is lost!

More info on the project: https://pwr-solaar.github.io/Solaar/

Backing up MongoDB

Explored in previous posts and videos, MongoDB is a great way to store information without a schema. All of the information is stored in a JSON like format called BSON which essentially are documents, but similar to a SQL Database the data cannot be directly read on the host system. 

I was interested in learning some more about how to backup the files. Turns out that MongoDB provides a dump tool similar to MySQL called mongodump and mongorestore. 

To get started one can run the following command on their host system or container. If you used the MongoDB Docker image, the mongodump command can be run inside the container directly. 

Local Run

mongodump -u=<user> -p=<password> --authenticationDatabase=admin --db=<database to download>

The part '--authenticationDatabase' is important. On the MongoDB container the users allowed to access the database are specified are in that 'admin' database on the Mongo instance, and it must be defined in the mongodump command for the connection to be successful.

Similar to mysqldump, the mongodump command can also be used from a remote host.

Install the MongoDB Command Line Database Tools on the remote host. They offer the tools for Windows, Linux, and MacOS. 

*Note on Fedora 36, using the RedHat/CentOS 8.0 package worked. *

Modify the previous command to the below:

Remote Run

mongodump --host=<Domain/IP> --port=<Mongo service port - default is 27017> -u=<user> -p=<password> --authenticationDatabase=admin --db=<database to download>

Assuming a connection can be made all of the database files and relevant metadata will come down to the directory where the command was run. A successful run will look similar to the below. 

Once complete you will see a directory called 'dump' with the database(s) listed as directories inside. 

To specify the output location on the local system (the system being used to access and backup the MongoDB instance) you may use the '-o' or '--output' flag. 

Remote Run | Specify Download Location

mongodump --host=<Domain/IP> --port=<Mongo service port> -u=<user> -p=<password> --authenticationDatabase=admin --db=<database to download> -o=<directory location>

Restoring data from a Mongodump

The companion utility to mongodump is mongorestore. Mongorestore works similarly to mongodump, but the user specifies the db and the back up files. 

mongorestore --host <Domian/IP:Port> --authenticationDatabase=admin -u <user> -p <password> -d <db to restore> <backup folder location>

One additional tip: 

If the container was installed using the same docker-compose from the PHP and MongoDB tutorial, the port for the MongoDB Instance isn't exposed on the local system. This is good from a security standpoint. The companion web server accesses using the hostname from the private Podman / Docker network. To access the container host from the physical machine you need to use the IP address of the MongoDB container itself.

Using either Docker or Podman, run the inspect command to check to find the Network information. 

sudo podman inspect <container name or ID> | grep "IPAddress" OR
sudo docker inspect <container name or ID> | grep "IPAddress"

Additional information:

Information about using mongodump

Information about using mongorestore