Firewalld and Podman - Protecting Your DB

 

The above output is an interesting discovery I found with the relationship between Podman and Firewalld (aka firewall-cmd). 

So I was tinkering with a semi-production environment that is a web server with Apache and PHP on it, and a MySQL database running as a container. This was originally just done quickly, but is becoming more and more important. Honestly, long term, I may want to just move the database to an actual VM that has all the bells and whistles - like Cron for scheduled backups, etc. I digress.

The point is I was reading an article that a lot of sites have MySQL databases without SSL. If you run your database in a container, there is a good chance it also doesn't have SSL since most Docker images do not include that out-of-the-box. I personally don't think it is a problem if the service is on a truly internal network, but I ran a Telnet check from a remote system and it responded. Holy ^$%%@#$!

Now the port shouldn't be exposed, the firewall-cmd rules for the external interface don't allow that port type. However, with further testing I was able to connect a phpmyadmin to that host from a completely remote system. Holy#$%^&*&^%$!

What happens is on RedHat / CentOS / Fedora, podman essentially knows about firewall-cmd and spins itself into the zone trusted. You get that source and everything is available from that source. This also allows the outside world to come to that port - which is NOT what we want.

Turns out though, that it is relatively straightforward to lock this down.

Backup all the databases using a mysqldump - or in this case just by using a PHPMyAdmin to export. 

Then destroy the current container. Run the following:

podman stop <DB container name>

podman rm <DB container name>

Assuming the container has its /var/lib/mysql directory mounted to the host using the -v or --volume at creation likely the backups won't need to be restored. Just re-run the container creation command (example below).

podman run -d --name mySQLdb -v <host directory>:/var/lib/mysql -p 127.0.0.1:3306:3306 --restart=unless-stopped -e MYSQL_ROOT_PASSWORD=<password> -e lower_case_table_names=1 mysql:5.7

The difference in the above command is that the localhost IP - 127.0.0.1 - is specified along with the port value. As such, the container is now only reachable by services or other containers running on the local system. 

Now testing with a telnet command to that host with port 3306 shows unreachable, but the local web server can still interact with the database. 

telnet: Unable to connect to remote host: No route to host

Most tutorials and explanations about Docker or Podman or containers in general usually cover the "-p" call as a network port. However, it can also be used to bind a specific IP:portnum which is very important to helping protect a system that is running on the Internet. 

I hope this can help others keep themselves safer when using containers in production.

Migrating Nextcloud from a Backup

Recently I was forced to retire my "server" because the performance and the file system were throwing errors. This warranted an upgrade from a 14 year-old laptop to an Intel Celeron NUC. Maybe one day I'll actually get a "real" server.....

I thought I'd share the steps I took, and also point out some gotchas that I came across. Let everyone learn from my mistakes :) . 

Some background:

The server essentially was a Ubuntu server instance with SMB (SAMBA) running locally on the system, and my Nextcloud with a separate MariaDB running in containers - the setup from this post. What I had done was back up all the files from the SMB share, and all the files from the mounted volumes of both the Nextcloud container and the MariaDB container. 

Backup:

Important: I also took a manual mysqldump of the 'nextcloud' database in the MariaDB container. This is important. 

Enter the MariaDB container by running:
sudo docker exec -it nxtclouddb bash

In the container, run 

mysqldump -u root -p nextcloud > nextclouddumpDATE.sql

Enter the database password and the file will be verified. I had a volume mounted to the /var/lib/mysql so the data for the MariaDB database was mapped to a directory in the host. Copying the backup file (nextclouddumpDATE.sql) to the /var/lib/mysql directory in the container allows the file to be accessible on the host so it can be copied to a separate backup. 

Back up complete and verified.

Migration:

At this time we can look at the reinstall. Since it was a huge jump in architecture I couldn't just pop in the hard drive from the old laptop to the new computer - needed to reinstall. Went with Ubuntu Server 20.04, ran updates, setup the firewall, installed SMB, set that up and verified the SSH. At this point I could add the external hard drives with the backups. 

Ubuntu Server is nice vs Ubuntu Desktop in that it doesn't auto-mount USB drives in a /media/<user name> directory. This is a feature in my opinion because with the server image the admin is in control. 

Update the /etc/fstab with drive labels (example):

LABEL=FourTBbackup /media2 ext4 defaults 0 0

sudo mount -a will bring mount the partition(s).

Here we can start copying data back. For the SMB share I have the share mounted to another external drive, so no data copying needed on that. The nextcloud instance was running off the main drive and that needed to be copied over. 

rsync -av --progress /media2/nxtcloud/* /home/user/nxtcloud 

Once complete, I tried to bring up the Nextcloud containers. This was a bit reckless in a way, but given all the data was backed up, it wasn't too bad. It was here I needed to dive down a few rabbit holes. 

Troubleshooting:

As soon as I visited the Nextcloud site, I was greeted by an "Internal Server Error" message. At the time I assumed the database was corrupt because I just ran off the restored database files - I didn't first import the database from backup like a good boy. This was not the problem. 

Turns out there were 3 problems:

• The new install warranted an IP change - need to update the Nextcloud config.php file allowed domains
• Permissions of the restored files and directories all need to be set as chown www-data:www-data - I believe this was the main reason for the 'Internal Server Error'
• The Docker image pulled a new version of Nextcloud and my version couldn't be migrated to the new version. You can check that in the config/config.php file.

With Nextcloud in Docker, even if you mount a volume from the host system, the image will add anything missing in the web server volume that it needs. In fact, all that was really needed where the /config, /data, /apps, /3rdparty directories. The /data directory is actually where all the files and other info is saved for the various users. All other files and directories can be removed from the volume mount on the host. More info here in this post.

So what was needed was to specify the older version of Nextcloud in my docker-compose.yml file (image: nextcloud:22.2), only copy the  /config, /data, /apps, /3rdparty directories to the mapped volume, and then rebuild the containers.

sudo docker-compose up -d

At that point, there was still a minor update on Nextcloud (from 22.2.02 to 22.2.10) but that was fine. 

Nextcloud was restored and everything was well.

I will say Docker is pretty handy for this - one can just docker-compose up and down to rebuild containers fast and test different things.

With Nextcloud, the platform is built fairly robust, but the mapping of files to the database and all the complex apps can make backup and restore tricky - particularly when migrating to a new system. Having all the database files, an actual DB dump, and all the Nextcloud files available is the #1 most important step. Back ups = options. 

I'm also pretty glad I made the move when I did - when I had a poor performing, but still usable system so I could verify the backups and be careful.

Hope this is helpful. I definitely learn more and more about Nextcloud every time I need to rebuild it, and hopefully this post can save others some valuable time.