What are we doing here?

This blog includes a series of videos and references to help new users or enthusiasts better understand how to use open source and free technology tools. The quick links includes more information for accessing many of the tools covered along with other references to learn more for taking advantage of these tools.

Click HERE to see the full list of topics covered!
Showing posts with label Linux. Show all posts
Showing posts with label Linux. Show all posts

The 'sed' command in Linux

So I am building a new project involving PHP and S3 object storage, and as part of the solution I am looking at adjusting some of the defaults in the PHP configuration. This is not too hard to do with interactive commands, but a bit more complex when scripting the solution in a Dockerfile for automated install.

First hurdle: The docker image doesn't have a text editor. No nano, no vi, no nothing. This is actually a security measure. The general rule with containers is to make them as small as possible to keep the images small, but also improve security - the less on the server, the less can go wrong or get used the wrong way.

Second hurdle: Replacing a couple of lines of text in the php.ini file. 

Here is where 'sed' comes in.

When learning how to pipe or append data to a file I've often used | tee, or > or >> to send data to a file. 

Example:

$echo "some random text" > test.txt

That would create a file called test.txt with the 'some random text' as the content of the file. 

To append you can use the >> flag. Building on the previous example:

$echo "some more random text" >> test.txt

This adds to the file like below. 

some random text
some more random text

That is all fine, and works in the PHP containers from Docker Hub. However, if one uses the example php.ini files - php.ini-development or php.ini-production some of the values are written already. 

Here is where the 'sed' command can be employed. 

We have a very long config file with a bunch of comments and several settings that PHP can use to overwrite or adjust the defaults. In my case I want to change the default upload file size from 2M to something larger. With a text editor it is pretty easy to just search for that line and change it. To do this automatically, 'sed' is perfect. 

The syntax for substituting a line of text with sed is as below:

sed 's/<original text>/<replacement text>/' <file to change>

An example, if a file has the phrase 'some random text' we can change the sed output to 'some more text' with the below command.

sed 's/random/more/' filename

The output would be 'some more text'. 

IMPORTANT: 'sed' just changes the output, but it does not change the information in the file. To insert the changes to the file use the '-i' flag.

For example, if I wish to permanently replace the word 'random' with the word 'more', I would change the previous command to something like the following:

sed -i 's/random/more/' filename

This is a super useful command that honestly doesn't get a lot of attention, at least not in my decade plus of putzing around with Linux. Again really useful for scripting and I should have an example build in another couple of weeks.

Package managers: APT vs DNF

 

Here we are looking at the package managers 'apt' and 'dnf'. Both are modern and powerful tools for searching, installing, and managing the applications and packages installed in their respective Linux derivatives. Both share a lot of common commands as well, so it is very familiar to work with either. 

Not mentioned in the video, but a wonderful feature of open source software, is both work in most distributions and are not 100% relegated to those specific distros that ship with each respectively. That said, the package manager is a powerful component of a distribution, and I feel it is worth trying to learn the default tooling for each distribution one really wants to spend time with.  

This effort is part of a series I have rolling around in my noggin around package managers and their importance to the adoption of Linux by most people. I strongly feel that while they predate, and perhaps even inspired the concepts of App stores or Google Play used by mobile devices, not a lot of credit is really given to them. What makes package managers and the relevant repositories even more powerful than app stores is, because they run as commands, they can be scripted. Scripting allows more ease in management by sysadmins and others who need to work with a large fleet of devices. 

Before App stores, Linux was always safer then Windows primarily because software was curated for the distribution of choice. Installed packages, so long as they were coming from the package manager and the associated, known-good repos, had, and continue to have, way less risk than just going to websites and downloading software to run on a Windows PC. Package managers are faster, more reliable, and safer than what many people use to install and manage software on their computing devices, making learning them very important. 

Hopefully with just a few simple commands like 'apt update', 'apt upgrade', 'dnf update', 'apt install <whatever package you want in the repo>', and 'apt remove <installed package you do not want>', etc. new users can better take advantage of Linux and open source tools. 

More reference:

https://docs.fedoraproject.org/en-US/quick-docs/dnf-vs-apt/

https://en.wikipedia.org/wiki/APT_(software)

https://en.wikipedia.org/wiki/DNF_(software)


More BASH - Auto Sorting Files

BASH SMASH REHASH

I'm sharing a recent script I wrote that might be helpful. 

I had a picture folder on my NAS that was just simply way too large for use on a relatively slow Wifi connection. Essentially hundreds of picture and video files from our family photos.

What would happen on our computers is that when you want to open one file it takes forever as the file viewer tries to cache all the images in the folder to make flipping through them faster. That's a nice feature, but not when the folder / directory is dozens to hundreds of gigabytes in size. 

This script essentially prompts for which folder to sort (must be absolute), then prompts for how many files the user wants to sort per folder. The example below prompts to sort a directory with around 200 files into directories of 20 files apiece. 


Link to the code: autosort.sh

I included a lot of comments so hopefully the script makes sense. Again just a useful tool and might be helpful for those interested in shell scripting. 

BASH! (Bourne Again SHell)

 

BASH and the Linux terminal are essential tools in Linux, and can also be be a very powerful programming language. As shown, it allows for a user to run through the operating system, create, edit, delete files and directories, and have full control over permissions. 

In regards to scripting, probably the most powerful thing about using BASH is the ability to interact and leverage existing Linux commands to grab appropriate input and output. The timeconverter.sh script demonstrates this capability by calling the default 'date' command and using it to get a point of reference of where the user is (or rather where the system is set to), and then using that to help calculate other times around the globe.

What I like about this script - some self pride showing through, sorry - is it's very fast to run and execute. So if one needs to schedule a meeting for 4 pm their local time and wants to see what that time would be in other parts of the world you can run through a slew of possible times in no time at all! 

A few corrections:
Please excuse some typos and little goofs in the video, like having the extra '/' at the end of #!/bin/bash, and finding the /usr/bin/ directory. Hopefully showing the fixes were equally helpful :). The script also doesn't necessarily need the .sh at the end. That is just a habit since many shell scripts have that, but one could simply name the file timeconverter or magictimes or whatever. I should add it is more common for users to add scripts to /usr/local/bin rather than /usr/bin, though as shown either works. 

Tip: cat /etc/environment to see where the PATH links to. PATH in Unix/BSD/Linux defines where all the commands a user can use are located so that they can be run without having to know where each command file is located.

If anyone notices any bugs - the trickiest part is the daylight savings / standard time section - just please leave a comment and I can update.

------------------------------------------

timeconverter:

Source code of the script

--------------------------------------------

Additional reference:

BASH shell

Linux Permissions

The Linux Command Line

LAMP stack setup on Alpine Linux



This video covers setting up a LAMP stack (Linux+Apache+MySQL(Maria)+PHP) in Alpine Linux. Alpine is a super lightweight distro popular with developers using Docker, but also a valid choice for VM work or standalone implementation. 

This video walks through the process of setting up the various components of LAMP, though omits the Alpine install (it takes minutes if you know what you want and what your doing - select sys). Overall I'm quite impressed with Alpine, though there would certainly be conveniences just running Ubuntu Server, Alpine does the job with a fraction of the CPU/memory/capacity/cruft so if you can figure out how it can work for you, I think its worth a look.

Here are the steps shown in the video:
Alpine LAMP deployment

apk add apache2

apk add mariadb mariadb-client

rc-service start apache2

Check the homepage at the machine's IP, confirm you see 'It works'

Source files are located in /var/www/localhost/htdocs#

Ensure repositories are added - /etc/apk/repositories  <- uncomment commmunity and edge

apk add php7 php7-mysqli phpmyadmin php7-apache2

service restart apache2

Create php test doc - nano phpinfo.php

service mariadb start

Setup Maria

Error message appears, need to first run the /etc/init.d/mariadb setup

Re-run service mariadb start

Then we change the password for mariadb root user (this gets created on the system by the mariadb-client).
Run mysql_secure_installation

Walk through the prompts to set a root user password

Setup phpmyadmin

chmod -R 777 /usr/share/webapps/phpmyadmin
chmod 755 /etc/phpmyadmin/config.inc.php
ln -s /usr/share/webapps/phpmyadmin/ /var/www/localhost/htdocs/phpmyadmin


Setup phpmyadmin user in MariaDB
mysql -u root -p
 - enter password

>CREATE USER 'pmauser'@'%' IDENTIFIED BY '<password of choice>';
>GRANT ALL PRIVILEGES ON *.* TO 'pmauser'@'%' WITH GRANT OPTION;
>FLUSH PRIVILEGES;

Open page server IP/phpmyadmin.

Enter the pmauser and given password.

Now phpmyadmin can be used to manage the mariadb and create new databases

<missed this in the video>
To ensure services start on boot need to run 
rc-update add <service name>

The last line are steps to ensure apache2 and mariadb are started if the system reboots, an 'auto-restart' or 'start on boot' behavior. 

References I found helpful for setting this up are below:

I do understand that this stack isn't necessarily the best in terms of performance or security, its simply the 'bare-minimum' to start developing. For optimization I'd suggest reading more on general Apache or NGINX settings, as well as looking to additional 3rd party sources about the various ways to integrate PHP with a web server. 

Overview of operating systems - Linux compared


This is an overview of Windows, macOS, and Linux operating systems from a usage perspective. While the other videos have been using Linux for various tutorials, this video attempts to just offer more background about how Linux is crafted and why it has so many distributions. The ugly term is Fragmentation, which is true, but one could also look at that as choice and options.

Since this is only focusing primarily on the desktop, desktop user interfaces (UI) are also explained a bit, as well as highlighting useful features/commands for installing applications on Linux.

This is not an in-depth comparison as there are many many differences in terms of how the architectures, permissions, and code bases differ between each OS. I hope this at least offers some background as to how the flexibility of Linux can lead to some complexity, and how that complexity can be managed.

Some more sources for images and market share are largely taken from Wikipedia:
https://en.wikipedia.org/wiki/Linux
https://en.wikipedia.org/wiki/Usage_share_of_operating_systems
https://netmarketshare.com
(Note: there is a bit of variance in the numbers I show and the raw desktop numbers from NetMarketShare data, but there is a constant variance. Windows is still king of desktops, Linux rules in the server space and, through Android, the mobile markets as well.)

Linux overview and introduction


This video is somewhat in depth on my thoughts about Linux and its background. For those of you just looking for how to install, feel free to jump to minute 35.

What we cover is a slightly more detailed overview of Linux, including how to get started and some things (like enable your firewall) to run when first installing Linux. Linux Mint is used as the example just because it is popular and has a more familiar user interface for people used to Windows.

For more information about the source material and references used, please see the links below. To download some popular Linux distros and find out more info about where to learn all the ins and outs of Linux, please check out the Quick Links in the sidebar.

Linux commands:

Creating a bootable USB:

Still have more to come, hope this is helpful.