This article explores the many ways content creators, photographers, videographers, and production companies can leverage open source tools to store, share, and collaborate on their media assets. Different methods can depend on the size the of the team - from a single individual to a multi-national teams - and examine the pluses and minuses of each approach.
A single creator
Creators making and storing large files have a plethora of options available. The most simple is just to copy files onto ones workstation or laptop, edit, and publish. Often this quickly becomes a problem, particularly for individuals working with 4K or even 8K content. For these editors, often the simplest approach is to simply use an external hard drive(s). This approach is perfectly viable, but can quickly become an issue as users fill up more and more hard drives.
Another issue related to scale, is performance. A direct attach drive over USB or USB-C in theory can handle upwards of 10 Gb/s, but larger drives that are still traditional spinning rust (a normal hard drive that spins) will have a max throughput of around 250 Mb/s - 1/4~1/2 the throughput of the interface. NVMe SSDs are available and becoming ever more cost competitive at the 500 GB, 1 TB and 2 TB sizes, but will be noticeably more expensive than traditional hard drives at higher capacities. However, even though external SSDs are performant, they are not redundant - so manual backups will be needed, and the SSDs will eventually wear out. Always make sure data is backed up in some form. Having external drives over time can also be unwieldy with lots of data since the storage per drive cannot be expanded, meaning content creators will often have a pile of drives carefully labeled with different projects potentially getting spread across multiple drives. Essentially sprawl.
The introduction of 8K footage, for production houses and creators is another major issue. 8K footage is truly massive in capacity creating upwards of over 120 GB of content per minute.* An external hard drive or SSD will quickly get filled potentially within a single shoot. Creators need more storage and in a different format to keep up.
Upgrading to a NAS
Network Attached Storage (NAS) is, as the name suggests, storage that is accessible over a network. What it means in practice is users of a subnet (IP range) can access and share files that are located in a single or multiple servers. In Windows land with Active Directory this feature is just the share file feature in the Windows Explorer. Typically when talking about 'a NAS' usually IT administrators refer to a specific server designed with storage in mind that has a drive management, RAID, a file system, and the ability to share the files using a file share protocol. The most common protocols are SMB or Samba (open source compatible SMB), NFS, and AFS. For most in the creator or video production space, SMB will be the primary protocol because it is well sported on Windows, Linux, and macOS environments. Even iPadOS for iPad devices has some support for SMB in the files app.
Moving from a single or multiple external drives to a NAS has several benefits. First most NAS appliances or software projects will have the ability to create a RAID group to span multiple drives together into a single storage pool. This is useful so that multiple hard drives or SSDs can be grouped into a larger total capacity than any single drive would offer, allow projects to all be grouped together in a single master folder. Additionally, RAID will allow for greater performance for reads and writes as it offers more drives and total bandwidth, plus RAID will offer some level of redundancy to help keep data available. Another advantage of using a NAS is that data can be shared across groups, no direct cables need to be plugged in, and everyone on the network can work off a joint project or folder(s). A NAS is always one of the first steps once creators move from a one man operation to a larger group.
CAUTION: RAID is not a back up, and a second pool that is perhaps larger in capacity, but slower in performance is always recommended to back up the data to.
Choosing or building a NAS
The likes of QNAP and Synology, or Asustor offer entry-level NAS appliances which are good first steps. Typically, the entry-level boxes are rather under powered, however, and very limited in terms of how many drives one can use, etc. For non-technical users perhaps an entry-level NAS makes sense, but building one on perhaps old or leftover hardware with new drives can often have more performance - plus reduce e-waste!
Users interested in building a NAS can look at a variety of open source projects, such as TrueNAS, Open Media Vault, Unraid, and more. Personally, I recommend TrueNAS as it is well supported, has a corporation maintaining the project with Enterprise options for larger organizations, and offers an attractive GUI available via a web browser for setting up and managing the drives, creating users and shares. TrueNAS also has a native implementation of the ZFS file system which is extremely robust with built-in RAID support, copy-on-write operations, unlimited snapshots, and almost unlimited scalability - 256 quaddrillion zetabytes. For perspective, that is similar to buying the entire storage market of all hard drives in a year and connecting them all together. ZFS can also be expanded buy adding more RAID groups (called VDEVs) to a pool so storage can always be expanded. ZFS also has a replicate function called 'zfs send' which can send a snapshot(s) of data to a separate pool either on the same or different host quickly. The second pool or the backup pool, can have completely different hardware, different RAID layout, etc., but the ZFS file structure can still operate and be recovered usually in seconds should there be a need. TrueNAS supports all the major NAS protocols, as well as WebDAV for HTTP/HTTPS transfers, and has the ability to expand functionality with 3rd party projects, VMs, Jails (TrueNAS CORE) or containers (TrueNAS SCALE), making the project quite versatile.
For users interested, more information about getting started with TrueNAS is here.
Cross-site and International Collaboration
In 2020, the world was introduced to lockdowns, disease, and working from home gained unheard of traction and interest. The old adage "necessity is the mother of invention" was never truer. Knowledge workers, including in the creative space, were some of the first to move to working from home leading to a major shift in the office paradigm and a boom in laptop sales. File access was suddenly something that needed to be reinvented.
For users connecting remotely, often a NAS will not be the correct choice, or at least not the total solution for a few reasons. Fist, remote workers are remote and on a different network. NAS protocols - the aforementiond SMB, NFS, AFS - are not built for Internet file access. Most NAS protocols expect a constant connection to the files and will create file locks for open documents. HTTP/HTTPS traffic was designed to handle gaps and mulitiple hops - routing between different servers and routers - when accessing files and is thus the preferred protocol for nearly all Internet-based traffic.
Another important reason not to expose a NAS to the Internet is security. Virtually no NAS provider ever recommends a user to expose the system to the Internet as the appliances are built for back end storage work over a LAN. Especially when using some proprietary systems, there is very little to any auditing being done on the system's firmware and base OS code. Examples abound.**
For multi-site, international collaboration, the most secure and reliable way to access files is via the same medium that gave birth to the Internet - a website. Nextcloud is a total collaboration platform for storing, sharing, and creating documents and files. It includes powerful tooling and apps to track notes, create user / team tasks, manage groups and access, create survey forms, and much more. For creators looking to collaborate with other team members, Nextcloud can even mount an local NAS to the platform so that users on the network editing video, sound or image files from the NAS can then share their results via Nextcloud using secure HTTPS without having to copy the collateral to the platform. The platform has robust file versioning, and with customizable logos and an app-based model for enabling different functionality, Nextcloud can be customized to almost any workflow desired.
Nextcloud is installed as a website and can be run with either Apache or NGINX web servers. The project has several ways to install and get started - raw source, bespoke VM images, or Docker images. Since the platform is built around web servers, it can adhere to the most robust TLS/SSL encryption standards that are well established, with additional security that could be added using load balancers and firewalls possible.
More information on getting started with Nextcloud is here.
Putting it all together
For industries dealing with or creating large files, there are a multitude of ways to store, share, and protect data. For individual users, local storage could be enough, but will quickly fill up and become hard to manage. Networked file storage in the form of a NAS system, make storage and file management easier, and also allow for teams of editors to more easily work together. Growing even larger, or for teams spread out across different locations, Nextcloud is a total platform that is both secure and capable not only for file sharing and storage, but also group collaboration.
Ref:
*8K file sizes https://www.signiant.com/resources/tech-article/file-size-growth-bandwidth-conundrum/
** Synology and WD vulnerabilitys: https://www.securityweek.com/western-digital-synology-nas-vulnerabilities-exposed-millions-of-users-files/
** Asustor vulnerabilities: https://www.theverge.com/2022/2/22/22945962/asustor-nas-deadbolt-ransomware-attack
Get TrueNAS: https://www.truenas.com/
Get Nextcloud: https://www.nextcloud.com/