Filesystem quota management in Go
Stephan demonstrate here how Go interacts with operating system and how he developed our quota library.
Filesystem quotas are a rather simple concept. Instead of allowing every user to use all of the available space on a filesystem (or drive, to be less technical) each user is confined to limits - so called quotas. These quotas determine how much space an individual user or group may use. Currently Go does not include support for controlling filesystem quotas in its otherwise vast standard library. Unfortunately, no third-party libraries which would add this functionality exist either. One of our current project at the Anexia R&D department has to control quotas from within Go, therefore, we decided to come up with a solution of our own: a Go-native implementation. This gives me the opportunity to demonstrate how Go interacts with the underlying operating system at the deepest layers, namely at the syscall level. We will go over the steps again taken during the development of our solution. This will explain how the neccessary research was conducted and how we developed our quota library.
Kernel interface
In Linux libc exposes support for controlling filesystem quotas via its quotactl function. musl-libc shows that quotactl can be a thin wrapper around the underlying syscall of the same name:
int quotactl(int cmd, const char *special, int id, char *addr) { return syscall(SYS_quotactl, cmd, special, id, addr); }
This function, along with a few C preprocessor macros which are defined in musl's include/sys/quota.h, provides full filesystem quota support. As we can see, we only required a thin wrapper like the one present in musl-libc to make Go aware of filesystem quotas - this is the point where we start out adventure into implementing this functionality on top of Go.
Preparing a testbed
Before being able to conduct any tests, we need a testbed. We chose to create a new ext4 filesystem in a file, activate quotas and loop-mount this file:
$ truncate -s 1G /tmp/test.ext4 $ /sbin/mkfs.ext4 /tmp/test.ext4 $ sudo mkdir -p /mnt/quota_test $ sudo mount -o usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv1 /tmp/test.ext4 /mnt/quota_test $ sudo quotacheck -vucm /mnt/quota_test $ sudo quotacheck -vgcm /mnt/quota_test $ sudo quotaon -v /mnt/quota_test
You might need to install the quota utilities first. On Debian and Ubuntu the required package is called quota. After finishing the operations above we get a new filesystem with user and group quotas enabled, available at /mnt/quota_test. Take a note of output of the last command. It should be prefixed with /dev/loopN, whereas N is an arbitrary number. This device name will be needed later on.
Syscalls in Go
Go's standard library provides various functions for working directly with syscalls via its syscall package. Fortunately, this package also defines the SYS_QUOTACTL constant - holding the desired syscall number. Additionally the package provides some very handy helper functions - like BytePtrFromString - to which we will get back to at a later point. The package further provides us with the Syscall and Syscall6 functions, which issue syscalls. The first variant supports up to three arguments and Syscall6 - as its name suggests - supports up to six. As the quotactl syscall requires four arguments, we will need to make use of Syscall6.
Digging through the man page
The quotactl(2) man page explains the interface in detail. However, to start off we only need basic functionality, namely the ability to retrieve and set quotas. These operations are implemented through "subcommands", Q_GETQUOTA, which retrieves quota information, is documented to require a pointer to a specific data structure named dqblk, which we will have to implement in Go. Another interesting detail we need to have a closer look at is how the cmd argument to the quotactl call is built. The man page references the QCMD(subcmd, type) macro, which is defined in sys/quota.h, taking a "subcommand" and a "quota type", and applies a bitwise shift operation, followed by masking the type with an "AND" operation.
#define QCMD(cmd, type) (((cmd) << SUBCMDSHIFT) | ((type) & SUBCMDMASK))
As we only want to be able to retrieve quota information, this newly aquired knowledge should suffice.
Let's Go
With all required information in hand, we define a wrapper similar to the one found in musl-libc in Go:
func quotactl(cmd int, special string, id int, target unsafe.Pointer) (err error) { var deviceNamePtr *byte if deviceNamePtr, err = syscall.BytePtrFromString(special); err != nil { return } if _, _, errno := syscall.RawSyscall6(syscall.SYS_QUOTACTL, uintptr(cmd), uintptr(unsafe.Pointer(deviceNamePtr)), uintptr(id), uintptr(target), 0, 0); errno != 0 { err = os.NewSyscallError("quotactl", errno) } return }
As unsafe.Pointer is used again and again, it is important to have a closer look at it. unsafe.Pointer is possibly best compared to a void pointer in C. The type does not matter, we just need a pointer to data. Also, syscall.BytePtrFromString is interesting when interacting at such a low level. This is required as Go's string type is not compatible with NULL-terminated strings, as used in the kernel or in C. syscall.BytePtrFromString converts a Go string into such a NULL-terminated string, which is represented as the pointer to its first byte. The target argument is intentionally an unsafe.Pointer, as quotactl is able to fill other data structures, besides the mentioned dqblk. Using this approach in the first place allows for easier extension later on. Which brings us right to dqblk:
type Dqblk struct { DqbBHardlimit uint64 DqbBSoftlimit uint64 DqbCurSpace uint64 DqbIHardlimit uint64 DqbISoftlimit uint64 DqbCurInodes uint64 DqbBTime uint64 DqbITime uint64 DqbValid uint32 }
As you can see we can freely change the names of all fields, but we ensure the field order is preserved. Defining the required constants and a function implementing the QCMD macro in Go is left as an exercise to the reader.
Retrieving quota information
Now, having defined the data structures, and having converted the C-style snake-case identifiers to more Go-like camel case identifiers, we can implement the first public function which retrieves quotas:
func GetQuota(typ int, special string, id int) (result *Dqblk, err error) { result = &Dqblk{} if err = quotactl(qCmd(qGetQuota, typ), special, id, unsafe.Pointer(result)); err != nil { result = nil } return }
A sample invocation might be GetQuota(UsrQuota, "/dev/loop0", 0), which retrieves the user quota for UID 0 from "/dev/loop0". Substitute /dev/loop0 with the device name, which was returned when the testbed was created and see what happens: the prototype works.
Summary
This article has outlined the steps required for bringing low-level Linux kernel functionality to the world of Go. Retrieving filesystem quota information from the operating system was implemented in just a few lines of code. The approach taken – implementing the functionality natively in Go – allows creation of statically linked Go binaries without any external dependencies, which also greatly simplifies deployment. The full implementation of this example can be found at https://github.com/anexia-it/wad2018-quotactl - containing the discussed pieces. A more complete Go filesystem quota library, which is based on the research that also led to this article, can be found at https://github.com/anexia-it/fsquota. Hoping you enjoyed this journey into the depths of Go and kernel interfaces as much as I enjoyed working on this article.
