From 2738f75a6835791cffdd4477db31e26a81d3585e Mon Sep 17 00:00:00 2001 From: Om Raheja Date: Thu, 15 Aug 2024 18:56:21 -0400 Subject: [PATCH] =?UTF-8?q?added=20version=201=20of=20writeup=20lets=20kee?= =?UTF-8?q?p=20going=20=F0=9F=94=A5?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- writeup/0.md | 26 ++++++++++++++++++++ writeup/1.md | 68 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 94 insertions(+) create mode 100644 writeup/0.md create mode 100644 writeup/1.md diff --git a/writeup/0.md b/writeup/0.md new file mode 100644 index 0000000..d4d80f0 --- /dev/null +++ b/writeup/0.md @@ -0,0 +1,26 @@ +# Introduction + +We created this guide because there was no good place to start on making an ext4 driver or how exactly the internals work. Most of the information is available on the [kernel specification](https://www.kernel.org/doc/html/v4.19/filesystems/ext4/index.html) but it's not like anyone is going to read through all that. We want to make low level development accessible to the average user today, so that we can [make abstractions](https://codeberg.org/gek/seabass.git) to focus on the high level stuff. + +# Motivation + +We are working on an ext4 driver for [OpenBSD](https://openbsd.org) + + +# Getting Started + +Even though we have code snippets in our code to mimic Jupyter notebooks, we recommend following along with the source code on your local device. For ease of understanding reasons, we are going to start with a thorough analysis of the OpenBSD ext3 source code, draw comparisons with the Linux ext4 source code, and then somehow "merge" the two, without plagarizing so we can keep our code in the standard OpenBSD license. + +For the first part of the tutorial, we will be using the OpenBSD `ext2fs` code. So run the following: + +```bash +sudo apt-get install cvs +# https://www.openbsd.org/anoncvs.html#CVSROOT +cvs -qd anoncvs@anoncvs1.usa.openbsd.org:/cvs checkout -P src +# wait patiently... +cd src/sys/ufs/ext2fs +``` + +Unless you are on an OpenBSD based system, you wil not be able to build the code. By the way, if you plan on messing with ext2fs too, you should probably setup a VM. + +Follow to the next page. diff --git a/writeup/1.md b/writeup/1.md new file mode 100644 index 0000000..0c3590f --- /dev/null +++ b/writeup/1.md @@ -0,0 +1,68 @@ +# A quick glance + +We can start at ext2fs/ext2fs.h and see what's in there. + +At first sight, we are presented with the following line: + +```c +#include +``` + +> Endianness is essentially bit order. On big endian systems, 0xABCDEF would be stored as the bytes [AB, CD, EF]. On little endian systems, the bytes would be [EF, CD, AB]. This is commonly implemented for computational efficiency, with little difference to the end user. Of course, as low level developers, this is something we will have to keep in mind. + +Now if we quickly glance at kernel.org, we can see the following: +> All fields in ext4 are written to disk in little-endian order. HOWEVER, all fields in jbd2 (the journal) are written to disk in big-endian order.^[1](https://docs.kernel.org/filesystems/ext4/overview.html)^ + +Journaling was already implemented in ext3, and the fields in the super block are implemented as incomplete features in the ext2 source. This is irrelevant to us for now. + +```c +#define BBSIZE 1024 +#define SBSIZE 1024 +#define BBOFF ((off_t)(0)) +#define SBOFF ((off_t)(BBOFF + BBSIZE)) +``` + +These are some standard macros to be used later in the code. All that matters is how the sizes of everything are all standardized. + +The super-block is the first 1 KB of data on the disk. It contains information about what files are present, disk health information, the amount of cylinders on the disk, and other technical information that allows us to optimize reading the filesystem, instead of the OS having to analyze each disk. + +The boot-block is 1 KB. ext4 allows no more than 1 KB of instructions to load up the filesystem. On MBR, the BIOS loads the first 512 bytes into memory, and on UEFI, there is a FAT32 filesystem with a bootable flag which instructs the BIOS on how to load the kernel. Those are "first-stage" bootloaders. The remainder of the 1 KB on disk is the "second-stage" bootloader. After this, you can load more bootloaders as necessary or get right into the kernel. + +The following line: + +```c +#define BBLOCK ((daddr_t)(0)) +#define SBLOCK ((daddr_t)(BBLOCK + BBSIZE / DEV_BSIZE)) +``` + +defines the block address of the boot and super block in memory. Obviously, the boot block is at address 0, but the super block's address depends on how many bytes fit in each block, or 1024 over that amount. `DEV_BSIZE` is a constant which has not been defined at this time, so hopefully we can figure out what it is going forward 🙏 + +> Inodes are, like in UFS, 32-bit unsigned integers and therefore ufsino_t. +> Disk blocks are 32-bit, if the filesystem isn't operating in 64-bit mode +> (the incompatible ext4 64BIT flag). More work is needed to properly use +> daddr_t as the disk block data type on both BE and LE architectures. +> XXX disk blocks are simply u_int32_t for now. +say the OpenBSD developers. The only point worth noting from this is that we have to implement 64-bit mode moving forward. + +```c +#define LOG_MINBSIZE 10 +#define MINBSIZE (1 << LOG_MINBSIZE) +#define LOG_MINFSIZE 10 +#define MINFSIZE (1 << LOG_MINFSIZE) +``` + +Each block is a fragment of 1024 bytes at minimum. ext4 was likely designed with the hope that eventually block sizes would increase as disk drive storage increases, and that it would remain extensible for years to come. + +```c +#define MAXMNTLEN 512 +``` + +The maximum length of a mount point is 512 characters. You can test this with `mount -t ext2fs `. Even on other systems, it's not likely to work beyond 512 bytes. + + + + + + + +