Blame view

Documentation/filesystems/erofs.rst 10.6 KB
e66d8631d   Mauro Carvalho Chehab   docs: filesystems...
1
2
3
4
5
  .. SPDX-License-Identifier: GPL-2.0
  
  ======================================
  Enhanced Read-Only File System - EROFS
  ======================================
fdb053646   Gao Xiang   staging: erofs: a...
6
7
8
9
10
11
12
13
  Overview
  ========
  
  EROFS file-system stands for Enhanced Read-Only File System. Different
  from other read-only file systems, it aims to be designed for flexibility,
  scalability, but be kept simple and high performance.
  
  It is designed as a better filesystem solution for the following scenarios:
e66d8631d   Mauro Carvalho Chehab   docs: filesystems...
14

fdb053646   Gao Xiang   staging: erofs: a...
15
16
17
18
19
20
21
22
23
24
25
   - read-only storage media or
  
   - part of a fully trusted read-only solution, which means it needs to be
     immutable and bit-for-bit identical to the official golden image for
     their releases due to security and other considerations and
  
   - hope to save some extra storage space with guaranteed end-to-end performance
     by using reduced metadata and transparent file compression, especially
     for those embedded devices with limited memory (ex, smartphone);
  
  Here is the main features of EROFS:
e66d8631d   Mauro Carvalho Chehab   docs: filesystems...
26

fdb053646   Gao Xiang   staging: erofs: a...
27
28
29
30
31
32
33
   - Little endian on-disk design;
  
   - Currently 4KB block size (nobh) and therefore maximum 16TB address space;
  
   - Metadata & data could be mixed by design;
  
   - 2 inode versions for different requirements:
e66d8631d   Mauro Carvalho Chehab   docs: filesystems...
34
35
  
     =====================  ============  =====================================
ffafde478   Gao Xiang   erofs: update doc...
36
                            compact (v1)  extended (v2)
e66d8631d   Mauro Carvalho Chehab   docs: filesystems...
37
38
39
40
41
42
43
44
     =====================  ============  =====================================
     Inode metadata size    32 bytes      64 bytes
     Max file size          4 GB          16 EB (also limited by max. vol size)
     Max uids/gids          65536         4294967296
     File change time       no            yes (64 + 32-bit timestamp)
     Max hardlinks          65536         4294967296
     Metadata reserved      4 bytes       14 bytes
     =====================  ============  =====================================
fdb053646   Gao Xiang   staging: erofs: a...
45
46
47
48
  
   - Support extended attributes (xattrs) as an option;
  
   - Support xattr inline and tail-end data inline for all files;
516c115c9   Gao Xiang   staging: erofs: c...
49
   - Support POSIX.1e ACLs by using xattrs;
fdb053646   Gao Xiang   staging: erofs: a...
50
   - Support transparent file compression as an option:
ffafde478   Gao Xiang   erofs: update doc...
51
     LZ4 algorithm with 4 KB fixed-sized output compression for high performance.
fdb053646   Gao Xiang   staging: erofs: a...
52
53
54
  
  The following git tree provides the file system user-space tools under
  development (ex, formatting tool mkfs.erofs):
e66d8631d   Mauro Carvalho Chehab   docs: filesystems...
55
56
  
  - git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git
fdb053646   Gao Xiang   staging: erofs: a...
57
58
59
  
  Bugs and patches are welcome, please kindly help us and send to the following
  linux-erofs mailing list:
e66d8631d   Mauro Carvalho Chehab   docs: filesystems...
60
61
  
  - linux-erofs mailing list   <linux-erofs@lists.ozlabs.org>
fdb053646   Gao Xiang   staging: erofs: a...
62

fdb053646   Gao Xiang   staging: erofs: a...
63
64
  Mount options
  =============
e66d8631d   Mauro Carvalho Chehab   docs: filesystems...
65
  ===================    =========================================================
fdb053646   Gao Xiang   staging: erofs: a...
66
67
68
69
  (no)user_xattr         Setup Extended User Attributes. Note: xattr is enabled
                         by default if CONFIG_EROFS_FS_XATTR is selected.
  (no)acl                Setup POSIX Access Control List. Note: acl is enabled
                         by default if CONFIG_EROFS_FS_POSIX_ACL is selected.
4279f3f98   Gao Xiang   staging: erofs: t...
70
  cache_strategy=%s      Select a strategy for cached decompression from now on:
e66d8631d   Mauro Carvalho Chehab   docs: filesystems...
71
72
73
74
  
  		       ==========  =============================================
                           disabled  In-place I/O decompression only;
                          readahead  Cache the last incomplete compressed physical
4279f3f98   Gao Xiang   staging: erofs: t...
75
76
77
                                     cluster for further reading. It still does
                                     in-place I/O decompression for the rest
                                     compressed physical clusters;
e66d8631d   Mauro Carvalho Chehab   docs: filesystems...
78
                         readaround  Cache the both ends of incomplete compressed
4279f3f98   Gao Xiang   staging: erofs: t...
79
80
81
                                     physical clusters for further reading.
                                     It still does in-place I/O decompression
                                     for the rest compressed physical clusters.
e66d8631d   Mauro Carvalho Chehab   docs: filesystems...
82
83
  		       ==========  =============================================
  ===================    =========================================================
fdb053646   Gao Xiang   staging: erofs: a...
84
85
86
87
88
89
90
  
  On-disk details
  ===============
  
  Summary
  -------
  Different from other read-only file systems, an EROFS volume is designed
e66d8631d   Mauro Carvalho Chehab   docs: filesystems...
91
  to be as simple as possible::
fdb053646   Gao Xiang   staging: erofs: a...
92
93
94
95
96
97
98
99
100
  
                                  |-> aligned with the block size
     ____________________________________________________________
    | |SB| | ... | Metadata | ... | Data | Metadata | ... | Data |
    |_|__|_|_____|__________|_____|______|__________|_____|______|
    0 +1K
  
  All data areas should be aligned with the block size, but metadata areas
  may not. All metadatas can be now observed in two different spaces (views):
e66d8631d   Mauro Carvalho Chehab   docs: filesystems...
101

fdb053646   Gao Xiang   staging: erofs: a...
102
   1. Inode metadata space
e66d8631d   Mauro Carvalho Chehab   docs: filesystems...
103

fdb053646   Gao Xiang   staging: erofs: a...
104
      Each valid inode should be aligned with an inode slot, which is a fixed
ffafde478   Gao Xiang   erofs: update doc...
105
      value (32 bytes) and designed to be kept in line with compact inode size.
fdb053646   Gao Xiang   staging: erofs: a...
106
107
108
  
      Each inode can be directly found with the following formula:
           inode offset = meta_blkaddr * block_size + 32 * nid
e66d8631d   Mauro Carvalho Chehab   docs: filesystems...
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
      ::
  
  				    |-> aligned with 8B
  					    |-> followed closely
  	+ meta_blkaddr blocks                                      |-> another slot
  	_____________________________________________________________________
  	|  ...   | inode |  xattrs  | extents  | data inline | ... | inode ...
  	|________|_______|(optional)|(optional)|__(optional)_|_____|__________
  		|-> aligned with the inode slot size
  		    .                   .
  		    .                         .
  		.                              .
  		.                                    .
  	    .                                         .
  	    .                                              .
  	.____________________________________________________|-> aligned with 4B
  	| xattr_ibody_header | shared xattrs | inline xattrs |
  	|____________________|_______________|_______________|
  	|->    12 bytes    <-|->x * 4 bytes<-|               .
  			    .                .                 .
  			.                      .                   .
  		.                           .                     .
  	    ._______________________________.______________________.
  	    | id | id | id | id |  ... | id | ent | ... | ent| ... |
  	    |____|____|____|____|______|____|_____|_____|____|_____|
  					    |-> aligned with 4B
  							|-> aligned with 4B
fdb053646   Gao Xiang   staging: erofs: a...
136
137
  
      Inode could be 32 or 64 bytes, which can be distinguished from a common
e66d8631d   Mauro Carvalho Chehab   docs: filesystems...
138
      field which all inode versions have -- i_format::
fdb053646   Gao Xiang   staging: erofs: a...
139
140
  
          __________________               __________________
ffafde478   Gao Xiang   erofs: update doc...
141
         |     i_format     |             |     i_format     |
fdb053646   Gao Xiang   staging: erofs: a...
142
143
144
145
146
147
148
149
         |__________________|             |__________________|
         |        ...       |             |        ...       |
         |                  |             |                  |
         |__________________| 32 bytes    |                  |
                                          |                  |
                                          |__________________| 64 bytes
  
      Xattrs, extents, data inline are followed by the corresponding inode with
ffafde478   Gao Xiang   erofs: update doc...
150
151
      proper alignment, and they could be optional for different data mappings.
      _currently_ total 4 valid data mappings are supported:
fdb053646   Gao Xiang   staging: erofs: a...
152

e66d8631d   Mauro Carvalho Chehab   docs: filesystems...
153
      ==  ====================================================================
ffafde478   Gao Xiang   erofs: update doc...
154
155
156
157
       0  flat file data without data inline (no extent);
       1  fixed-sized output data compression (with non-compacted indexes);
       2  flat file data with tail packing data inline (no extent);
       3  fixed-sized output data compression (with compacted indexes, v5.3+).
e66d8631d   Mauro Carvalho Chehab   docs: filesystems...
158
      ==  ====================================================================
fdb053646   Gao Xiang   staging: erofs: a...
159
160
161
162
163
164
  
      The size of the optional xattrs is indicated by i_xattr_count in inode
      header. Large xattrs or xattrs shared by many different files can be
      stored in shared xattrs metadata rather than inlined right after inode.
  
   2. Shared xattrs metadata space
e66d8631d   Mauro Carvalho Chehab   docs: filesystems...
165

fdb053646   Gao Xiang   staging: erofs: a...
166
167
168
169
170
171
      Shared xattrs space is similar to the above inode space, started with
      a specific block indicated by xattr_blkaddr, organized one by one with
      proper align.
  
      Each share xattr can also be directly found by the following formula:
           xattr offset = xattr_blkaddr * block_size + 4 * xattr_id
e66d8631d   Mauro Carvalho Chehab   docs: filesystems...
172
173
174
175
176
177
178
      ::
  
  			    |-> aligned by  4 bytes
  	+ xattr_blkaddr blocks                     |-> aligned with 4 bytes
  	_________________________________________________________________________
  	|  ...   | xattr_entry |  xattr data | ... |  xattr_entry | xattr data  ...
  	|________|_____________|_____________|_____|______________|_______________
fdb053646   Gao Xiang   staging: erofs: a...
179
180
181
182
183
184
185
186
  
  Directories
  -----------
  All directories are now organized in a compact on-disk format. Note that
  each directory block is divided into index and name areas in order to support
  random file lookup, and all directory entries are _strictly_ recorded in
  alphabetical order in order to support improved prefix binary search
  algorithm (could refer to the related source code).
e66d8631d   Mauro Carvalho Chehab   docs: filesystems...
187
188
189
190
191
192
193
194
195
196
197
198
199
  ::
  
  		    ___________________________
  		    /                           |
  		/              ______________|________________
  		/              /              | nameoff1       | nameoffN-1
      ____________.______________._______________v________________v__________
      | dirent | dirent | ... | dirent | filename | filename | ... | filename |
      |___.0___|____1___|_____|___N-1__|____0_____|____1_____|_____|___N-1____|
  	\                           ^
  	\                          |                           * could have
  	\                         |                             trailing '\0'
  	    \________________________| nameoff0
fdb053646   Gao Xiang   staging: erofs: a...
200

e66d8631d   Mauro Carvalho Chehab   docs: filesystems...
201
  				Directory block
fdb053646   Gao Xiang   staging: erofs: a...
202
203
204
205
206
207
208
  
  Note that apart from the offset of the first filename, nameoff0 also indicates
  the total number of directory entries in this block since it is no need to
  introduce another on-disk field at all.
  
  Compression
  -----------
ffafde478   Gao Xiang   erofs: update doc...
209
  Currently, EROFS supports 4KB fixed-sized output transparent file compression,
e66d8631d   Mauro Carvalho Chehab   docs: filesystems...
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
  as illustrated below::
  
  	    |---- Variant-Length Extent ----|-------- VLE --------|----- VLE -----
  	    clusterofs                      clusterofs            clusterofs
  	    |                               |                     |   logical data
      _________v_______________________________v_____________________v_______________
      ... |    .        |             |        .    |             |  .          | ...
      ____|____.________|_____________|________.____|_____________|__.__________|____
  	|-> cluster <-|-> cluster <-|-> cluster <-|-> cluster <-|-> cluster <-|
  	    size          size          size          size          size
  	    .                             .                .                   .
  	    .                       .               .                  .
  		.                  .              .                .
  	_______._____________._____________._____________._____________________
  	    ... |             |             |             | ... physical data
  	_______|_____________|_____________|_____________|_____________________
  		|-> cluster <-|-> cluster <-|-> cluster <-|
  		    size          size          size
fdb053646   Gao Xiang   staging: erofs: a...
228
229
230
231
232
233
  
  Currently each on-disk physical cluster can contain 4KB (un)compressed data
  at most. For each logical cluster, there is a corresponding on-disk index to
  describe its cluster type, physical cluster address, etc.
  
  See "struct z_erofs_vle_decompressed_index" in erofs_fs.h for more details.