Blame view

Documentation/filesystems/squashfs.txt 10.4 KB
9eb425c04   Phillip Lougher   Squashfs: documen...
1
2
3
4
  SQUASHFS 4.0 FILESYSTEM
  =======================
  
  Squashfs is a compressed read-only filesystem for Linux.
812753d66   Phillip Lougher   Squashfs: Update ...
5
  It uses zlib/lzo/xz compression to compress files, inodes and directories.
9eb425c04   Phillip Lougher   Squashfs: documen...
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
  Inodes in the system are very small and all blocks are packed to minimise
  data overhead. Block sizes greater than 4K are supported up to a maximum
  of 1Mbytes (default block size 128K).
  
  Squashfs is intended for general read-only filesystem use, for archival
  use (i.e. in cases where a .tar.gz file may be used), and in constrained
  block device/memory systems (e.g. embedded systems) where low overhead is
  needed.
  
  Mailing list: squashfs-devel@lists.sourceforge.net
  Web site: www.squashfs.org
  
  1. FILESYSTEM FEATURES
  ----------------------
  
  Squashfs filesystem features versus Cramfs:
  
  				Squashfs		Cramfs
edf2e2811   Phillip Lougher   Squashfs: fix doc...
24
  Max filesystem size:		2^64			256 MiB
9eb425c04   Phillip Lougher   Squashfs: documen...
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
  Max file size:			~ 2 TiB			16 MiB
  Max files:			unlimited		unlimited
  Max directories:		unlimited		unlimited
  Max entries per directory:	unlimited		unlimited
  Max block size:			1 MiB			4 KiB
  Metadata compression:		yes			no
  Directory indexes:		yes			no
  Sparse file support:		yes			no
  Tail-end packing (fragments):	yes			no
  Exportable (NFS etc.):		yes			no
  Hard link support:		yes			no
  "." and ".." in readdir:	yes			no
  Real inode numbers:		yes			no
  32-bit uids/gids:		yes			no
  File creation time:		yes			no
899f45303   Phillip Lougher   squashfs: update ...
40
41
  Xattr support:			yes			no
  ACL support:			no			no
9eb425c04   Phillip Lougher   Squashfs: documen...
42
43
44
45
46
47
48
49
50
51
52
53
54
55
  
  Squashfs compresses data, inodes and directories.  In addition, inode and
  directory data are highly compacted, and packed on byte boundaries.  Each
  compressed inode is on average 8 bytes in length (the exact length varies on
  file type, i.e. regular file, directory, symbolic link, and block/char device
  inodes have different sizes).
  
  2. USING SQUASHFS
  -----------------
  
  As squashfs is a read-only filesystem, the mksquashfs program must be used to
  create populated squashfs filesystems.  This and other squashfs utilities
  can be obtained from http://www.squashfs.org.  Usage instructions can be
  obtained from this site also.
812753d66   Phillip Lougher   Squashfs: Update ...
56
57
  The squashfs-tools development tree is now located on kernel.org
  	git://git.kernel.org/pub/scm/fs/squashfs/squashfs-tools.git
9eb425c04   Phillip Lougher   Squashfs: documen...
58
59
60
  
  3. SQUASHFS FILESYSTEM DESIGN
  -----------------------------
4c1d204cd   Phillip Lougher   Squashfs: Update ...
61
62
  A squashfs filesystem consists of a maximum of nine parts, packed together on a
  byte alignment:
9eb425c04   Phillip Lougher   Squashfs: documen...
63
64
65
66
  
  	 ---------------
  	|  superblock 	|
  	|---------------|
4c1d204cd   Phillip Lougher   Squashfs: Update ...
67
68
69
  	|  compression  |
  	|    options    |
  	|---------------|
9eb425c04   Phillip Lougher   Squashfs: documen...
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
  	|  datablocks   |
  	|  & fragments  |
  	|---------------|
  	|  inode table	|
  	|---------------|
  	|   directory	|
  	|     table     |
  	|---------------|
  	|   fragment	|
  	|    table      |
  	|---------------|
  	|    export     |
  	|    table      |
  	|---------------|
  	|    uid/gid	|
  	|  lookup table	|
899f45303   Phillip Lougher   squashfs: update ...
86
87
88
  	|---------------|
  	|     xattr     |
  	|     table	|
9eb425c04   Phillip Lougher   Squashfs: documen...
89
90
91
92
  	 ---------------
  
  Compressed data blocks are written to the filesystem as files are read from
  the source directory, and checked for duplicates.  Once all file data has been
89cab5b57   Phillip Lougher   Squashfs: Update ...
93
94
  written the completed inode, directory, fragment, export, uid/gid lookup and
  xattr tables are written.
9eb425c04   Phillip Lougher   Squashfs: documen...
95

4c1d204cd   Phillip Lougher   Squashfs: Update ...
96
97
98
99
100
101
102
103
  3.1 Compression options
  -----------------------
  
  Compressors can optionally support compression specific options (e.g.
  dictionary size).  If non-default compression options have been used, then
  these are stored here.
  
  3.2 Inodes
9eb425c04   Phillip Lougher   Squashfs: documen...
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
  ----------
  
  Metadata (inodes and directories) are compressed in 8Kbyte blocks.  Each
  compressed block is prefixed by a two byte length, the top bit is set if the
  block is uncompressed.  A block will be uncompressed if the -noI option is set,
  or if the compressed block was larger than the uncompressed block.
  
  Inodes are packed into the metadata blocks, and are not aligned to block
  boundaries, therefore inodes overlap compressed blocks.  Inodes are identified
  by a 48-bit number which encodes the location of the compressed metadata block
  containing the inode, and the byte offset into that block where the inode is
  placed (<block, offset>).
  
  To maximise compression there are different inodes for each file type
  (regular file, directory, device, etc.), the inode contents and length
  varying with the type.
  
  To further maximise compression, two types of regular file inode and
  directory inode are defined: inodes optimised for frequently occurring
  regular files and directories, and extended types where extra
  information has to be stored.
4c1d204cd   Phillip Lougher   Squashfs: Update ...
125
  3.3 Directories
9eb425c04   Phillip Lougher   Squashfs: documen...
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
  ---------------
  
  Like inodes, directories are packed into compressed metadata blocks, stored
  in a directory table.  Directories are accessed using the start address of
  the metablock containing the directory and the offset into the
  decompressed block (<block, offset>).
  
  Directories are organised in a slightly complex way, and are not simply
  a list of file names.  The organisation takes advantage of the
  fact that (in most cases) the inodes of the files will be in the same
  compressed metadata block, and therefore, can share the start block.
  Directories are therefore organised in a two level list, a directory
  header containing the shared start block value, and a sequence of directory
  entries, each of which share the shared start block.  A new directory header
  is written once/if the inode start block changes.  The directory
  header/directory entry list is repeated as many times as necessary.
  
  Directories are sorted, and can contain a directory index to speed up
  file lookup.  Directory indexes store one entry per metablock, each entry
  storing the index/filename mapping to the first directory header
  in each metadata block.  Directories are sorted in alphabetical order,
  and at lookup the index is scanned linearly looking for the first filename
  alphabetically larger than the filename being looked up.  At this point the
  location of the metadata block the filename is in has been found.
89cab5b57   Phillip Lougher   Squashfs: Update ...
150
  The general idea of the index is to ensure only one metadata block needs to be
9eb425c04   Phillip Lougher   Squashfs: documen...
151
152
153
  decompressed to do a lookup irrespective of the length of the directory.
  This scheme has the advantage that it doesn't require extra memory overhead
  and doesn't require much extra storage on disk.
4c1d204cd   Phillip Lougher   Squashfs: Update ...
154
  3.4 File data
9eb425c04   Phillip Lougher   Squashfs: documen...
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
  -------------
  
  Regular files consist of a sequence of contiguous compressed blocks, and/or a
  compressed fragment block (tail-end packed block).   The compressed size
  of each datablock is stored in a block list contained within the
  file inode.
  
  To speed up access to datablocks when reading 'large' files (256 Mbytes or
  larger), the code implements an index cache that caches the mapping from
  block index to datablock location on disk.
  
  The index cache allows Squashfs to handle large files (up to 1.75 TiB) while
  retaining a simple and space-efficient block list on disk.  The cache
  is split into slots, caching up to eight 224 GiB files (128 KiB blocks).
  Larger files use multiple slots, with 1.75 TiB files using all 8 slots.
  The index cache is designed to be memory efficient, and by default uses
  16 KiB.
4c1d204cd   Phillip Lougher   Squashfs: Update ...
172
  3.5 Fragment lookup table
9eb425c04   Phillip Lougher   Squashfs: documen...
173
174
175
176
177
178
179
180
  -------------------------
  
  Regular files can contain a fragment index which is mapped to a fragment
  location on disk and compressed size using a fragment lookup table.  This
  fragment lookup table is itself stored compressed into metadata blocks.
  A second index table is used to locate these.  This second index table for
  speed of access (and because it is small) is read at mount time and cached
  in memory.
4c1d204cd   Phillip Lougher   Squashfs: Update ...
181
  3.6 Uid/gid lookup table
9eb425c04   Phillip Lougher   Squashfs: documen...
182
183
184
185
186
187
188
  ------------------------
  
  For space efficiency regular files store uid and gid indexes, which are
  converted to 32-bit uids/gids using an id look up table.  This table is
  stored compressed into metadata blocks.  A second index table is used to
  locate these.  This second index table for speed of access (and because it
  is small) is read at mount time and cached in memory.
4c1d204cd   Phillip Lougher   Squashfs: Update ...
189
  3.7 Export table
9eb425c04   Phillip Lougher   Squashfs: documen...
190
191
192
193
194
195
196
197
198
199
200
201
  ----------------
  
  To enable Squashfs filesystems to be exportable (via NFS etc.) filesystems
  can optionally (disabled with the -no-exports Mksquashfs option) contain
  an inode number to inode disk location lookup table.  This is required to
  enable Squashfs to map inode numbers passed in filehandles to the inode
  location on disk, which is necessary when the export code reinstantiates
  expired/flushed inodes.
  
  This table is stored compressed into metadata blocks.  A second index table is
  used to locate these.  This second index table for speed of access (and because
  it is small) is read at mount time and cached in memory.
4c1d204cd   Phillip Lougher   Squashfs: Update ...
202
  3.8 Xattr table
899f45303   Phillip Lougher   squashfs: update ...
203
204
205
206
207
208
209
210
211
212
213
214
  ---------------
  
  The xattr table contains extended attributes for each inode.  The xattrs
  for each inode are stored in a list, each list entry containing a type,
  name and value field.  The type field encodes the xattr prefix
  ("user.", "trusted." etc) and it also encodes how the name/value fields
  should be interpreted.  Currently the type indicates whether the value
  is stored inline (in which case the value field contains the xattr value),
  or if it is stored out of line (in which case the value field stores a
  reference to where the actual value is stored).  This allows large values
  to be stored out of line improving scanning and lookup performance and it
  also allows values to be de-duplicated, the value being stored once, and
25985edce   Lucas De Marchi   Fix common misspe...
215
  all other occurrences holding an out of line reference to that value.
899f45303   Phillip Lougher   squashfs: update ...
216
217
218
219
220
221
  
  The xattr lists are packed into compressed 8K metadata blocks.
  To reduce overhead in inodes, rather than storing the on-disk
  location of the xattr list inside each inode, a 32-bit xattr id
  is stored.  This xattr id is mapped into the location of the xattr
  list using a second xattr id lookup table.
9eb425c04   Phillip Lougher   Squashfs: documen...
222
223
224
225
226
227
  
  4. TODOS AND OUTSTANDING ISSUES
  -------------------------------
  
  4.1 Todo list
  -------------
899f45303   Phillip Lougher   squashfs: update ...
228
  Implement ACL support.
9eb425c04   Phillip Lougher   Squashfs: documen...
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
  
  4.2 Squashfs internal cache
  ---------------------------
  
  Blocks in Squashfs are compressed.  To avoid repeatedly decompressing
  recently accessed data Squashfs uses two small metadata and fragment caches.
  
  The cache is not used for file datablocks, these are decompressed and cached in
  the page-cache in the normal way.  The cache is used to temporarily cache
  fragment and metadata blocks which have been read as a result of a metadata
  (i.e. inode or directory) or fragment access.  Because metadata and fragments
  are packed together into blocks (to gain greater compression) the read of a
  particular piece of metadata or fragment will retrieve other metadata/fragments
  which have been packed with it, these because of locality-of-reference may be
  read in the near future. Temporarily caching them ensures they are available
  for near future access without requiring an additional read and decompress.
  
  In the future this internal cache may be replaced with an implementation which
  uses the kernel page cache.  Because the page cache operates on page sized
  units this may introduce additional complexity in terms of locking and
  associated race conditions.