29 Aug, 2013

4 commits

  • If the group descriptor fails validation, mark the whole blockgroup
    corrupt so that the inode/block allocators skip this group. The
    previous approach takes the risk of writing to a damaged group
    descriptor; hopefully it was never the case that the [ib]bitmap fields
    pointed to another valid block and got dirtied, since the memset would
    fill the page with 1s.

    Signed-off-by: Darrick J. Wong
    Signed-off-by: "Theodore Ts'o"

    Darrick J. Wong
     
  • When we notice a block-bitmap corruption (because of device failure or
    something else), we should mark this group as corrupt and prevent
    further block allocations/deallocations from it. Currently, we end up
    generating one error message for every block in the bitmap. This
    potentially could make the system unstable as noticed in some
    bugs. With this patch, the error will be printed only the first time
    and mark the entire block group as corrupted. This prevents future
    access allocations/deallocations from it.

    Also tested by corrupting the block
    bitmap and forcefully introducing the mb_free_blocks error:
    (1) create a largefile (2Gb)
    $ dd if=/dev/zero of=largefile oflag=direct bs=10485760 count=200
    (2) umount filesystem. use dumpe2fs to see which block-bitmaps
    are in use by largefile and note their block numbers
    (3) use dd to zero-out the used block bitmaps
    $ dd if=/dev/zero of=/dev/hdc4 bs=4096 seek=14 count=8 oflag=direct
    (4) mount the FS and delete the largefile.
    (5) recreate the largefile. verify that the new largefile does not
    get any blocks from the groups marked as bad.
    Without the patch, we will see mb_free_blocks error for each bit in
    each zero'ed out bitmap at (4). With the patch, we only see the error
    once per blockgroup:
    [ 309.706803] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 15: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
    [ 309.720824] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 14: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
    [ 309.732858] EXT4-fs error (device sdb4) in ext4_free_blocks:4802: IO failure
    [ 309.748321] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 13: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
    [ 309.760331] EXT4-fs error (device sdb4) in ext4_free_blocks:4802: IO failure
    [ 309.769695] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 12: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
    [ 309.781721] EXT4-fs error (device sdb4) in ext4_free_blocks:4802: IO failure
    [ 309.798166] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 11: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.
    [ 309.810184] EXT4-fs error (device sdb4) in ext4_free_blocks:4802: IO failure
    [ 309.819532] EXT4-fs error (device sdb4): ext4_mb_generate_buddy:735: group 10: 32768 clusters in bitmap, 0 in gd. blk grp corrupted.

    Google-Bug-Id: 7258357

    [darrick.wong@oracle.com]
    Further modifications (by Darrick) to make more obvious that this corruption
    bit applies to blocks only. Set the corruption flag if the block group bitmap
    verification fails.

    Original-author: Aditya Kali
    Signed-off-by: Darrick J. Wong
    Signed-off-by: "Theodore Ts'o"

    Darrick J. Wong
     
  • The block_group parameter to ext4_validate_block_bitmap is both used
    as a ext4_group_t inside the function and the same type is passed in
    by all callers. We might as well use the typedef consistently instead
    of open-coding the 'unsigned int'.

    Signed-off-by: Darrick J. Wong
    Signed-off-by: "Theodore Ts'o"

    Darrick J. Wong
     
  • The block bitmap verification code assumes that calling ext4_error()
    either panics the system or makes the fs readonly. However, this is
    not always true: when 'errors=continue' is specified, an error is
    printed but we don't return any indication of error to the caller,
    which is (probably) the block allocator, which pretends that the crud
    we read in off the disk is a usable bitmap. Yuck.

    A block bitmap that fails the check should at least return no bitmap
    to the caller. The block allocator should be told to go look in a
    different group, but that's a separate issue.

    The easiest way to reproduce this is to modify bg_block_bitmap (on a
    ^flex_bg fs) to point to a block outside the block group; or you can
    create a metadata_csum filesystem and zero out the block bitmaps.

    Signed-off-by: Darrick J. Wong
    Signed-off-by: "Theodore Ts'o"

    Darrick J. Wong
     

06 Jul, 2013

1 commit

  • The function ext4_get_group_number() was introduced as an optimization
    in commit bd86298e60b8. Unfortunately, this commit incorrectly
    calculate the group number for file systems with a 1k block size (when
    s_first_data_block is 1 instead of zero). This could cause the
    following kernel BUG:

    [ 568.877799] ------------[ cut here ]------------
    [ 568.877833] kernel BUG at fs/ext4/mballoc.c:3728!
    [ 568.877840] Oops: Exception in kernel mode, sig: 5 [#1]
    [ 568.877845] SMP NR_CPUS=32 NUMA pSeries
    [ 568.877852] Modules linked in: binfmt_misc
    [ 568.877861] CPU: 1 PID: 3516 Comm: fs_mark Not tainted 3.10.0-03216-g7c6809f-dirty #1
    [ 568.877867] task: c0000001fb0b8000 ti: c0000001fa954000 task.ti: c0000001fa954000
    [ 568.877873] NIP: c0000000002f42a4 LR: c0000000002f4274 CTR: c000000000317ef8
    [ 568.877879] REGS: c0000001fa956ed0 TRAP: 0700 Not tainted (3.10.0-03216-g7c6809f-dirty)
    [ 568.877884] MSR: 8000000000029032 CR: 24000428 XER: 00000000
    [ 568.877902] SOFTE: 1
    [ 568.877905] CFAR: c0000000002b5464
    [ 568.877908]
    GPR00: 0000000000000001 c0000001fa957150 c000000000c6a408 c0000001fb588000
    GPR04: 0000000000003fff c0000001fa9571c0 c0000001fa9571c4 000138098c50625f
    GPR08: 1301200000000000 0000000000000002 0000000000000001 0000000000000000
    GPR12: 0000000024000422 c00000000f33a300 0000000000008000 c0000001fa9577f0
    GPR16: c0000001fb7d0100 c000000000c29190 c0000000007f46e8 c000000000a14672
    GPR20: 0000000000000001 0000000000000008 ffffffffffffffff 0000000000000000
    GPR24: 0000000000000100 c0000001fa957278 c0000001fdb2bc78 c0000001fa957288
    GPR28: 0000000000100100 c0000001fa957288 c0000001fb588000 c0000001fdb2bd10
    [ 568.877993] NIP [c0000000002f42a4] .ext4_mb_release_group_pa+0xec/0x1c0
    [ 568.877999] LR [c0000000002f4274] .ext4_mb_release_group_pa+0xbc/0x1c0
    [ 568.878004] Call Trace:
    [ 568.878008] [c0000001fa957150] [c0000000002f4274] .ext4_mb_release_group_pa+0xbc/0x1c0 (unreliable)
    [ 568.878017] [c0000001fa957200] [c0000000002fb070] .ext4_mb_discard_lg_preallocations+0x394/0x444
    [ 568.878025] [c0000001fa957340] [c0000000002fb45c] .ext4_mb_release_context+0x33c/0x734
    [ 568.878032] [c0000001fa957440] [c0000000002fbcf8] .ext4_mb_new_blocks+0x4a4/0x5f4
    [ 568.878039] [c0000001fa957510] [c0000000002ef56c] .ext4_ext_map_blocks+0xc28/0x1178
    [ 568.878047] [c0000001fa957640] [c0000000002c1a94] .ext4_map_blocks+0x2c8/0x490
    [ 568.878054] [c0000001fa957730] [c0000000002c536c] .ext4_writepages+0x738/0xc60
    [ 568.878062] [c0000001fa957950] [c000000000168a78] .do_writepages+0x5c/0x80
    [ 568.878069] [c0000001fa9579d0] [c00000000015d1c4] .__filemap_fdatawrite_range+0x88/0xb0
    [ 568.878078] [c0000001fa957aa0] [c00000000015d23c] .filemap_write_and_wait_range+0x50/0xfc
    [ 568.878085] [c0000001fa957b30] [c0000000002b8edc] .ext4_sync_file+0x220/0x3c4
    [ 568.878092] [c0000001fa957be0] [c0000000001f849c] .vfs_fsync_range+0x64/0x80
    [ 568.878098] [c0000001fa957c70] [c0000000001f84f0] .vfs_fsync+0x38/0x4c
    [ 568.878105] [c0000001fa957d00] [c0000000001f87f4] .do_fsync+0x54/0x90
    [ 568.878111] [c0000001fa957db0] [c0000000001f8894] .SyS_fsync+0x28/0x3c
    [ 568.878120] [c0000001fa957e30] [c000000000009c88] syscall_exit+0x0/0x7c
    [ 568.878125] Instruction dump:
    [ 568.878130] 60000000 813d0034 81610070 38000000 7f8b4800 419e001c 813f007c 7d2bfe70
    [ 568.878144] 7d604a78 7c005850 54000ffe 7c0007b4 e8a10076 e87f0090 7fa4eb78
    [ 568.878160] ---[ end trace 594d911d9654770b ]---

    In addition fix the STD_GROUP optimization so that it works for
    bigalloc file systems as well.

    Signed-off-by: "Theodore Ts'o"
    Reported-by: Li Zhong
    Reviewed-by: Lukas Czerner
    Cc: stable@vger.kernel.org # 3.10

    Theodore Ts'o
     

06 Jun, 2013

1 commit

  • The test_root() function could potentially loop forever due to
    overflow issues. So rewrite test_root() to avoid this issue; as a
    bonus, it is 38% faster when benchmarked via a test loop:

    int main(int argc, char **argv)
    {
    int i;

    for (i = 0; i < 1 << 24; i++) {
    if (test_root(i, 7))
    printf("%d\n", i);
    }
    }

    Signed-off-by: "Theodore Ts'o"

    Theodore Ts'o
     

21 Apr, 2013

1 commit


10 Apr, 2013

1 commit

  • Currently in ENOSPC condition when writing into unwritten space, or
    punching a hole, we might need to split the extent and grow extent tree.
    However since we can not allocate any new metadata blocks we'll have to
    zero out unwritten part of extent or punched out part of extent, or in
    the worst case return ENOSPC even though use actually does not allocate
    any space.

    Also in delalloc path we do reserve metadata and data blocks for the
    time we're going to write out, however metadata block reservation is
    very tricky especially since we expect that logical connectivity implies
    physical connectivity, however that might not be the case and hence we
    might end up allocating more metadata blocks than previously reserved.
    So in future, metadata reservation checks should be removed since we can
    not assure that we do not under reserve.

    And this is where reserved space comes into the picture. When mounting
    the file system we slice off a little bit of the file system space (2%
    or 4096 clusters, whichever is smaller) which can be then used for the
    cases mentioned above to prevent costly zeroout, or unexpected ENOSPC.

    The number of reserved clusters can be set via sysfs, however it can
    never be bigger than number of free clusters in the file system.

    Note that this patch fixes the failure of xfstest 274 as expected.

    Signed-off-by: Lukas Czerner
    Signed-off-by: "Theodore Ts'o"
    Reviewed-by: Carlos Maiolino

    Lukas Czerner
     

04 Apr, 2013

2 commits

  • Currently on many places in ext4 we're using
    ext4_get_group_no_and_offset() even though we're only interested in
    knowing the block group of the particular block, not the offset within
    the block group so we can use more efficient way to compute block
    group.

    This patch introduces ext4_get_group_number() which computes block
    group for a given block much more efficiently. Use this function
    instead of ext4_get_group_no_and_offset() everywhere where we're only
    interested in knowing the block group.

    Signed-off-by: Lukas Czerner
    Signed-off-by: "Theodore Ts'o"

    Lukas Czerner
     
  • Currently in when getting the block group number for a particular
    block in ext4_block_in_group() we're using
    ext4_get_group_no_and_offset() which uses do_div() to get the block
    group and the remainer which is offset within the group.

    We don't need all of that in ext4_block_in_group() as we only need to
    figure out the group number.

    This commit changes ext4_block_in_group() to calculate group number
    directly. This shows as a big improvement with regards to cpu
    utilization. Measuring fallocate -l 15T on fresh file system with perf
    showed that 23% of cpu time was spend in the
    ext4_get_group_no_and_offset(). With this change it completely
    disappears from the list only bumping the occurrence of
    ext4_init_block_bitmap() which is the biggest user of
    ext4_block_in_group() by 4%. As the result of this change on my system
    the fallocate call was approx. 10% faster.

    However since there is '-g' option in mkfs which allow us setting
    different groups size (mostly for developers) I've introduced new per
    file system flag whether we have a standard block group size or
    not. The flag is used to determine whether we can use the bit shift
    optimization or not.

    Signed-off-by: Lukas Czerner
    Signed-off-by: "Theodore Ts'o"

    Lukas Czerner
     

03 Mar, 2013

1 commit


23 Feb, 2013

1 commit

  • ext4_has_free_clusters() should tell us whether there is enough free
    clusters to allocate, however number of free clusters in the file system
    is converted to blocks using EXT4_C2B() which is not only wrong use of
    the macro (we should have used EXT4_NUM_B2C) but it's also completely
    wrong concept since everything else is in cluster units.

    Moreover when calculating number of root clusters we should be using
    macro EXT4_NUM_B2C() instead of EXT4_B2C() otherwise the result might be
    off by one. However r_blocks_count should always be a multiple of the
    cluster ratio so doing a plain bit shift should be enough here. We
    avoid using EXT4_B2C() because it's confusing.

    As a result of the first problem number of free clusters is much bigger
    than it should have been and ext4_has_free_clusters() would return 1 even
    if there is really not enough free clusters available.

    Fix this by removing the EXT4_C2B() conversion of free clusters and
    using bit shift when calculating number of root clusters. This bug
    affects number of xfstests tests covering file system ENOSPC situation
    handling. With this patch most of the ENOSPC problems with bigalloc file
    system disappear, especially the errors caused by delayed allocation not
    having enough space when the actual allocation is finally requested.

    Signed-off-by: Lukas Czerner
    Signed-off-by: "Theodore Ts'o"
    Cc: stable@vger.kernel.org

    Lukas Czerner
     

13 Jan, 2013

1 commit

  • Validate the bh pointer before using it, since
    ext4_read_block_bitmap_nowait() might return NULL.

    I've seen this in fsfuzz testing.

    EXT4-fs error (device loop0): ext4_read_block_bitmap_nowait:385: comm touch: Cannot get buffer for block bitmap - block_group = 0, block_bitmap = 3925999616
    BUG: unable to handle kernel NULL pointer dereference at (null)
    IP: [] ext4_wait_block_bitmap+0x25/0xe0
    ...
    Call Trace:
    [] ext4_read_block_bitmap+0x35/0x60
    [] ext4_free_blocks+0x236/0xb80
    [] ? __getblk+0x36/0x70
    [] ? __find_get_block+0x8f/0x210
    [] ? kmem_cache_free+0x33/0x140
    [] ext4_xattr_release_block+0x1b5/0x1d0
    [] ext4_xattr_delete_inode+0xbe/0x100
    [] ext4_free_inode+0x7c/0x4d0
    [] ? ext4_mark_inode_dirty+0x88/0x230
    [] ext4_evict_inode+0x32c/0x490
    [] evict+0xa7/0x1c0
    [] iput_final+0xe3/0x170
    [] iput+0x3e/0x50
    [] ext4_add_nondir+0x4d/0x90
    [] ext4_create+0xeb/0x170
    [] vfs_create+0xac/0xd0
    [] lookup_open+0x185/0x1c0
    [] ? selinux_inode_permission+0xa9/0x170
    [] do_last+0x2d4/0x7a0
    [] path_openat+0xb3/0x480
    [] ? handle_mm_fault+0x251/0x3b0
    [] do_filp_open+0x49/0xa0
    [] ? __alloc_fd+0xdd/0x150
    [] do_sys_open+0x108/0x1f0
    [] sys_open+0x21/0x30
    [] system_call_fastpath+0x16/0x1b

    Also fix comment for ext4_read_block_bitmap_nowait()

    Signed-off-by: Eryu Guan
    Signed-off-by: "Theodore Ts'o"
    Cc: stable@vger.kernel.org

    Eryu Guan
     

22 Oct, 2012

1 commit

  • In mke2fs, we only checksum the whole bitmap block and it is right.
    While in the kernel, we use EXT4_BLOCKS_PER_GROUP to indicate the
    size of the checksumed bitmap which is wrong when we enable bigalloc.
    The right size should be EXT4_CLUSTERS_PER_GROUP and this patch fixes
    it.

    Also as every caller of ext4_block_bitmap_csum_set and
    ext4_block_bitmap_csum_verify pass in EXT4_BLOCKS_PER_GROUP(sb)/8,
    we'd better removes this parameter and sets it in the function itself.

    Signed-off-by: Tao Ma
    Signed-off-by: "Theodore Ts'o"
    Reviewed-by: Lukas Czerner
    Cc: stable@vger.kernel.org

    Tao Ma
     

17 Aug, 2012

1 commit

  • While in ext4_validate_block_bitmap(), if an block allocation bitmap
    is found to be invalid, we call ext4_error() while the block group is
    still locked. This causes ext4_commit_super() to call a function
    which might sleep while in an atomic context.

    There's no need to keep the block group locked at this point, so hoist
    the ext4_error() call up to ext4_validate_block_bitmap() and release
    the block group spinlock before calling ext4_error().

    The reported stack trace can be found at:

    http://article.gmane.org/gmane.comp.file-systems.ext4/33731

    Reported-by: Dave Jones
    Signed-off-by: "Theodore Ts'o"
    Cc: stable@vger.kernel.org

    Theodore Ts'o
     

01 Jul, 2012

1 commit


08 Jun, 2012

1 commit

  • Ext3 filesystems that are converted to use as many ext4 file system
    features as possible will enable uninit_bg to speed up e2fsck times.
    These file systems will have a native ext3 layout of inode tables and
    block allocation bitmaps (as opposed to ext4's flex_bg layout).
    Unfortunately, in these cases, when first allocating a block in an
    uninitialized block group, ext4 would incorrectly calculate the number
    of free blocks in that block group, and then errorneously report that
    the file system was corrupt:

    EXT4-fs error (device vdd): ext4_mb_generate_buddy:741: group 30, 32254 clusters in bitmap, 32258 in gd

    This problem can be reproduced via:

    mke2fs -q -t ext4 -O ^flex_bg /dev/vdd 5g
    mount -t ext4 /dev/vdd /mnt
    fallocate -l 4600m /mnt/test

    The problem was caused by a bone headed mistake in the check to see if a
    particular metadata block was part of the block group.

    Many thanks to Kees Cook for finding and bisecting the buggy commit
    which introduced this bug (commit fd034a84e1, present since v3.2).

    Reported-by: Sander Eikelenboom
    Reported-by: Kees Cook
    Signed-off-by: "Theodore Ts'o"
    Tested-by: Kees Cook
    Cc: stable@kernel.org

    Theodore Ts'o
     

02 Jun, 2012

1 commit

  • Pull Ext4 updates from Theodore Ts'o:
    "The major new feature added in this update is Darrick J Wong's
    metadata checksum feature, which adds crc32 checksums to ext4's
    metadata fields.

    There is also the usual set of cleanups and bug fixes."

    * tag 'ext4_for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4: (44 commits)
    ext4: hole-punch use truncate_pagecache_range
    jbd2: use kmem_cache_zalloc wrapper instead of flag
    ext4: remove mb_groups before tearing down the buddy_cache
    ext4: add ext4_mb_unload_buddy in the error path
    ext4: don't trash state flags in EXT4_IOC_SETFLAGS
    ext4: let getattr report the right blocks in delalloc+bigalloc
    ext4: add missing save_error_info() to ext4_error()
    ext4: add debugging trigger for ext4_error()
    ext4: protect group inode free counting with group lock
    ext4: use consistent ssize_t type in ext4_file_write()
    ext4: fix format flag in ext4_ext_binsearch_idx()
    ext4: cleanup in ext4_discard_allocated_blocks()
    ext4: return ENOMEM when mounts fail due to lack of memory
    ext4: remove redundundant "(char *) bh->b_data" casts
    ext4: disallow hard-linked directory in ext4_lookup
    ext4: fix potential integer overflow in alloc_flex_gd()
    ext4: remove needs_recovery in ext4_mb_init()
    ext4: force ro mount if ext4_setup_super() fails
    ext4: fix potential NULL dereference in ext4_free_inodes_counts()
    ext4/jbd2: add metadata checksumming to the list of supported features
    ...

    Linus Torvalds
     

16 May, 2012

1 commit


30 Apr, 2012

2 commits


21 Feb, 2012

2 commits

  • Get rid of this one:

    fs/ext4/balloc.c: In function 'ext4_wait_block_bitmap':
    fs/ext4/balloc.c:405:3: warning: format '%llu' expects argument of
    type 'long long unsigned int', but argument 6 has type 'sector_t' [-Wformat]

    Happens because sector_t is u64 (unsigned long long) or unsigned long
    dependent on CONFIG_64BIT.

    Signed-off-by: Heiko Carstens
    Acked-by: Randy Dunlap
    Signed-off-by: "Theodore Ts'o"

    Heiko Carstens
     
  • In ext4_read_{inode,block}_bitmap() we were setting bitmap_uptodate()
    before submitting the buffer for read. The is bad, since we check
    bitmap_uptodate() without locking the buffer, and so if another
    process is racing with us, it's possible that they will think the
    bitmap is uptodate even though the read has not completed yet,
    resulting in inodes and blocks potentially getting allocated more than
    once if we get really unlucky.

    Addresses-Google-Bug: 2828254

    Signed-off-by: "Theodore Ts'o"

    Theodore Ts'o
     

05 Jan, 2012

1 commit


22 Nov, 2011

1 commit


10 Sep, 2011

11 commits


28 Jun, 2011

1 commit

  • I found that ext4_ext_find_goal() and ext4_find_near()
    share the same code for returning a coloured start block
    based on i_block_group.

    We can refactor this into a common function so that they
    don't diverge in the future.

    Thanks to adilger for suggesting the new function name.

    Signed-off-by: Eric Sandeen
    Signed-off-by: "Theodore Ts'o"

    Eric Sandeen
     

25 May, 2011

1 commit

  • This patch adds an allocation request flag to the ext4_has_free_blocks
    function which enables the use of reserved blocks. This will allow a
    punch hole to proceed even if the disk is full. Punching a hole may
    require additional blocks to first split the extents.

    Because ext4_has_free_blocks is a low level function, the flag needs
    to be passed down through several functions listed below:

    ext4_ext_insert_extent
    ext4_ext_create_new_leaf
    ext4_ext_grow_indepth
    ext4_ext_split
    ext4_ext_new_meta_block
    ext4_mb_new_blocks
    ext4_claim_free_blocks
    ext4_has_free_blocks

    [ext4 punch hole patch series 1/5 v7]

    Signed-off-by: Allison Henderson
    Signed-off-by: "Theodore Ts'o"
    Reviewed-by: Mingming Cao

    Allison Henderson
     

09 May, 2011

1 commit


02 May, 2011

1 commit