08 Aug, 2010

1 commit

  • * 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4: (40 commits)
    ext4: Adding error check after calling ext4_mb_regular_allocator()
    ext4: Fix dirtying of journalled buffers in data=journal mode
    ext4: re-inline ext4_rec_len_(to|from)_disk functions
    jbd2: Remove t_handle_lock from start_this_handle()
    jbd2: Change j_state_lock to be a rwlock_t
    jbd2: Use atomic variables to avoid taking t_handle_lock in jbd2_journal_stop
    ext4: Add mount options in superblock
    ext4: force block allocation on quota_off
    ext4: fix freeze deadlock under IO
    ext4: drop inode from orphan list if ext4_delete_inode() fails
    ext4: check to make make sure bd_dev is set before dereferencing it
    jbd2: Make barrier messages less scary
    ext4: don't print scary messages for allocation failures post-abort
    ext4: fix EFBIG edge case when writing to large non-extent file
    ext4: fix ext4_get_blocks references
    ext4: Always journal quota file modifications
    ext4: Fix potential memory leak in ext4_fill_super
    ext4: Don't error out the fs if the user tries to make a file too big
    ext4: allocate stripe-multiple IOs on stripe boundaries
    ext4: move aio completion after unwritten extent conversion
    ...

    Fix up conflicts in fs/ext4/inode.c as per Ted.

    Fix up xfs conflicts as per earlier xfs merge.

    Linus Torvalds
     

04 Aug, 2010

2 commits


02 Aug, 2010

1 commit


27 Jul, 2010

1 commit

  • __GFP_NOFAIL is going away, so add our own retry loop. Also add
    jbd2__journal_start() and jbd2__journal_restart() which take a gfp
    mask, so that file systems can optionally (re)start transaction
    handles using GFP_KERNEL. If they do this, then they need to be
    prepared to handle receiving an PTR_ERR(-ENOMEM) error, and be ready
    to reflect that error up to userspace.

    Signed-off-by: "Theodore Ts'o"

    Theodore Ts'o
     

16 Jul, 2010

1 commit

  • OCFS2 uses t_commit trigger to compute and store checksum of the just
    committed blocks. When a buffer has b_frozen_data, checksum is computed
    for it instead of b_data but this can result in an old checksum being
    written to the filesystem in the following scenario:

    1) transaction1 is opened
    2) handle1 is opened
    3) journal_access(handle1, bh)
    - This sets jh->b_transaction to transaction1
    4) modify(bh)
    5) journal_dirty(handle1, bh)
    6) handle1 is closed
    7) start committing transaction1, opening transaction2
    8) handle2 is opened
    9) journal_access(handle2, bh)
    - This copies off b_frozen_data to make it safe for transaction1 to commit.
    jh->b_next_transaction is set to transaction2.
    10) jbd2_journal_write_metadata() checksums b_frozen_data
    11) the journal correctly writes b_frozen_data to the disk journal
    12) handle2 is closed
    - There was no dirty call for the bh on handle2, so it is never queued for
    any more journal operation
    13) Checkpointing finally happens, and it just spools the bh via normal buffer
    writeback. This will write b_data, which was never triggered on and thus
    contains a wrong (old) checksum.

    This patch fixes the problem by calling the trigger at the moment data is
    frozen for journal commit - i.e., either when b_frozen_data is created by
    do_get_write_access or just before we write a buffer to the log if
    b_frozen_data does not exist. We also rename the trigger to t_frozen as
    that better describes when it is called.

    Signed-off-by: Jan Kara
    Signed-off-by: Mark Fasheh
    Signed-off-by: Joel Becker

    Jan Kara
     

30 Mar, 2010

1 commit

  • …it slab.h inclusion from percpu.h

    percpu.h is included by sched.h and module.h and thus ends up being
    included when building most .c files. percpu.h includes slab.h which
    in turn includes gfp.h making everything defined by the two files
    universally available and complicating inclusion dependencies.

    percpu.h -> slab.h dependency is about to be removed. Prepare for
    this change by updating users of gfp and slab facilities include those
    headers directly instead of assuming availability. As this conversion
    needs to touch large number of source files, the following script is
    used as the basis of conversion.

    http://userweb.kernel.org/~tj/misc/slabh-sweep.py

    The script does the followings.

    * Scan files for gfp and slab usages and update includes such that
    only the necessary includes are there. ie. if only gfp is used,
    gfp.h, if slab is used, slab.h.

    * When the script inserts a new include, it looks at the include
    blocks and try to put the new include such that its order conforms
    to its surrounding. It's put in the include block which contains
    core kernel includes, in the same order that the rest are ordered -
    alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
    doesn't seem to be any matching order.

    * If the script can't find a place to put a new include (mostly
    because the file doesn't have fitting include block), it prints out
    an error message indicating which .h file needs to be added to the
    file.

    The conversion was done in the following steps.

    1. The initial automatic conversion of all .c files updated slightly
    over 4000 files, deleting around 700 includes and adding ~480 gfp.h
    and ~3000 slab.h inclusions. The script emitted errors for ~400
    files.

    2. Each error was manually checked. Some didn't need the inclusion,
    some needed manual addition while adding it to implementation .h or
    embedding .c file was more appropriate for others. This step added
    inclusions to around 150 files.

    3. The script was run again and the output was compared to the edits
    from #2 to make sure no file was left behind.

    4. Several build tests were done and a couple of problems were fixed.
    e.g. lib/decompress_*.c used malloc/free() wrappers around slab
    APIs requiring slab.h to be added manually.

    5. The script was run on all .h files but without automatically
    editing them as sprinkling gfp.h and slab.h inclusions around .h
    files could easily lead to inclusion dependency hell. Most gfp.h
    inclusion directives were ignored as stuff from gfp.h was usually
    wildly available and often used in preprocessor macros. Each
    slab.h inclusion directive was examined and added manually as
    necessary.

    6. percpu.h was updated not to include slab.h.

    7. Build test were done on the following configurations and failures
    were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
    distributed build env didn't work with gcov compiles) and a few
    more options had to be turned off depending on archs to make things
    build (like ipr on powerpc/64 which failed due to missing writeq).

    * x86 and x86_64 UP and SMP allmodconfig and a custom test config.
    * powerpc and powerpc64 SMP allmodconfig
    * sparc and sparc64 SMP allmodconfig
    * ia64 SMP allmodconfig
    * s390 SMP allmodconfig
    * alpha SMP allmodconfig
    * um on x86_64 SMP allmodconfig

    8. percpu.h modifications were reverted so that it could be applied as
    a separate patch and serve as bisection point.

    Given the fact that I had only a couple of failures from tests on step
    6, I'm fairly confident about the coverage of this conversion patch.
    If there is a breakage, it's likely to be something in one of the arch
    headers which should be easily discoverable easily on most builds of
    the specific arch.

    Signed-off-by: Tejun Heo <tj@kernel.org>
    Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
    Cc: Ingo Molnar <mingo@redhat.com>
    Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>

    Tejun Heo
     

06 Mar, 2010

1 commit

  • * 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs-2.6: (33 commits)
    quota: stop using QUOTA_OK / NO_QUOTA
    dquot: cleanup dquot initialize routine
    dquot: move dquot initialization responsibility into the filesystem
    dquot: cleanup dquot drop routine
    dquot: move dquot drop responsibility into the filesystem
    dquot: cleanup dquot transfer routine
    dquot: move dquot transfer responsibility into the filesystem
    dquot: cleanup inode allocation / freeing routines
    dquot: cleanup space allocation / freeing routines
    ext3: add writepage sanity checks
    ext3: Truncate allocated blocks if direct IO write fails to update i_size
    quota: Properly invalidate caches even for filesystems with blocksize < pagesize
    quota: generalize quota transfer interface
    quota: sb_quota state flags cleanup
    jbd: Delay discarding buffers in journal_unmap_buffer
    ext3: quota_write cross block boundary behaviour
    quota: drop permission checks from xfs_fs_set_xstate/xfs_fs_set_xquota
    quota: split out compat_sys_quotactl support from quota.c
    quota: split out netlink notification support from quota.c
    quota: remove invalid optimization from quota_sync_all
    ...

    Fixed trivial conflicts in fs/namei.c and fs/ufs/inode.c

    Linus Torvalds
     

05 Mar, 2010

1 commit

  • CONFIG_BUFFER_DEBUG seems to have been removed from the documentation
    somewhere around 2.4.15 and seemingly hasn't been available even
    longer. It is, however, still referenced at one place from the jbd
    code (one is a copy of the other header). Time to clean it up

    Signed-off-by: Christoph Egger
    Signed-off-by: Jan Kara

    Christoph Egger
     

23 Dec, 2009

1 commit

  • This is a bit complicated because we are trying to optimize when we
    send barriers to the fs data disk. We could just throw in an extra
    barrier to the data disk whenever we send a barrier to the journal
    disk, but that's not always strictly necessary.

    We only need to send a barrier during a commit when there are data
    blocks which are must be written out due to an inode written in
    ordered mode, or if fsync() depends on the commit to force data blocks
    to disk. Finally, before we drop transactions from the beginning of
    the journal during a checkpoint operation, we need to guarantee that
    any blocks that were flushed out to the data disk are firmly on the
    rust platter before we drop the transaction from the journal.

    Thanks to Oleg Drokin for pointing out this flaw in ext3/ext4.

    Signed-off-by: "Theodore Ts'o"

    Theodore Ts'o
     

07 Dec, 2009

1 commit

  • Now that the SLUB seems to be fixed so that it respects the requested
    alignment, use kmem_cache_alloc() to allocator if the block size of
    the buffer heads to be allocated is less than the page size.
    Previously, we were using 16k page on a Power system for each buffer,
    even when the file system was using 1k or 4k block size.

    Signed-off-by: "Theodore Ts'o"

    Theodore Ts'o
     

30 Sep, 2009

1 commit

  • The /proc/fs/jbd2//history was maintained manually; by using
    tracepoints, we can get all of the existing functionality of the /proc
    file plus extra capabilities thanks to the ftrace infrastructure. We
    save memory as a bonus.

    Signed-off-by: "Theodore Ts'o"

    Theodore Ts'o
     

18 Aug, 2009

1 commit


17 Jun, 2009

1 commit


14 Apr, 2009

1 commit


26 Mar, 2009

1 commit


11 Feb, 2009

1 commit

  • If we race with commit code setting i_transaction to NULL, we could
    possibly dereference it. Proper locking requires the journal pointer
    (to access journal->j_list_lock), which we don't have. So we have to
    change the prototype of the function so that filesystem passes us the
    journal pointer. Also add a more detailed comment about why the
    function jbd2_journal_begin_ordered_truncate() does what it does and
    how it should be used.

    Thanks to Dan Carpenter for pointing to the
    suspitious code.

    Signed-off-by: Jan Kara
    Signed-off-by: "Theodore Ts'o"
    Acked-by: Joel Becker
    CC: linux-ext4@vger.kernel.org
    CC: ocfs2-devel@oss.oracle.com
    CC: mfasheh@suse.de
    CC: Dan Carpenter

    Jan Kara
     

17 Jan, 2009

1 commit


09 Jan, 2009

1 commit

  • * 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4: (57 commits)
    jbd2: Fix oops in jbd2_journal_init_inode() on corrupted fs
    ext4: Remove "extents" mount option
    block: Add Kconfig help which notes that ext4 needs CONFIG_LBD
    ext4: Make printk's consistently prefixed with "EXT4-fs: "
    ext4: Add sanity checks for the superblock before mounting the filesystem
    ext4: Add mount option to set kjournald's I/O priority
    jbd2: Submit writes to the journal using WRITE_SYNC
    jbd2: Add pid and journal device name to the "kjournald2 starting" message
    ext4: Add markers for better debuggability
    ext4: Remove code to create the journal inode
    ext4: provide function to release metadata pages under memory pressure
    ext3: provide function to release metadata pages under memory pressure
    add releasepage hooks to block devices which can be used by file systems
    ext4: Fix s_dirty_blocks_counter if block allocation failed with nodelalloc
    ext4: Init the complete page while building buddy cache
    ext4: Don't allow new groups to be added during block allocation
    ext4: mark the blocks/inode bitmap beyond end of group as used
    ext4: Use new buffer_head flag to check uninit group bitmaps initialization
    ext4: Fix the race between read_inode_bitmap() and ext4_new_inode()
    ext4: code cleanup
    ...

    Linus Torvalds
     

07 Jan, 2009

1 commit


06 Jan, 2009

2 commits

  • Filesystems often to do compute intensive operation on some
    metadata. If this operation is repeated many times, it can be very
    expensive. It would be much nicer if the operation could be performed
    once before a buffer goes to disk.

    This adds triggers to jbd2 buffer heads. Just before writing a metadata
    buffer to the journal, jbd2 will optionally call a commit trigger associated
    with the buffer. If the journal is aborted, an abort trigger will be
    called on any dirty buffers as they are dropped from pending
    transactions.

    ocfs2 will use this feature.

    Initially I tried to come up with a more generic trigger that could be
    used for non-buffer-related events like transaction completion. It
    doesn't tie nicely, because the information a buffer trigger needs
    (specific to a journal_head) isn't the same as what a transaction
    trigger needs (specific to a tranaction_t or perhaps journal_t). So I
    implemented a buffer set, with the understanding that
    journal/transaction wide triggers should be implemented separately.

    There is only one trigger set allowed per buffer. I can't think of any
    reason to attach more than one set. Contrast this with a journal or
    transaction in which multiple places may want to watch the entire
    transaction separately.

    The trigger sets are considered static allocation from the jbd2
    perspective. ocfs2 will just have one trigger set per block type,
    setting the same set on every bh of the same type.

    Signed-off-by: Joel Becker
    Cc: "Theodore Ts'o"
    Cc:
    Signed-off-by: Mark Fasheh

    Joel Becker
     
  • Add this so that file systems using JBD2 can safely allocate unused b_state
    bits.

    In this case, we add it so that Ocfs2 can define a single bit for tracking
    the validation state of a buffer.

    Acked-by: "Theodore Ts'o"
    Signed-off-by: Mark Fasheh

    Mark Fasheh
     

04 Jan, 2009

1 commit

  • Add new mount options, min_batch_time and max_batch_time, which
    controls how long the jbd2 layer should wait for additional filesystem
    operations to get batched with a synchronous write transaction.

    Signed-off-by: "Theodore Ts'o"

    Theodore Ts'o
     

26 Nov, 2008

2 commits

  • This patch removes the static sleep time in favor of a more self
    optimizing approach where we measure the average amount of time it
    takes to commit a transaction to disk and the ammount of time a
    transaction has been running. If somebody does a sync write or an
    fsync() traditionally we would sleep for 1 jiffies, which depending on
    the value of HZ could be a significant amount of time compared to how
    long it takes to commit a transaction to the underlying storage. With
    this patch instead of sleeping for a jiffie, we check to see if the
    amount of time this transaction has been running is less than the
    average commit time, and if it is we sleep for the delta using
    schedule_hrtimeout to give us a higher precision sleep time. This
    greatly benefits high end storage where you could end up sleeping for
    longer than it takes to commit the transaction and therefore sitting
    idle instead of allowing the transaction to be committed by keeping
    the sleep time to a minimum so you are sure to always be doing
    something.

    Signed-off-by: Josef Bacik
    Signed-off-by: "Theodore Ts'o"

    Josef Bacik
     
  • Add this so that file systems using JBD2 can safely allocate unused b_state
    bits.

    In this case, we add it so that Ocfs2 can define a single bit for tracking
    the validation state of a buffer.

    Signed-off-by: Mark Fasheh
    Signed-off-by: "Theodore Ts'o"

    Mark Fasheh
     

07 Nov, 2008

1 commit


05 Nov, 2008

1 commit


17 Oct, 2008

2 commits


11 Oct, 2008

2 commits

  • If the journal doesn't abort when it gets an IO error in file data
    blocks, the file data corruption will spread silently. Because
    most of applications and commands do buffered writes without fsync(),
    they don't notice the IO error. It's scary for mission critical
    systems. On the other hand, if the journal aborts whenever it gets
    an IO error in file data blocks, the system will easily become
    inoperable. So this patch introduces a filesystem option to
    determine whether it aborts the journal or just call printk() when
    it gets an IO error in file data.

    If you mount an ext4 fs with data_err=abort option, it aborts on file
    data write error. If you mount it with data_err=ignore, it doesn't
    abort, just call printk(). data_err=ignore is the default.

    Here is the corresponding patch of the ext3 version:
    http://kerneltrap.org/mailarchive/linux-kernel/2008/9/9/3239374

    Signed-off-by: Hidehiro Kawai
    Signed-off-by: Theodore Ts'o

    Hidehiro Kawai
     
  • When a checkpointing IO fails, current JBD2 code doesn't check the
    error and continue journaling. This means latest metadata can be
    lost from both the journal and filesystem.

    This patch leaves the failed metadata blocks in the journal space
    and aborts journaling in the case of jbd2_log_do_checkpoint().
    To achieve this, we need to do:

    1. don't remove the failed buffer from the checkpoint list where in
    the case of __try_to_free_cp_buf() because it may be released or
    overwritten by a later transaction
    2. jbd2_log_do_checkpoint() is the last chance, remove the failed
    buffer from the checkpoint list and abort the journal
    3. when checkpointing fails, don't update the journal super block to
    prevent the journaled contents from being cleaned. For safety,
    don't update j_tail and j_tail_sequence either
    4. when checkpointing fails, notify this error to the ext4 layer so
    that ext4 don't clear the needs_recovery flag, otherwise the
    journaled contents are ignored and cleaned in the recovery phase
    5. if the recovery fails, keep the needs_recovery flag
    6. prevent jbd2_cleanup_journal_tail() from being called between
    __jbd2_journal_drop_transaction() and jbd2_journal_abort()
    (a possible race issue between jbd2_log_do_checkpoint()s called by
    jbd2_journal_flush() and __jbd2_log_wait_for_space())

    Signed-off-by: Hidehiro Kawai
    Signed-off-by: Theodore Ts'o

    Hidehiro Kawai
     

17 Sep, 2008

1 commit

  • Calculate the journal device name once and stash it away in the
    journal_s structure. This avoids needing to call bdevname()
    everywhere and reduces stack usage by not needing to allocate an
    on-stack buffer. In addition, we eliminate the '/' that can appear in
    device names (e.g. "cciss/c0d0p9" --- see kernel bugzilla #11321) that
    can cause problems when creating proc directory names, and include the
    inode number to support ocfs2 which creates multiple journals with
    different inode numbers.

    Signed-off-by: "Theodore Ts'o"

    Theodore Ts'o
     

12 Jul, 2008

3 commits


07 Jun, 2008

1 commit


19 Apr, 2008

1 commit


29 Jan, 2008

3 commits

  • Ported from similar patch for the jbd layer.

    Signed-off-by: Mingming Cao
    Signed-off-by: "Theodore Ts'o"

    Mingming Cao
     
  • The journal checksum feature adds two new flags i.e
    JBD2_FEATURE_INCOMPAT_ASYNC_COMMIT and JBD2_FEATURE_COMPAT_CHECKSUM.

    JBD2_FEATURE_CHECKSUM flag indicates that the commit block contains the
    checksum for the blocks described by the descriptor blocks.
    Due to checksums, writing of the commit record no longer needs to be
    synchronous. Now commit record can be sent to disk without waiting for
    descriptor blocks to be written to disk. This behavior is controlled
    using JBD2_FEATURE_ASYNC_COMMIT flag. Older kernels/e2fsck should not be
    able to recover the journal with _ASYNC_COMMIT hence it is made
    incompat.
    The commit header has been extended to hold the checksum along with the
    type of the checksum.

    For recovery in pass scan checksums are verified to ensure the sanity
    and completeness(in case of _ASYNC_COMMIT) of every transaction.

    Signed-off-by: Andreas Dilger
    Signed-off-by: Girish Shilamkar
    Signed-off-by: Dave Kleikamp
    Signed-off-by: Mingming Cao

    Girish Shilamkar
     
  • The patch below updates the jbd stats patch to 2.6.20/jbd2.
    The initial patch was posted by Alex Tomas in December 2005
    (http://marc.info/?l=linux-ext4&m=113538565128617&w=2).
    It provides statistics via procfs such as transaction lifetime and size.

    Sometimes, investigating performance problems, i find useful to have
    stats from jbd about transaction's lifetime, size, etc. here is a
    patch for review and inclusion probably.

    for example, stats after creation of 3M files in htree directory:

    [root@bob ~]# cat /proc/fs/jbd/sda/history
    R/C tid wait run lock flush log hndls block inlog ctime write drop close
    R 261 8260 2720 0 0 750 9892 8170 8187
    C 259 750 0 4885 1
    R 262 20 2200 10 0 770 9836 8170 8187
    R 263 30 2200 10 0 3070 9812 8170 8187
    R 264 0 5000 10 0 1340 0 0 0
    C 261 8240 3212 4957 0
    R 265 8260 1470 0 0 4640 9854 8170 8187
    R 266 0 5000 10 0 1460 0 0 0
    C 262 8210 2989 4868 0
    R 267 8230 1490 10 0 4440 9875 8171 8188
    R 268 0 5000 10 0 1260 0 0 0
    C 263 7710 2937 4908 0
    R 269 7730 1470 10 0 3330 9841 8170 8187
    R 270 0 5000 10 0 830 0 0 0
    C 265 8140 3234 4898 0
    C 267 720 0 4849 1
    R 271 8630 2740 20 0 740 9819 8170 8187
    C 269 800 0 4214 1
    R 272 40 2170 10 0 830 9716 8170 8187
    R 273 40 2280 0 0 3530 9799 8170 8187
    R 274 0 5000 10 0 990 0 0 0

    where,

    R - line for transaction's life from T_RUNNING to T_FINISHED
    C - line for transaction's checkpointing
    tid - transaction's id
    wait - for how long we were waiting for new transaction to start
    (the longest period journal_start() took in this transaction)
    run - real transaction's lifetime (from T_RUNNING to T_LOCKED
    lock - how long we were waiting for all handles to close
    (time the transaction was in T_LOCKED)
    flush - how long it took to flush all data (data=ordered)
    log - how long it took to write the transaction to the log
    hndls - how many handles got to the transaction
    block - how many blocks got to the transaction
    inlog - how many blocks are written to the log (block + descriptors)
    ctime - how long it took to checkpoint the transaction
    write - how many blocks have been written during checkpointing
    drop - how many blocks have been dropped during checkpointing
    close - how many running transactions have been closed to checkpoint this one

    all times are in msec.

    [root@bob ~]# cat /proc/fs/jbd/sda/info
    280 transaction, each upto 8192 blocks
    average:
    1633ms waiting for transaction
    3616ms running transaction
    5ms transaction was being locked
    1ms flushing data (in ordered mode)
    1799ms logging transaction
    11781 handles per transaction
    5629 blocks per transaction
    5641 logged blocks per transaction

    Signed-off-by: Johann Lombardi
    Signed-off-by: Mariusz Kozlowski
    Signed-off-by: Mingming Cao
    Signed-off-by: Eric Sandeen

    Johann Lombardi