Eric Lee / linux-smarc-t335x-v3.2

Commit ec8c0446b6e2b67b5c8813eb517f4bf00efa99a9

Authored by Ralf Baechle 2006-12-13 01:14:57 +0800

Committed by Linus Torvalds 2006-12-14 01:27:08 +0800

Exists in master and in 4 other branches

[PATCH] Optimize D-cache alias handling on fork

Virtually index, physically tagged cache architectures can get away
without cache flushing when forking.  This patch adds a new cache
flushing function flush_cache_dup_mm(struct mm_struct *) which for the
moment I've implemented to do the same thing on all architectures
except on MIPS where it's a no-op.

Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>

Showing 27 changed files with 54 additions and 7 deletions Inline Diff

Documentation/cachetlb.txt
include/asm-alpha/cacheflush.h
include/asm-arm/cacheflush.h
include/asm-arm26/cacheflush.h
include/asm-avr32/cacheflush.h
include/asm-cris/cacheflush.h
include/asm-frv/cacheflush.h
include/asm-h8300/cacheflush.h
include/asm-i386/cacheflush.h
include/asm-ia64/cacheflush.h
include/asm-m32r/cacheflush.h
include/asm-m68k/cacheflush.h
include/asm-m68knommu/cacheflush.h
include/asm-mips/cacheflush.h
include/asm-parisc/cacheflush.h
include/asm-powerpc/cacheflush.h
include/asm-s390/cacheflush.h
include/asm-sh/cpu-sh2/cacheflush.h
include/asm-sh/cpu-sh3/cacheflush.h
include/asm-sh/cpu-sh4/cacheflush.h
include/asm-sh64/cacheflush.h
include/asm-sparc/cacheflush.h
include/asm-sparc64/cacheflush.h
include/asm-v850/cacheflush.h
include/asm-x86_64/cacheflush.h
include/asm-xtensa/cacheflush.h
kernel/fork.c

Documentation/cachetlb.txt

Diff comments View file @ ec8c044

 		Cache and TLB Flushing
 		     Under Linux
 	    David S. Miller <davem@redhat.com>
 This document describes the cache/tlb flushing interfaces called
 by the Linux VM subsystem.  It enumerates over each interface,
 describes it's intended purpose, and what side effect is expected
 after the interface is invoked.
 The side effects described below are stated for a uniprocessor
 implementation, and what is to happen on that single processor.  The
 SMP cases are a simple extension, in that you just extend the
 definition such that the side effect for a particular interface occurs
 on all processors in the system.  Don't let this scare you into
 thinking SMP cache/tlb flushing must be so inefficient, this is in
 fact an area where many optimizations are possible.  For example,
 if it can be proven that a user address space has never executed
 on a cpu (see vma->cpu_vm_mask), one need not perform a flush
 for this address space on that cpu.
 First, the TLB flushing interfaces, since they are the simplest.  The
 "TLB" is abstracted under Linux as something the cpu uses to cache
 virtual-->physical address translations obtained from the software
 page tables.  Meaning that if the software page tables change, it is
 possible for stale translations to exist in this "TLB" cache.
 Therefore when software page table changes occur, the kernel will
 invoke one of the following flush methods _after_ the page table
 changes occur:
 1) void flush_tlb_all(void)
 	The most severe flush of all.  After this interface runs,
 	any previous page table modification whatsoever will be
 	visible to the cpu.
 	This is usually invoked when the kernel page tables are
 	changed, since such translations are "global" in nature.
 2) void flush_tlb_mm(struct mm_struct *mm)
 	This interface flushes an entire user address space from
 	the TLB.  After running, this interface must make sure that
 	any previous page table modifications for the address space
 	'mm' will be visible to the cpu.  That is, after running,
 	there will be no entries in the TLB for 'mm'.
 	This interface is used to handle whole address space
 	page table operations such as what happens during
 	fork, and exec.
 3) void flush_tlb_range(struct vm_area_struct *vma,
 			unsigned long start, unsigned long end)
 	Here we are flushing a specific range of (user) virtual
 	address translations from the TLB.  After running, this
 	interface must make sure that any previous page table
 	modifications for the address space 'vma->vm_mm' in the range
 	'start' to 'end-1' will be visible to the cpu.  That is, after
 	running, here will be no entries in the TLB for 'mm' for
 	virtual addresses in the range 'start' to 'end-1'.
 	The "vma" is the backing store being used for the region.
 	Primarily, this is used for munmap() type operations.
 	The interface is provided in hopes that the port can find
 	a suitably efficient method for removing multiple page
 	sized translations from the TLB, instead of having the kernel
 	call flush_tlb_page (see below) for each entry which may be
 	modified.
 4) void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr)
 	This time we need to remove the PAGE_SIZE sized translation
 	from the TLB.  The 'vma' is the backing structure used by
 	Linux to keep track of mmap'd regions for a process, the
 	address space is available via vma->vm_mm.  Also, one may
 	test (vma->vm_flags & VM_EXEC) to see if this region is
 	executable (and thus could be in the 'instruction TLB' in
 	split-tlb type setups).
 	After running, this interface must make sure that any previous
 	page table modification for address space 'vma->vm_mm' for
 	user virtual address 'addr' will be visible to the cpu.  That
 	is, after running, there will be no entries in the TLB for
 	'vma->vm_mm' for virtual address 'addr'.
 	This is used primarily during fault processing.
 5) void flush_tlb_pgtables(struct mm_struct *mm,
 			   unsigned long start, unsigned long end)
    The software page tables for address space 'mm' for virtual
    addresses in the range 'start' to 'end-1' are being torn down.
    Some platforms cache the lowest level of the software page tables
    in a linear virtually mapped array, to make TLB miss processing
    more efficient.  On such platforms, since the TLB is caching the
    software page table structure, it needs to be flushed when parts
    of the software page table tree are unlinked/freed.
    Sparc64 is one example of a platform which does this.
    Usually, when munmap()'ing an area of user virtual address
    space, the kernel leaves the page table parts around and just
    marks the individual pte's as invalid.  However, if very large
    portions of the address space are unmapped, the kernel frees up
    those portions of the software page tables to prevent potential
    excessive kernel memory usage caused by erratic mmap/mmunmap
    sequences.  It is at these times that flush_tlb_pgtables will
    be invoked.
 6) void update_mmu_cache(struct vm_area_struct *vma,
 			 unsigned long address, pte_t pte)
 	At the end of every page fault, this routine is invoked to
 	tell the architecture specific code that a translation
 	described by "pte" now exists at virtual address "address"
 	for address space "vma->vm_mm", in the software page tables.
 	A port may use this information in any way it so chooses.
 	For example, it could use this event to pre-load TLB
 	translations for software managed TLB configurations.
 	The sparc64 port currently does this.
 7) void tlb_migrate_finish(struct mm_struct *mm)
 	This interface is called at the end of an explicit
 	process migration. This interface provides a hook
 	to allow a platform to update TLB or context-specific
 	information for the address space.
 	The ia64 sn2 platform is one example of a platform
 	that uses this interface.
 8) void lazy_mmu_prot_update(pte_t pte)
 	This interface is called whenever the protection on
 	any user PTEs change.  This interface provides a notification
 	to architecture specific code to take appropriate action.
 Next, we have the cache flushing interfaces.  In general, when Linux
 is changing an existing virtual-->physical mapping to a new value,
 the sequence will be in one of the following forms:
 	1) flush_cache_mm(mm);
 	   change_all_page_tables_of(mm);
 	   flush_tlb_mm(mm);
 	2) flush_cache_range(vma, start, end);
 	   change_range_of_page_tables(mm, start, end);
 	   flush_tlb_range(vma, start, end);
 	3) flush_cache_page(vma, addr, pfn);
 	   set_pte(pte_pointer, new_pte_val);
 	   flush_tlb_page(vma, addr);
 The cache level flush will always be first, because this allows
 us to properly handle systems whose caches are strict and require
 a virtual-->physical translation to exist for a virtual address
 when that virtual address is flushed from the cache.  The HyperSparc
 cpu is one such cpu with this attribute.
 The cache flushing routines below need only deal with cache flushing
 to the extent that it is necessary for a particular cpu.  Mostly,
 these routines must be implemented for cpus which have virtually
 indexed caches which must be flushed when virtual-->physical
 translations are changed or removed.  So, for example, the physically
 indexed physically tagged caches of IA32 processors have no need to
 implement these interfaces since the caches are fully synchronized
 and have no dependency on translation information.
 Here are the routines, one by one:
 1) void flush_cache_mm(struct mm_struct *mm)
 	This interface flushes an entire user address space from
 	the caches.  That is, after running, there will be no cache
 	lines associated with 'mm'.
 	This interface is used to handle whole address space
-	page table operations such as what happens during
+	page table operations such as what happens during exit and exec.
-	fork, exit, and exec.
-2) void flush_cache_range(struct vm_area_struct *vma,
+2) void flush_cache_dup_mm(struct mm_struct *mm)
+	This interface flushes an entire user address space from
+	the caches.  That is, after running, there will be no cache
+	lines associated with 'mm'.
+	This interface is used to handle whole address space
+	page table operations such as what happens during fork.
+	This option is separate from flush_cache_mm to allow some
+	optimizations for VIPT caches.
+3) void flush_cache_range(struct vm_area_struct *vma,
 			  unsigned long start, unsigned long end)
 	Here we are flushing a specific range of (user) virtual
 	addresses from the cache.  After running, there will be no
 	entries in the cache for 'vma->vm_mm' for virtual addresses in
 	the range 'start' to 'end-1'.
 	The "vma" is the backing store being used for the region.
 	Primarily, this is used for munmap() type operations.
 	The interface is provided in hopes that the port can find
 	a suitably efficient method for removing multiple page
 	sized regions from the cache, instead of having the kernel
 	call flush_cache_page (see below) for each entry which may be
 	modified.
-3) void flush_cache_page(struct vm_area_struct *vma, unsigned long addr, unsigned long pfn)
+4) void flush_cache_page(struct vm_area_struct *vma, unsigned long addr, unsigned long pfn)
 	This time we need to remove a PAGE_SIZE sized range
 	from the cache.  The 'vma' is the backing structure used by
 	Linux to keep track of mmap'd regions for a process, the
 	address space is available via vma->vm_mm.  Also, one may
 	test (vma->vm_flags & VM_EXEC) to see if this region is
 	executable (and thus could be in the 'instruction cache' in
 	"Harvard" type cache layouts).
 	The 'pfn' indicates the physical page frame (shift this value
 	left by PAGE_SHIFT to get the physical address) that 'addr'
 	translates to.  It is this mapping which should be removed from
 	the cache.
 	After running, there will be no entries in the cache for
 	'vma->vm_mm' for virtual address 'addr' which translates
 	to 'pfn'.
 	This is used primarily during fault processing.
-4) void flush_cache_kmaps(void)
+5) void flush_cache_kmaps(void)
 	This routine need only be implemented if the platform utilizes
 	highmem.  It will be called right before all of the kmaps
 	are invalidated.
 	After running, there will be no entries in the cache for
 	the kernel virtual address range PKMAP_ADDR(0) to
 	PKMAP_ADDR(LAST_PKMAP).
 	This routing should be implemented in asm/highmem.h
-5) void flush_cache_vmap(unsigned long start, unsigned long end)
+6) void flush_cache_vmap(unsigned long start, unsigned long end)
    void flush_cache_vunmap(unsigned long start, unsigned long end)
 	Here in these two interfaces we are flushing a specific range
 	of (kernel) virtual addresses from the cache.  After running,
 	there will be no entries in the cache for the kernel address
 	space for virtual addresses in the range 'start' to 'end-1'.
 	The first of these two routines is invoked after map_vm_area()
 	has installed the page table entries.  The second is invoked
 	before unmap_vm_area() deletes the page table entries.
 There exists another whole class of cpu cache issues which currently
 require a whole different set of interfaces to handle properly.
 The biggest problem is that of virtual aliasing in the data cache
 of a processor.
 Is your port susceptible to virtual aliasing in it's D-cache?
 Well, if your D-cache is virtually indexed, is larger in size than
 PAGE_SIZE, and does not prevent multiple cache lines for the same
 physical address from existing at once, you have this problem.
 If your D-cache has this problem, first define asm/shmparam.h SHMLBA
 properly, it should essentially be the size of your virtually
 addressed D-cache (or if the size is variable, the largest possible
 size).  This setting will force the SYSv IPC layer to only allow user
 processes to mmap shared memory at address which are a multiple of
 this value.
 NOTE: This does not fix shared mmaps, check out the sparc64 port for
 one way to solve this (in particular SPARC_FLAG_MMAPSHARED).
 Next, you have to solve the D-cache aliasing issue for all
 other cases.  Please keep in mind that fact that, for a given page
 mapped into some user address space, there is always at least one more
 mapping, that of the kernel in it's linear mapping starting at
 PAGE_OFFSET.  So immediately, once the first user maps a given
 physical page into its address space, by implication the D-cache
 aliasing problem has the potential to exist since the kernel already
 maps this page at its virtual address.
   void copy_user_page(void *to, void *from, unsigned long addr, struct page *page)
   void clear_user_page(void *to, unsigned long addr, struct page *page)
 	These two routines store data in user anonymous or COW
 	pages.  It allows a port to efficiently avoid D-cache alias
 	issues between userspace and the kernel.
 	For example, a port may temporarily map 'from' and 'to' to
 	kernel virtual addresses during the copy.  The virtual address
 	for these two pages is chosen in such a way that the kernel
 	load/store instructions happen to virtual addresses which are
 	of the same "color" as the user mapping of the page.  Sparc64
 	for example, uses this technique.
 	The 'addr' parameter tells the virtual address where the
 	user will ultimately have this page mapped, and the 'page'
 	parameter gives a pointer to the struct page of the target.
 	If D-cache aliasing is not an issue, these two routines may
 	simply call memcpy/memset directly and do nothing more.
   void flush_dcache_page(struct page *page)
 	Any time the kernel writes to a page cache page, _OR_
 	the kernel is about to read from a page cache page and
 	user space shared/writable mappings of this page potentially
 	exist, this routine is called.
 	NOTE: This routine need only be called for page cache pages
 	      which can potentially ever be mapped into the address
 	      space of a user process.  So for example, VFS layer code
 	      handling vfs symlinks in the page cache need not call
 	      this interface at all.
 	The phrase "kernel writes to a page cache page" means,
 	specifically, that the kernel executes store instructions
 	that dirty data in that page at the page->virtual mapping
 	of that page.  It is important to flush here to handle
 	D-cache aliasing, to make sure these kernel stores are
 	visible to user space mappings of that page.
 	The corollary case is just as important, if there are users
 	which have shared+writable mappings of this file, we must make
 	sure that kernel reads of these pages will see the most recent
 	stores done by the user.
 	If D-cache aliasing is not an issue, this routine may
 	simply be defined as a nop on that architecture.
         There is a bit set aside in page->flags (PG_arch_1) as
 	"architecture private".  The kernel guarantees that,
 	for pagecache pages, it will clear this bit when such
 	a page first enters the pagecache.
 	This allows these interfaces to be implemented much more
 	efficiently.  It allows one to "defer" (perhaps indefinitely)
 	the actual flush if there are currently no user processes
 	mapping this page.  See sparc64's flush_dcache_page and
 	update_mmu_cache implementations for an example of how to go
 	about doing this.
 	The idea is, first at flush_dcache_page() time, if
 	page->mapping->i_mmap is an empty tree and ->i_mmap_nonlinear
 	an empty list, just mark the architecture private page flag bit.
 	Later, in update_mmu_cache(), a check is made of this flag bit,
 	and if set the flush is done and the flag bit is cleared.
 	IMPORTANT NOTE: It is often important, if you defer the flush,
 			that the actual flush occurs on the same CPU
 			as did the cpu stores into the page to make it
 			dirty.  Again, see sparc64 for examples of how
 			to deal with this.
   void copy_to_user_page(struct vm_area_struct *vma, struct page *page,
                          unsigned long user_vaddr,
                          void *dst, void *src, int len)
   void copy_from_user_page(struct vm_area_struct *vma, struct page *page,
                            unsigned long user_vaddr,
                            void *dst, void *src, int len)
 	When the kernel needs to copy arbitrary data in and out
 	of arbitrary user pages (f.e. for ptrace()) it will use
 	these two routines.
 	Any necessary cache flushing or other coherency operations
 	that need to occur should happen here.  If the processor's
 	instruction cache does not snoop cpu stores, it is very
 	likely that you will need to flush the instruction cache
 	for copy_to_user_page().
   void flush_anon_page(struct page *page, unsigned long vmaddr)
   	When the kernel needs to access the contents of an anonymous
 	page, it calls this function (currently only
 	get_user_pages()).  Note: flush_dcache_page() deliberately
 	doesn't work for an anonymous page.  The default
 	implementation is a nop (and should remain so for all coherent
 	architectures).  For incoherent architectures, it should flush
 	the cache of the page at vmaddr in the current user process.
   void flush_kernel_dcache_page(struct page *page)
 	When the kernel needs to modify a user page is has obtained
 	with kmap, it calls this function after all modifications are
 	complete (but before kunmapping it) to bring the underlying
 	page up to date.  It is assumed here that the user has no
 	incoherent cached copies (i.e. the original page was obtained
 	from a mechanism like get_user_pages()).  The default
 	implementation is a nop and should remain so on all coherent
 	architectures.  On incoherent architectures, this should flush
 	the kernel cache for page (using page_address(page)).
   void flush_icache_range(unsigned long start, unsigned long end)
   	When the kernel stores into addresses that it will execute
 	out of (eg when loading modules), this function is called.
 	If the icache does not snoop stores then this routine will need
 	to flush it.
   void flush_icache_page(struct vm_area_struct *vma, struct page *page)
 	All the functionality of flush_icache_page can be implemented in
 	flush_dcache_page and update_mmu_cache. In 2.7 the hope is to
 	remove this interface completely.

include/asm-alpha/cacheflush.h

Diff comments View file @ ec8c044

 #ifndef _ALPHA_CACHEFLUSH_H
 #define _ALPHA_CACHEFLUSH_H
 #include <linux/mm.h>
 /* Caches aren't brain-dead on the Alpha. */
 #define flush_cache_all()			do { } while (0)
 #define flush_cache_mm(mm)			do { } while (0)
+#define flush_cache_dup_mm(mm)			do { } while (0)
 #define flush_cache_range(vma, start, end)	do { } while (0)
 #define flush_cache_page(vma, vmaddr, pfn)	do { } while (0)
 #define flush_dcache_page(page)			do { } while (0)
 #define flush_dcache_mmap_lock(mapping)		do { } while (0)
 #define flush_dcache_mmap_unlock(mapping)	do { } while (0)
 #define flush_cache_vmap(start, end)		do { } while (0)
 #define flush_cache_vunmap(start, end)		do { } while (0)
 /* Note that the following two definitions are _highly_ dependent
    on the contexts in which they are used in the kernel.  I personally
    think it is criminal how loosely defined these macros are.  */
 /* We need to flush the kernel's icache after loading modules.  The
    only other use of this macro is in load_aout_interp which is not
    used on Alpha.
    Note that this definition should *not* be used for userspace
    icache flushing.  While functional, it is _way_ overkill.  The
    icache is tagged with ASNs and it suffices to allocate a new ASN
    for the process.  */
 #ifndef CONFIG_SMP
 #define flush_icache_range(start, end)		imb()
 #else
 #define flush_icache_range(start, end)		smp_imb()
 extern void smp_imb(void);
 #endif
 /* We need to flush the userspace icache after setting breakpoints in
    ptrace.
    Instead of indiscriminately using imb, take advantage of the fact
    that icache entries are tagged with the ASN and load a new mm context.  */
 /* ??? Ought to use this in arch/alpha/kernel/signal.c too.  */
 #ifndef CONFIG_SMP
 extern void __load_new_mm_context(struct mm_struct *);
 static inline void
 flush_icache_user_range(struct vm_area_struct *vma, struct page *page,
 			unsigned long addr, int len)
 {
 	if (vma->vm_flags & VM_EXEC) {
 		struct mm_struct *mm = vma->vm_mm;
 		if (current->active_mm == mm)
 			__load_new_mm_context(mm);
 		else
 			mm->context[smp_processor_id()] = 0;
 	}
 }
 #else
 extern void flush_icache_user_range(struct vm_area_struct *vma,
 		struct page *page, unsigned long addr, int len);
 #endif
 /* This is used only in do_no_page and do_swap_page.  */
 #define flush_icache_page(vma, page) \
   flush_icache_user_range((vma), (page), 0, 0)
 #define copy_to_user_page(vma, page, vaddr, dst, src, len) \
 do { memcpy(dst, src, len); \
      flush_icache_user_range(vma, page, vaddr, len); \
 } while (0)
 #define copy_from_user_page(vma, page, vaddr, dst, src, len) \
 	memcpy(dst, src, len)
 #endif /* _ALPHA_CACHEFLUSH_H */

include/asm-arm/cacheflush.h

Diff comments View file @ ec8c044

 /*
  *  linux/include/asm-arm/cacheflush.h
  *
  *  Copyright (C) 1999-2002 Russell King
  *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU General Public License version 2 as
  * published by the Free Software Foundation.
  */
 #ifndef _ASMARM_CACHEFLUSH_H
 #define _ASMARM_CACHEFLUSH_H
 #include <linux/sched.h>
 #include <linux/mm.h>
 #include <asm/glue.h>
 #include <asm/shmparam.h>
 #define CACHE_COLOUR(vaddr)	((vaddr & (SHMLBA - 1)) >> PAGE_SHIFT)
 /*
  *	Cache Model
  *	===========
  */
 #undef _CACHE
 #undef MULTI_CACHE
 #if defined(CONFIG_CPU_CACHE_V3)
 # ifdef _CACHE
 #  define MULTI_CACHE 1
 # else
 #  define _CACHE v3
 # endif
 #endif
 #if defined(CONFIG_CPU_CACHE_V4)
 # ifdef _CACHE
 #  define MULTI_CACHE 1
 # else
 #  define _CACHE v4
 # endif
 #endif
 #if defined(CONFIG_CPU_ARM920T) || defined(CONFIG_CPU_ARM922T) || \
     defined(CONFIG_CPU_ARM925T) || defined(CONFIG_CPU_ARM1020)
 # define MULTI_CACHE 1
 #endif
 #if defined(CONFIG_CPU_ARM926T)
 # ifdef _CACHE
 #  define MULTI_CACHE 1
 # else
 #  define _CACHE arm926
 # endif
 #endif
 #if defined(CONFIG_CPU_ARM940T)
 # ifdef _CACHE
 #  define MULTI_CACHE 1
 # else
 #  define _CACHE arm940
 # endif
 #endif
 #if defined(CONFIG_CPU_ARM946E)
 # ifdef _CACHE
 #  define MULTI_CACHE 1
 # else
 #  define _CACHE arm946
 # endif
 #endif
 #if defined(CONFIG_CPU_CACHE_V4WB)
 # ifdef _CACHE
 #  define MULTI_CACHE 1
 # else
 #  define _CACHE v4wb
 # endif
 #endif
 #if defined(CONFIG_CPU_XSCALE)
 # ifdef _CACHE
 #  define MULTI_CACHE 1
 # else
 #  define _CACHE xscale
 # endif
 #endif
 #if defined(CONFIG_CPU_XSC3)
 # ifdef _CACHE
 #  define MULTI_CACHE 1
 # else
 #  define _CACHE xsc3
 # endif
 #endif
 #if defined(CONFIG_CPU_V6)
 //# ifdef _CACHE
 #  define MULTI_CACHE 1
 //# else
 //#  define _CACHE v6
 //# endif
 #endif
 #if !defined(_CACHE) && !defined(MULTI_CACHE)
 #error Unknown cache maintainence model
 #endif
 /*
  * This flag is used to indicate that the page pointed to by a pte
  * is dirty and requires cleaning before returning it to the user.
  */
 #define PG_dcache_dirty PG_arch_1
 /*
  *	MM Cache Management
  *	===================
  *
  *	The arch/arm/mm/cache-*.S and arch/arm/mm/proc-*.S files
  *	implement these methods.
  *
  *	Start addresses are inclusive and end addresses are exclusive;
  *	start addresses should be rounded down, end addresses up.
  *
  *	See Documentation/cachetlb.txt for more information.
  *	Please note that the implementation of these, and the required
  *	effects are cache-type (VIVT/VIPT/PIPT) specific.
  *
  *	flush_cache_kern_all()
  *
  *		Unconditionally clean and invalidate the entire cache.
  *
  *	flush_cache_user_mm(mm)
  *
  *		Clean and invalidate all user space cache entries
  *		before a change of page tables.
  *
  *	flush_cache_user_range(start, end, flags)
  *
  *		Clean and invalidate a range of cache entries in the
  *		specified address space before a change of page tables.
  *		- start - user start address (inclusive, page aligned)
  *		- end   - user end address   (exclusive, page aligned)
  *		- flags - vma->vm_flags field
  *
  *	coherent_kern_range(start, end)
  *
  *		Ensure coherency between the Icache and the Dcache in the
  *		region described by start, end.  If you have non-snooping
  *		Harvard caches, you need to implement this function.
  *		- start  - virtual start address
  *		- end    - virtual end address
  *
  *	DMA Cache Coherency
  *	===================
  *
  *	dma_inv_range(start, end)
  *
  *		Invalidate (discard) the specified virtual address range.
  *		May not write back any entries.  If 'start' or 'end'
  *		are not cache line aligned, those lines must be written
  *		back.
  *		- start  - virtual start address
  *		- end    - virtual end address
  *
  *	dma_clean_range(start, end)
  *
  *		Clean (write back) the specified virtual address range.
  *		- start  - virtual start address
  *		- end    - virtual end address
  *
  *	dma_flush_range(start, end)
  *
  *		Clean and invalidate the specified virtual address range.
  *		- start  - virtual start address
  *		- end    - virtual end address
  */
 struct cpu_cache_fns {
 	void (*flush_kern_all)(void);
 	void (*flush_user_all)(void);
 	void (*flush_user_range)(unsigned long, unsigned long, unsigned int);
 	void (*coherent_kern_range)(unsigned long, unsigned long);
 	void (*coherent_user_range)(unsigned long, unsigned long);
 	void (*flush_kern_dcache_page)(void *);
 	void (*dma_inv_range)(unsigned long, unsigned long);
 	void (*dma_clean_range)(unsigned long, unsigned long);
 	void (*dma_flush_range)(unsigned long, unsigned long);
 };
 /*
  * Select the calling method
  */
 #ifdef MULTI_CACHE
 extern struct cpu_cache_fns cpu_cache;
 #define __cpuc_flush_kern_all		cpu_cache.flush_kern_all
 #define __cpuc_flush_user_all		cpu_cache.flush_user_all
 #define __cpuc_flush_user_range		cpu_cache.flush_user_range
 #define __cpuc_coherent_kern_range	cpu_cache.coherent_kern_range
 #define __cpuc_coherent_user_range	cpu_cache.coherent_user_range
 #define __cpuc_flush_dcache_page	cpu_cache.flush_kern_dcache_page
 /*
  * These are private to the dma-mapping API.  Do not use directly.
  * Their sole purpose is to ensure that data held in the cache
  * is visible to DMA, or data written by DMA to system memory is
  * visible to the CPU.
  */
 #define dmac_inv_range			cpu_cache.dma_inv_range
 #define dmac_clean_range		cpu_cache.dma_clean_range
 #define dmac_flush_range		cpu_cache.dma_flush_range
 #else
 #define __cpuc_flush_kern_all		__glue(_CACHE,_flush_kern_cache_all)
 #define __cpuc_flush_user_all		__glue(_CACHE,_flush_user_cache_all)
 #define __cpuc_flush_user_range		__glue(_CACHE,_flush_user_cache_range)
 #define __cpuc_coherent_kern_range	__glue(_CACHE,_coherent_kern_range)
 #define __cpuc_coherent_user_range	__glue(_CACHE,_coherent_user_range)
 #define __cpuc_flush_dcache_page	__glue(_CACHE,_flush_kern_dcache_page)
 extern void __cpuc_flush_kern_all(void);
 extern void __cpuc_flush_user_all(void);
 extern void __cpuc_flush_user_range(unsigned long, unsigned long, unsigned int);
 extern void __cpuc_coherent_kern_range(unsigned long, unsigned long);
 extern void __cpuc_coherent_user_range(unsigned long, unsigned long);
 extern void __cpuc_flush_dcache_page(void *);
 /*
  * These are private to the dma-mapping API.  Do not use directly.
  * Their sole purpose is to ensure that data held in the cache
  * is visible to DMA, or data written by DMA to system memory is
  * visible to the CPU.
  */
 #define dmac_inv_range			__glue(_CACHE,_dma_inv_range)
 #define dmac_clean_range		__glue(_CACHE,_dma_clean_range)
 #define dmac_flush_range		__glue(_CACHE,_dma_flush_range)
 extern void dmac_inv_range(unsigned long, unsigned long);
 extern void dmac_clean_range(unsigned long, unsigned long);
 extern void dmac_flush_range(unsigned long, unsigned long);
 #endif
 /*
  * flush_cache_vmap() is used when creating mappings (eg, via vmap,
  * vmalloc, ioremap etc) in kernel space for pages.  Since the
  * direct-mappings of these pages may contain cached data, we need
  * to do a full cache flush to ensure that writebacks don't corrupt
  * data placed into these pages via the new mappings.
  */
 #define flush_cache_vmap(start, end)		flush_cache_all()
 #define flush_cache_vunmap(start, end)		flush_cache_all()
 /*
  * Copy user data from/to a page which is mapped into a different
  * processes address space.  Really, we want to allow our "user
  * space" model to handle this.
  */
 #define copy_to_user_page(vma, page, vaddr, dst, src, len) \
 	do {							\
 		memcpy(dst, src, len);				\
 		flush_ptrace_access(vma, page, vaddr, dst, len, 1);\
 	} while (0)
 #define copy_from_user_page(vma, page, vaddr, dst, src, len) \
 	do {							\
 		memcpy(dst, src, len);				\
 	} while (0)
 /*
  * Convert calls to our calling convention.
  */
 #define flush_cache_all()		__cpuc_flush_kern_all()
 #ifndef CONFIG_CPU_CACHE_VIPT
 static inline void flush_cache_mm(struct mm_struct *mm)
 {
 	if (cpu_isset(smp_processor_id(), mm->cpu_vm_mask))
 		__cpuc_flush_user_all();
 }
 static inline void
 flush_cache_range(struct vm_area_struct *vma, unsigned long start, unsigned long end)
 {
 	if (cpu_isset(smp_processor_id(), vma->vm_mm->cpu_vm_mask))
 		__cpuc_flush_user_range(start & PAGE_MASK, PAGE_ALIGN(end),
 					vma->vm_flags);
 }
 static inline void
 flush_cache_page(struct vm_area_struct *vma, unsigned long user_addr, unsigned long pfn)
 {
 	if (cpu_isset(smp_processor_id(), vma->vm_mm->cpu_vm_mask)) {
 		unsigned long addr = user_addr & PAGE_MASK;
 		__cpuc_flush_user_range(addr, addr + PAGE_SIZE, vma->vm_flags);
 	}
 }
 static inline void
 flush_ptrace_access(struct vm_area_struct *vma, struct page *page,
 			 unsigned long uaddr, void *kaddr,
 			 unsigned long len, int write)
 {
 	if (cpu_isset(smp_processor_id(), vma->vm_mm->cpu_vm_mask)) {
 		unsigned long addr = (unsigned long)kaddr;
 		__cpuc_coherent_kern_range(addr, addr + len);
 	}
 }
 #else
 extern void flush_cache_mm(struct mm_struct *mm);
 extern void flush_cache_range(struct vm_area_struct *vma, unsigned long start, unsigned long end);
 extern void flush_cache_page(struct vm_area_struct *vma, unsigned long user_addr, unsigned long pfn);
 extern void flush_ptrace_access(struct vm_area_struct *vma, struct page *page,
 				unsigned long uaddr, void *kaddr,
 				unsigned long len, int write);
 #endif
+#define flush_cache_dup_mm(mm) flush_cache_mm(mm)
 /*
  * flush_cache_user_range is used when we want to ensure that the
  * Harvard caches are synchronised for the user space address range.
  * This is used for the ARM private sys_cacheflush system call.
  */
 #define flush_cache_user_range(vma,start,end) \
 	__cpuc_coherent_user_range((start) & PAGE_MASK, PAGE_ALIGN(end))
 /*
  * Perform necessary cache operations to ensure that data previously
  * stored within this range of addresses can be executed by the CPU.
  */
 #define flush_icache_range(s,e)		__cpuc_coherent_kern_range(s,e)
 /*
  * Perform necessary cache operations to ensure that the TLB will
  * see data written in the specified area.
  */
 #define clean_dcache_area(start,size)	cpu_dcache_clean_area(start, size)
 /*
  * flush_dcache_page is used when the kernel has written to the page
  * cache page at virtual address page->virtual.
  *
  * If this page isn't mapped (ie, page_mapping == NULL), or it might
  * have userspace mappings, then we _must_ always clean + invalidate
  * the dcache entries associated with the kernel mapping.
  *
  * Otherwise we can defer the operation, and clean the cache when we are
  * about to change to user space.  This is the same method as used on SPARC64.
  * See update_mmu_cache for the user space part.
  */
 extern void flush_dcache_page(struct page *);
 #define flush_dcache_mmap_lock(mapping) \
 	write_lock_irq(&(mapping)->tree_lock)
 #define flush_dcache_mmap_unlock(mapping) \
 	write_unlock_irq(&(mapping)->tree_lock)
 #define flush_icache_user_range(vma,page,addr,len) \
 	flush_dcache_page(page)
 /*
  * We don't appear to need to do anything here.  In fact, if we did, we'd
  * duplicate cache flushing elsewhere performed by flush_dcache_page().
  */
 #define flush_icache_page(vma,page)	do { } while (0)
 #define __cacheid_present(val)		(val != read_cpuid(CPUID_ID))
 #define __cacheid_vivt(val)		((val & (15 << 25)) != (14 << 25))
 #define __cacheid_vipt(val)		((val & (15 << 25)) == (14 << 25))
 #define __cacheid_vipt_nonaliasing(val)	((val & (15 << 25 | 1 << 23)) == (14 << 25))
 #define __cacheid_vipt_aliasing(val)	((val & (15 << 25 | 1 << 23)) == (14 << 25 | 1 << 23))
 #if defined(CONFIG_CPU_CACHE_VIVT) && !defined(CONFIG_CPU_CACHE_VIPT)
 #define cache_is_vivt()			1
 #define cache_is_vipt()			0
 #define cache_is_vipt_nonaliasing()	0
 #define cache_is_vipt_aliasing()	0
 #elif defined(CONFIG_CPU_CACHE_VIPT)
 #define cache_is_vivt()			0
 #define cache_is_vipt()			1
 #define cache_is_vipt_nonaliasing()					\
 	({								\
 		unsigned int __val = read_cpuid(CPUID_CACHETYPE);	\
 		__cacheid_vipt_nonaliasing(__val);			\
 	})
 #define cache_is_vipt_aliasing()					\
 	({								\
 		unsigned int __val = read_cpuid(CPUID_CACHETYPE);	\
 		__cacheid_vipt_aliasing(__val);				\
 	})
 #else
 #define cache_is_vivt()							\
 	({								\
 		unsigned int __val = read_cpuid(CPUID_CACHETYPE);	\
 		(!__cacheid_present(__val)) || __cacheid_vivt(__val);	\
 	})
 #define cache_is_vipt()							\
 	({								\
 		unsigned int __val = read_cpuid(CPUID_CACHETYPE);	\
 		__cacheid_present(__val) && __cacheid_vipt(__val);	\
 	})
 #define cache_is_vipt_nonaliasing()					\
 	({								\
 		unsigned int __val = read_cpuid(CPUID_CACHETYPE);	\
 		__cacheid_present(__val) &&				\
 		 __cacheid_vipt_nonaliasing(__val);			\
 	})
 #define cache_is_vipt_aliasing()					\
 	({								\
 		unsigned int __val = read_cpuid(CPUID_CACHETYPE);	\
 		__cacheid_present(__val) &&				\
 		 __cacheid_vipt_aliasing(__val);			\
 	})
 #endif
 #endif

include/asm-arm26/cacheflush.h

Diff comments View file @ ec8c044

 /*
  *  linux/include/asm-arm/cacheflush.h
  *
  *  Copyright (C) 2000-2002 Russell King
  *  Copyright (C) 2003 Ian Molton
  *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU General Public License version 2 as
  * published by the Free Software Foundation.
  *
  * ARM26 cache 'functions'
  *
  */
 #ifndef _ASMARM_CACHEFLUSH_H
 #define _ASMARM_CACHEFLUSH_H
 #if 1     //FIXME - BAD INCLUDES!!!
 #include <linux/sched.h>
 #include <linux/mm.h>
 #endif
 #define flush_cache_all()                       do { } while (0)
 #define flush_cache_mm(mm)                      do { } while (0)
+#define flush_cache_dup_mm(mm)                  do { } while (0)
 #define flush_cache_range(vma,start,end)        do { } while (0)
 #define flush_cache_page(vma,vmaddr,pfn)        do { } while (0)
 #define flush_cache_vmap(start, end)		do { } while (0)
 #define flush_cache_vunmap(start, end)		do { } while (0)
 #define invalidate_dcache_range(start,end)      do { } while (0)
 #define clean_dcache_range(start,end)           do { } while (0)
 #define flush_dcache_range(start,end)           do { } while (0)
 #define flush_dcache_page(page)                 do { } while (0)
 #define flush_dcache_mmap_lock(mapping)		do { } while (0)
 #define flush_dcache_mmap_unlock(mapping)	do { } while (0)
 #define clean_dcache_entry(_s)                  do { } while (0)
 #define clean_cache_entry(_start)               do { } while (0)
 #define flush_icache_user_range(start,end, bob, fred) do { } while (0)
 #define flush_icache_range(start,end)           do { } while (0)
 #define flush_icache_page(vma,page)             do { } while (0)
 #define copy_to_user_page(vma, page, vaddr, dst, src, len) \
 	memcpy(dst, src, len)
 #define copy_from_user_page(vma, page, vaddr, dst, src, len) \
 	memcpy(dst, src, len)
 /* DAG: ARM3 will flush cache on MEMC updates anyway? so don't bother */
 /* IM : Yes, it will, but only if setup to do so (we do this). */
 #define clean_cache_area(_start,_size)          do { } while (0)
 #endif

include/asm-avr32/cacheflush.h

Diff comments View file @ ec8c044

 /*
  * Copyright (C) 2004-2006 Atmel Corporation
  *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU General Public License version 2 as
  * published by the Free Software Foundation.
  */
 #ifndef __ASM_AVR32_CACHEFLUSH_H
 #define __ASM_AVR32_CACHEFLUSH_H
 /* Keep includes the same across arches.  */
 #include <linux/mm.h>
 #define CACHE_OP_ICACHE_INVALIDATE	0x01
 #define CACHE_OP_DCACHE_INVALIDATE	0x0b
 #define CACHE_OP_DCACHE_CLEAN		0x0c
 #define CACHE_OP_DCACHE_CLEAN_INVAL	0x0d
 /*
  * Invalidate any cacheline containing virtual address vaddr without
  * writing anything back to memory.
  *
  * Note that this function may corrupt unrelated data structures when
  * applied on buffers that are not cacheline aligned in both ends.
  */
 static inline void invalidate_dcache_line(void *vaddr)
 {
 	asm volatile("cache %0[0], %1"
 		     :
 		     : "r"(vaddr), "n"(CACHE_OP_DCACHE_INVALIDATE)
 		     : "memory");
 }
 /*
  * Make sure any cacheline containing virtual address vaddr is written
  * to memory.
  */
 static inline void clean_dcache_line(void *vaddr)
 {
 	asm volatile("cache %0[0], %1"
 		     :
 		     : "r"(vaddr), "n"(CACHE_OP_DCACHE_CLEAN)
 		     : "memory");
 }
 /*
  * Make sure any cacheline containing virtual address vaddr is written
  * to memory and then invalidate it.
  */
 static inline void flush_dcache_line(void *vaddr)
 {
 	asm volatile("cache %0[0], %1"
 		     :
 		     : "r"(vaddr), "n"(CACHE_OP_DCACHE_CLEAN_INVAL)
 		     : "memory");
 }
 /*
  * Invalidate any instruction cacheline containing virtual address
  * vaddr.
  */
 static inline void invalidate_icache_line(void *vaddr)
 {
 	asm volatile("cache %0[0], %1"
 		     :
 		     : "r"(vaddr), "n"(CACHE_OP_ICACHE_INVALIDATE)
 		     : "memory");
 }
 /*
  * Applies the above functions on all lines that are touched by the
  * specified virtual address range.
  */
 void invalidate_dcache_region(void *start, size_t len);
 void clean_dcache_region(void *start, size_t len);
 void flush_dcache_region(void *start, size_t len);
 void invalidate_icache_region(void *start, size_t len);
 /*
  * Make sure any pending writes are completed before continuing.
  */
 #define flush_write_buffer() asm volatile("sync 0" : : : "memory")
 /*
  * The following functions are called when a virtual mapping changes.
  * We do not need to flush anything in this case.
  */
 #define flush_cache_all()			do { } while (0)
 #define flush_cache_mm(mm)			do { } while (0)
+#define flush_cache_dup_mm(mm)			do { } while (0)
 #define flush_cache_range(vma, start, end)	do { } while (0)
 #define flush_cache_page(vma, vmaddr, pfn)	do { } while (0)
 #define flush_cache_vmap(start, end)		do { } while (0)
 #define flush_cache_vunmap(start, end)		do { } while (0)
 /*
  * I think we need to implement this one to be able to reliably
  * execute pages from RAMDISK. However, if we implement the
  * flush_dcache_*() functions, it might not be needed anymore.
  *
  * #define flush_icache_page(vma, page)		do { } while (0)
  */
 extern void flush_icache_page(struct vm_area_struct *vma, struct page *page);
 /*
  * These are (I think) related to D-cache aliasing.  We might need to
  * do something here, but only for certain configurations.  No such
  * configurations exist at this time.
  */
 #define flush_dcache_page(page)			do { } while (0)
 #define flush_dcache_mmap_lock(page)		do { } while (0)
 #define flush_dcache_mmap_unlock(page)		do { } while (0)
 /*
  * These are for I/D cache coherency. In this case, we do need to
  * flush with all configurations.
  */
 extern void flush_icache_range(unsigned long start, unsigned long end);
 extern void flush_icache_user_range(struct vm_area_struct *vma,
 				    struct page *page,
 				    unsigned long addr, int len);
 #define copy_to_user_page(vma, page, vaddr, dst, src, len) do {	\
 	memcpy(dst, src, len);					\
 	flush_icache_user_range(vma, page, vaddr, len);		\
 } while(0)
 #define copy_from_user_page(vma, page, vaddr, dst, src, len)	\
 	memcpy(dst, src, len)
 #endif /* __ASM_AVR32_CACHEFLUSH_H */

include/asm-cris/cacheflush.h

Diff comments View file @ ec8c044

 #ifndef _CRIS_CACHEFLUSH_H
 #define _CRIS_CACHEFLUSH_H
 /* Keep includes the same across arches.  */
 #include <linux/mm.h>
 /* The cache doesn't need to be flushed when TLB entries change because
  * the cache is mapped to physical memory, not virtual memory
  */
 #define flush_cache_all()			do { } while (0)
 #define flush_cache_mm(mm)			do { } while (0)
+#define flush_cache_dup_mm(mm)			do { } while (0)
 #define flush_cache_range(vma, start, end)	do { } while (0)
 #define flush_cache_page(vma, vmaddr, pfn)	do { } while (0)
 #define flush_dcache_page(page)			do { } while (0)
 #define flush_dcache_mmap_lock(mapping)		do { } while (0)
 #define flush_dcache_mmap_unlock(mapping)	do { } while (0)
 #define flush_icache_range(start, end)		do { } while (0)
 #define flush_icache_page(vma,pg)		do { } while (0)
 #define flush_icache_user_range(vma,pg,adr,len)	do { } while (0)
 #define flush_cache_vmap(start, end)		do { } while (0)
 #define flush_cache_vunmap(start, end)		do { } while (0)
 #define copy_to_user_page(vma, page, vaddr, dst, src, len) \
 	memcpy(dst, src, len)
 #define copy_from_user_page(vma, page, vaddr, dst, src, len) \
 	memcpy(dst, src, len)
 void global_flush_tlb(void);
 int change_page_attr(struct page *page, int numpages, pgprot_t prot);
 #endif /* _CRIS_CACHEFLUSH_H */

include/asm-frv/cacheflush.h

Diff comments View file @ ec8c044

 /* cacheflush.h: FRV cache flushing routines
  *
  * Copyright (C) 2004 Red Hat, Inc. All Rights Reserved.
  * Written by David Howells (dhowells@redhat.com)
  *
  * This program is free software; you can redistribute it and/or
  * modify it under the terms of the GNU General Public License
  * as published by the Free Software Foundation; either version
  * 2 of the License, or (at your option) any later version.
  */
 #ifndef _ASM_CACHEFLUSH_H
 #define _ASM_CACHEFLUSH_H
 /* Keep includes the same across arches.  */
 #include <linux/mm.h>
 /*
  * virtually-indexed cache management (our cache is physically indexed)
  */
 #define flush_cache_all()			do {} while(0)
 #define flush_cache_mm(mm)			do {} while(0)
+#define flush_cache_dup_mm(mm)			do {} while(0)
 #define flush_cache_range(mm, start, end)	do {} while(0)
 #define flush_cache_page(vma, vmaddr, pfn)	do {} while(0)
 #define flush_cache_vmap(start, end)		do {} while(0)
 #define flush_cache_vunmap(start, end)		do {} while(0)
 #define flush_dcache_mmap_lock(mapping)		do {} while(0)
 #define flush_dcache_mmap_unlock(mapping)	do {} while(0)
 /*
  * physically-indexed cache managment
  * - see arch/frv/lib/cache.S
  */
 extern void frv_dcache_writeback(unsigned long start, unsigned long size);
 extern void frv_cache_invalidate(unsigned long start, unsigned long size);
 extern void frv_icache_invalidate(unsigned long start, unsigned long size);
 extern void frv_cache_wback_inv(unsigned long start, unsigned long size);
 static inline void __flush_cache_all(void)
 {
 	asm volatile("	dcef	@(gr0,gr0),#1	\n"
 		     "	icei	@(gr0,gr0),#1	\n"
 		     "	membar			\n"
 		     : : : "memory"
 		     );
 }
 /* dcache/icache coherency... */
 #ifdef CONFIG_MMU
 extern void flush_dcache_page(struct page *page);
 #else
 static inline void flush_dcache_page(struct page *page)
 {
 	unsigned long addr = page_to_phys(page);
 	frv_dcache_writeback(addr, addr + PAGE_SIZE);
 }
 #endif
 static inline void flush_page_to_ram(struct page *page)
 {
 	flush_dcache_page(page);
 }
 static inline void flush_icache(void)
 {
 	__flush_cache_all();
 }
 static inline void flush_icache_range(unsigned long start, unsigned long end)
 {
 	frv_cache_wback_inv(start, end);
 }
 #ifdef CONFIG_MMU
 extern void flush_icache_user_range(struct vm_area_struct *vma, struct page *page,
 				    unsigned long start, unsigned long len);
 #else
 static inline void flush_icache_user_range(struct vm_area_struct *vma, struct page *page,
 					   unsigned long start, unsigned long len)
 {
 	frv_cache_wback_inv(start, start + len);
 }
 #endif
 static inline void flush_icache_page(struct vm_area_struct *vma, struct page *page)
 {
 	flush_icache_user_range(vma, page, page_to_phys(page), PAGE_SIZE);
 }
 /*
  * permit ptrace to access another process's address space through the icache
  * and the dcache
  */
 #define copy_to_user_page(vma, page, vaddr, dst, src, len)	\
 do {								\
 	memcpy((dst), (src), (len));				\
 	flush_icache_user_range((vma), (page), (vaddr), (len));	\
 } while(0)
 #define copy_from_user_page(vma, page, vaddr, dst, src, len)	\
 	memcpy((dst), (src), (len))
 #endif /* _ASM_CACHEFLUSH_H */

include/asm-h8300/cacheflush.h

Diff comments View file @ ec8c044

 /*
  * (C) Copyright 2002, Yoshinori Sato <ysato@users.sourceforge.jp>
  */
 #ifndef _ASM_H8300_CACHEFLUSH_H
 #define _AMS_H8300_CACHEFLUSH_H
 /*
  * Cache handling functions
  * No Cache memory all dummy functions
  */
 #define flush_cache_all()
 #define	flush_cache_mm(mm)
+#define	flush_cache_dup_mm(mm)		do { } while (0)
 #define	flush_cache_range(vma,a,b)
 #define	flush_cache_page(vma,p,pfn)
 #define	flush_dcache_page(page)
 #define	flush_dcache_mmap_lock(mapping)
 #define	flush_dcache_mmap_unlock(mapping)
 #define	flush_icache()
 #define	flush_icache_page(vma,page)
 #define	flush_icache_range(start,len)
 #define flush_cache_vmap(start, end)
 #define flush_cache_vunmap(start, end)
 #define	cache_push_v(vaddr,len)
 #define	cache_push(paddr,len)
 #define	cache_clear(paddr,len)
 #define	flush_dcache_range(a,b)
 #define	flush_icache_user_range(vma,page,addr,len)
 #define copy_to_user_page(vma, page, vaddr, dst, src, len) \
 	memcpy(dst, src, len)
 #define copy_from_user_page(vma, page, vaddr, dst, src, len) \
 	memcpy(dst, src, len)
 #endif /* _ASM_H8300_CACHEFLUSH_H */

include/asm-i386/cacheflush.h

Diff comments View file @ ec8c044

 #ifndef _I386_CACHEFLUSH_H
 #define _I386_CACHEFLUSH_H
 /* Keep includes the same across arches.  */
 #include <linux/mm.h>
 /* Caches aren't brain-dead on the intel. */
 #define flush_cache_all()			do { } while (0)
 #define flush_cache_mm(mm)			do { } while (0)
+#define flush_cache_dup_mm(mm)			do { } while (0)
 #define flush_cache_range(vma, start, end)	do { } while (0)
 #define flush_cache_page(vma, vmaddr, pfn)	do { } while (0)
 #define flush_dcache_page(page)			do { } while (0)
 #define flush_dcache_mmap_lock(mapping)		do { } while (0)
 #define flush_dcache_mmap_unlock(mapping)	do { } while (0)
 #define flush_icache_range(start, end)		do { } while (0)
 #define flush_icache_page(vma,pg)		do { } while (0)
 #define flush_icache_user_range(vma,pg,adr,len)	do { } while (0)
 #define flush_cache_vmap(start, end)		do { } while (0)
 #define flush_cache_vunmap(start, end)		do { } while (0)
 #define copy_to_user_page(vma, page, vaddr, dst, src, len) \
 	memcpy(dst, src, len)
 #define copy_from_user_page(vma, page, vaddr, dst, src, len) \
 	memcpy(dst, src, len)
 void global_flush_tlb(void);
 int change_page_attr(struct page *page, int numpages, pgprot_t prot);
 #ifdef CONFIG_DEBUG_PAGEALLOC
 /* internal debugging function */
 void kernel_map_pages(struct page *page, int numpages, int enable);
 #endif
 #ifdef CONFIG_DEBUG_RODATA
 void mark_rodata_ro(void);
 #endif
 #endif /* _I386_CACHEFLUSH_H */

include/asm-ia64/cacheflush.h

Diff comments View file @ ec8c044

 #ifndef _ASM_IA64_CACHEFLUSH_H
 #define _ASM_IA64_CACHEFLUSH_H
 /*
  * Copyright (C) 2002 Hewlett-Packard Co
  *	David Mosberger-Tang <davidm@hpl.hp.com>
  */
 #include <linux/page-flags.h>
 #include <asm/bitops.h>
 #include <asm/page.h>
 /*
  * Cache flushing routines.  This is the kind of stuff that can be very expensive, so try
  * to avoid them whenever possible.
  */
 #define flush_cache_all()			do { } while (0)
 #define flush_cache_mm(mm)			do { } while (0)
+#define flush_cache_dup_mm(mm)			do { } while (0)
 #define flush_cache_range(vma, start, end)	do { } while (0)
 #define flush_cache_page(vma, vmaddr, pfn)	do { } while (0)
 #define flush_icache_page(vma,page)		do { } while (0)
 #define flush_cache_vmap(start, end)		do { } while (0)
 #define flush_cache_vunmap(start, end)		do { } while (0)
 #define flush_dcache_page(page)			\
 do {						\
 	clear_bit(PG_arch_1, &(page)->flags);	\
 } while (0)
 #define flush_dcache_mmap_lock(mapping)		do { } while (0)
 #define flush_dcache_mmap_unlock(mapping)	do { } while (0)
 extern void flush_icache_range (unsigned long start, unsigned long end);
 #define flush_icache_user_range(vma, page, user_addr, len)					\
 do {												\
 	unsigned long _addr = (unsigned long) page_address(page) + ((user_addr) & ~PAGE_MASK);	\
 	flush_icache_range(_addr, _addr + (len));						\
 } while (0)
 #define copy_to_user_page(vma, page, vaddr, dst, src, len) \
 do { memcpy(dst, src, len); \
      flush_icache_user_range(vma, page, vaddr, len); \
 } while (0)
 #define copy_from_user_page(vma, page, vaddr, dst, src, len) \
 	memcpy(dst, src, len)
 #endif /* _ASM_IA64_CACHEFLUSH_H */

include/asm-m32r/cacheflush.h

Diff comments View file @ ec8c044

 #ifndef _ASM_M32R_CACHEFLUSH_H
 #define _ASM_M32R_CACHEFLUSH_H
 #include <linux/mm.h>
 extern void _flush_cache_all(void);
 extern void _flush_cache_copyback_all(void);
 #if defined(CONFIG_CHIP_M32700) || defined(CONFIG_CHIP_OPSP) || defined(CONFIG_CHIP_M32104)
 #define flush_cache_all()			do { } while (0)
 #define flush_cache_mm(mm)			do { } while (0)
+#define flush_cache_dup_mm(mm)			do { } while (0)
 #define flush_cache_range(vma, start, end)	do { } while (0)
 #define flush_cache_page(vma, vmaddr, pfn)	do { } while (0)
 #define flush_dcache_page(page)			do { } while (0)
 #define flush_dcache_mmap_lock(mapping)		do { } while (0)
 #define flush_dcache_mmap_unlock(mapping)	do { } while (0)
 #ifndef CONFIG_SMP
 #define flush_icache_range(start, end)		_flush_cache_copyback_all()
 #define flush_icache_page(vma,pg)		_flush_cache_copyback_all()
 #define flush_icache_user_range(vma,pg,adr,len)	_flush_cache_copyback_all()
 #define flush_cache_sigtramp(addr)		_flush_cache_copyback_all()
 #else	/* CONFIG_SMP */
 extern void smp_flush_cache_all(void);
 #define flush_icache_range(start, end)		smp_flush_cache_all()
 #define flush_icache_page(vma,pg)		smp_flush_cache_all()
 #define flush_icache_user_range(vma,pg,adr,len)	smp_flush_cache_all()
 #define flush_cache_sigtramp(addr)		_flush_cache_copyback_all()
 #endif	/* CONFIG_SMP */
 #elif defined(CONFIG_CHIP_M32102)
 #define flush_cache_all()			do { } while (0)
 #define flush_cache_mm(mm)			do { } while (0)
+#define flush_cache_dup_mm(mm)			do { } while (0)
 #define flush_cache_range(vma, start, end)	do { } while (0)
 #define flush_cache_page(vma, vmaddr, pfn)	do { } while (0)
 #define flush_dcache_page(page)			do { } while (0)
 #define flush_dcache_mmap_lock(mapping)		do { } while (0)
 #define flush_dcache_mmap_unlock(mapping)	do { } while (0)
 #define flush_icache_range(start, end)		_flush_cache_all()
 #define flush_icache_page(vma,pg)		_flush_cache_all()
 #define flush_icache_user_range(vma,pg,adr,len)	_flush_cache_all()
 #define flush_cache_sigtramp(addr)		_flush_cache_all()
 #else
 #define flush_cache_all()			do { } while (0)
 #define flush_cache_mm(mm)			do { } while (0)
+#define flush_cache_dup_mm(mm)			do { } while (0)
 #define flush_cache_range(vma, start, end)	do { } while (0)
 #define flush_cache_page(vma, vmaddr, pfn)	do { } while (0)
 #define flush_dcache_page(page)			do { } while (0)
 #define flush_dcache_mmap_lock(mapping)		do { } while (0)
 #define flush_dcache_mmap_unlock(mapping)	do { } while (0)
 #define flush_icache_range(start, end)		do { } while (0)
 #define flush_icache_page(vma,pg)		do { } while (0)
 #define flush_icache_user_range(vma,pg,adr,len)	do { } while (0)
 #define flush_cache_sigtramp(addr)		do { } while (0)
 #endif	/* CONFIG_CHIP_* */
 #define flush_cache_vmap(start, end)	do { } while (0)
 #define flush_cache_vunmap(start, end)	do { } while (0)
 #define copy_to_user_page(vma, page, vaddr, dst, src, len)	\
 do {								\
 	memcpy(dst, src, len);					\
 	flush_icache_user_range(vma, page, vaddr, len);		\
 } while (0)
 #define copy_from_user_page(vma, page, vaddr, dst, src, len)	\
 	memcpy(dst, src, len)
 #endif /* _ASM_M32R_CACHEFLUSH_H */

include/asm-m68k/cacheflush.h

Diff comments View file @ ec8c044

 #ifndef _M68K_CACHEFLUSH_H
 #define _M68K_CACHEFLUSH_H
 #include <linux/mm.h>
 /* cache code */
 #define FLUSH_I_AND_D	(0x00000808)
 #define FLUSH_I		(0x00000008)
 /*
  * Cache handling functions
  */
 static inline void flush_icache(void)
 {
 	if (CPU_IS_040_OR_060)
 		asm volatile (	"nop\n"
 			"	.chip	68040\n"
 			"	cpusha	%bc\n"
 			"	.chip	68k");
 	else {
 		unsigned long tmp;
 		asm volatile (	"movec	%%cacr,%0\n"
 			"	or.w	%1,%0\n"
 			"	movec	%0,%%cacr"
 			: "=&d" (tmp)
 			: "id" (FLUSH_I));
 	}
 }
 /*
  * invalidate the cache for the specified memory range.
  * It starts at the physical address specified for
  * the given number of bytes.
  */
 extern void cache_clear(unsigned long paddr, int len);
 /*
  * push any dirty cache in the specified memory range.
  * It starts at the physical address specified for
  * the given number of bytes.
  */
 extern void cache_push(unsigned long paddr, int len);
 /*
  * push and invalidate pages in the specified user virtual
  * memory range.
  */
 extern void cache_push_v(unsigned long vaddr, int len);
 /* This is needed whenever the virtual mapping of the current
    process changes.  */
 #define __flush_cache_all()					\
 ({								\
 	if (CPU_IS_040_OR_060)					\
 		__asm__ __volatile__("nop\n\t"			\
 				     ".chip 68040\n\t"		\
 				     "cpusha %dc\n\t"		\
 				     ".chip 68k");		\
 	else {							\
 		unsigned long _tmp;				\
 		__asm__ __volatile__("movec %%cacr,%0\n\t"	\
 				     "orw %1,%0\n\t"		\
 				     "movec %0,%%cacr"		\
 				     : "=&d" (_tmp)		\
 				     : "di" (FLUSH_I_AND_D));	\
 	}							\
 })
 #define __flush_cache_030()					\
 ({								\
 	if (CPU_IS_020_OR_030) {				\
 		unsigned long _tmp;				\
 		__asm__ __volatile__("movec %%cacr,%0\n\t"	\
 				     "orw %1,%0\n\t"		\
 				     "movec %0,%%cacr"		\
 				     : "=&d" (_tmp)		\
 				     : "di" (FLUSH_I_AND_D));	\
 	}							\
 })
 #define flush_cache_all() __flush_cache_all()
 #define flush_cache_vmap(start, end)		flush_cache_all()
 #define flush_cache_vunmap(start, end)		flush_cache_all()
 static inline void flush_cache_mm(struct mm_struct *mm)
 {
 	if (mm == current->mm)
 		__flush_cache_030();
 }
+#define flush_cache_dup_mm(mm)			flush_cache_mm(mm)
 /* flush_cache_range/flush_cache_page must be macros to avoid
    a dependency on linux/mm.h, which includes this file... */
 static inline void flush_cache_range(struct vm_area_struct *vma,
 				     unsigned long start,
 				     unsigned long end)
 {
 	if (vma->vm_mm == current->mm)
 	        __flush_cache_030();
 }
 static inline void flush_cache_page(struct vm_area_struct *vma, unsigned long vmaddr, unsigned long pfn)
 {
 	if (vma->vm_mm == current->mm)
 	        __flush_cache_030();
 }
 /* Push the page at kernel virtual address and clear the icache */
 /* RZ: use cpush %bc instead of cpush %dc, cinv %ic */
 static inline void __flush_page_to_ram(void *vaddr)
 {
 	if (CPU_IS_040_OR_060) {
 		__asm__ __volatile__("nop\n\t"
 				     ".chip 68040\n\t"
 				     "cpushp %%bc,(%0)\n\t"
 				     ".chip 68k"
 				     : : "a" (__pa(vaddr)));
 	} else {
 		unsigned long _tmp;
 		__asm__ __volatile__("movec %%cacr,%0\n\t"
 				     "orw %1,%0\n\t"
 				     "movec %0,%%cacr"
 				     : "=&d" (_tmp)
 				     : "di" (FLUSH_I));
 	}
 }
 #define flush_dcache_page(page)		__flush_page_to_ram(page_address(page))
 #define flush_dcache_mmap_lock(mapping)		do { } while (0)
 #define flush_dcache_mmap_unlock(mapping)	do { } while (0)
 #define flush_icache_page(vma, page)	__flush_page_to_ram(page_address(page))
 extern void flush_icache_user_range(struct vm_area_struct *vma, struct page *page,
 				    unsigned long addr, int len);
 extern void flush_icache_range(unsigned long address, unsigned long endaddr);
 static inline void copy_to_user_page(struct vm_area_struct *vma,
 				     struct page *page, unsigned long vaddr,
 				     void *dst, void *src, int len)
 {
 	flush_cache_page(vma, vaddr, page_to_pfn(page));
 	memcpy(dst, src, len);
 	flush_icache_user_range(vma, page, vaddr, len);
 }
 static inline void copy_from_user_page(struct vm_area_struct *vma,
 				       struct page *page, unsigned long vaddr,
 				       void *dst, void *src, int len)
 {
 	flush_cache_page(vma, vaddr, page_to_pfn(page));
 	memcpy(dst, src, len);
 }
 #endif /* _M68K_CACHEFLUSH_H */

include/asm-m68knommu/cacheflush.h

Diff comments View file @ ec8c044

 #ifndef _M68KNOMMU_CACHEFLUSH_H
 #define _M68KNOMMU_CACHEFLUSH_H
 /*
  * (C) Copyright 2000-2004, Greg Ungerer <gerg@snapgear.com>
  */
 #include <linux/mm.h>
 #define flush_cache_all()			__flush_cache_all()
 #define flush_cache_mm(mm)			do { } while (0)
+#define flush_cache_dup_mm(mm)			do { } while (0)
 #define flush_cache_range(vma, start, end)	__flush_cache_all()
 #define flush_cache_page(vma, vmaddr)		do { } while (0)
 #define flush_dcache_range(start,len)		__flush_cache_all()
 #define flush_dcache_page(page)			do { } while (0)
 #define flush_dcache_mmap_lock(mapping)		do { } while (0)
 #define flush_dcache_mmap_unlock(mapping)	do { } while (0)
 #define flush_icache_range(start,len)		__flush_cache_all()
 #define flush_icache_page(vma,pg)		do { } while (0)
 #define flush_icache_user_range(vma,pg,adr,len)	do { } while (0)
 #define flush_cache_vmap(start, end)		do { } while (0)
 #define flush_cache_vunmap(start, end)		do { } while (0)
 #define copy_to_user_page(vma, page, vaddr, dst, src, len) \
 	memcpy(dst, src, len)
 #define copy_from_user_page(vma, page, vaddr, dst, src, len) \
 	memcpy(dst, src, len)
 static inline void __flush_cache_all(void)
 {
 #ifdef CONFIG_M5407
 	/*
 	 *	Use cpushl to push and invalidate all cache lines.
 	 *	Gas doesn't seem to know how to generate the ColdFire
 	 *	cpushl instruction... Oh well, bit stuff it for now.
 	 */
 	__asm__ __volatile__ (
 		"nop\n\t"
 		"clrl	%%d0\n\t"
 		"1:\n\t"
 		"movel	%%d0,%%a0\n\t"
 		"2:\n\t"
 		".word	0xf468\n\t"
 		"addl	#0x10,%%a0\n\t"
 		"cmpl	#0x00000800,%%a0\n\t"
 		"blt	2b\n\t"
 		"addql	#1,%%d0\n\t"
 		"cmpil	#4,%%d0\n\t"
 		"bne	1b\n\t"
 		"movel	#0xb6088500,%%d0\n\t"
 		"movec	%%d0,%%CACR\n\t"
 		: : : "d0", "a0" );
 #endif /* CONFIG_M5407 */
 #if defined(CONFIG_M527x) || defined(CONFIG_M528x)
 	__asm__ __volatile__ (
         	"movel	#0x81400100, %%d0\n\t"
         	"movec	%%d0, %%CACR\n\t"
 		"nop\n\t"
 		: : : "d0" );
 #endif /* CONFIG_M527x || CONFIG_M528x */
 #if defined(CONFIG_M5206) || defined(CONFIG_M5206e) || defined(CONFIG_M5272)
 	__asm__ __volatile__ (
         	"movel	#0x81000100, %%d0\n\t"
         	"movec	%%d0, %%CACR\n\t"
 		"nop\n\t"
 		: : : "d0" );
 #endif /* CONFIG_M5206 || CONFIG_M5206e || CONFIG_M5272 */
 #ifdef CONFIG_M5249
 	__asm__ __volatile__ (
         	"movel	#0xa1000200, %%d0\n\t"
         	"movec	%%d0, %%CACR\n\t"
 		"nop\n\t"
 		: : : "d0" );
 #endif /* CONFIG_M5249 */
 #ifdef CONFIG_M532x
 	__asm__ __volatile__ (
         	"movel	#0x81000200, %%d0\n\t"
         	"movec	%%d0, %%CACR\n\t"
 		"nop\n\t"
 		: : : "d0" );
 #endif /* CONFIG_M532x */
 }
 #endif /* _M68KNOMMU_CACHEFLUSH_H */

include/asm-mips/cacheflush.h

Diff comments View file @ ec8c044

 /*
  * This file is subject to the terms and conditions of the GNU General Public
  * License.  See the file "COPYING" in the main directory of this archive
  * for more details.
  *
  * Copyright (C) 1994, 95, 96, 97, 98, 99, 2000, 01, 02, 03 by Ralf Baechle
  * Copyright (C) 1999, 2000, 2001 Silicon Graphics, Inc.
  */
 #ifndef _ASM_CACHEFLUSH_H
 #define _ASM_CACHEFLUSH_H
 /* Keep includes the same across arches.  */
 #include <linux/mm.h>
 #include <asm/cpu-features.h>
 /* Cache flushing:
  *
  *  - flush_cache_all() flushes entire cache
  *  - flush_cache_mm(mm) flushes the specified mm context's cache lines
+ *  - flush_cache_dup mm(mm) handles cache flushing when forking
  *  - flush_cache_page(mm, vmaddr, pfn) flushes a single page
  *  - flush_cache_range(vma, start, end) flushes a range of pages
  *  - flush_icache_range(start, end) flush a range of instructions
  *  - flush_dcache_page(pg) flushes(wback&invalidates) a page for dcache
  *
  * MIPS specific flush operations:
  *
  *  - flush_cache_sigtramp() flush signal trampoline
  *  - flush_icache_all() flush the entire instruction cache
  *  - flush_data_cache_page() flushes a page from the data cache
  */
 extern void (*flush_cache_all)(void);
 extern void (*__flush_cache_all)(void);
 extern void (*flush_cache_mm)(struct mm_struct *mm);
+#define flush_cache_dup_mm(mm)	do { (void) (mm); } while (0)
 extern void (*flush_cache_range)(struct vm_area_struct *vma,
 	unsigned long start, unsigned long end);
 extern void (*flush_cache_page)(struct vm_area_struct *vma, unsigned long page, unsigned long pfn);
 extern void __flush_dcache_page(struct page *page);
 static inline void flush_dcache_page(struct page *page)
 {
 	if (cpu_has_dc_aliases || !cpu_has_ic_fills_f_dc)
 		__flush_dcache_page(page);
 }
 #define flush_dcache_mmap_lock(mapping)		do { } while (0)
 #define flush_dcache_mmap_unlock(mapping)	do { } while (0)
 static inline void flush_icache_page(struct vm_area_struct *vma,
 	struct page *page)
 {
 }
 extern void (*flush_icache_range)(unsigned long start, unsigned long end);
 #define flush_cache_vmap(start, end)		flush_cache_all()
 #define flush_cache_vunmap(start, end)		flush_cache_all()
 extern void copy_to_user_page(struct vm_area_struct *vma,
 	struct page *page, unsigned long vaddr, void *dst, const void *src,
 	unsigned long len);
 extern void copy_from_user_page(struct vm_area_struct *vma,
 	struct page *page, unsigned long vaddr, void *dst, const void *src,
 	unsigned long len);
 extern void (*flush_cache_sigtramp)(unsigned long addr);
 extern void (*flush_icache_all)(void);
 extern void (*local_flush_data_cache_page)(void * addr);
 extern void (*flush_data_cache_page)(unsigned long addr);
 /*
  * This flag is used to indicate that the page pointed to by a pte
  * is dirty and requires cleaning before returning it to the user.
  */
 #define PG_dcache_dirty			PG_arch_1
 #define Page_dcache_dirty(page)		\
 	test_bit(PG_dcache_dirty, &(page)->flags)
 #define SetPageDcacheDirty(page)	\
 	set_bit(PG_dcache_dirty, &(page)->flags)
 #define ClearPageDcacheDirty(page)	\
 	clear_bit(PG_dcache_dirty, &(page)->flags)
 /* Run kernel code uncached, useful for cache probing functions. */
 unsigned long __init run_uncached(void *func);
 #endif /* _ASM_CACHEFLUSH_H */

include/asm-parisc/cacheflush.h

Diff comments View file @ ec8c044

 #ifndef _PARISC_CACHEFLUSH_H
 #define _PARISC_CACHEFLUSH_H
 #include <linux/mm.h>
 #include <asm/cache.h>	/* for flush_user_dcache_range_asm() proto */
 /* The usual comment is "Caches aren't brain-dead on the <architecture>".
  * Unfortunately, that doesn't apply to PA-RISC. */
 /* Cache flush operations */
 #ifdef CONFIG_SMP
 #define flush_cache_mm(mm) flush_cache_all()
 #else
 #define flush_cache_mm(mm) flush_cache_all_local()
 #endif
+#define flush_cache_dup_mm(mm) flush_cache_mm(mm)
 #define flush_kernel_dcache_range(start,size) \
 	flush_kernel_dcache_range_asm((start), (start)+(size));
 extern void flush_cache_all_local(void);
 static inline void cacheflush_h_tmp_function(void *dummy)
 {
 	flush_cache_all_local();
 }
 static inline void flush_cache_all(void)
 {
 	on_each_cpu(cacheflush_h_tmp_function, NULL, 1, 1);
 }
 #define flush_cache_vmap(start, end)		flush_cache_all()
 #define flush_cache_vunmap(start, end)		flush_cache_all()
 extern int parisc_cache_flush_threshold;
 void parisc_setup_cache_timing(void);
 static inline void
 flush_user_dcache_range(unsigned long start, unsigned long end)
 {
 	if ((end - start) < parisc_cache_flush_threshold)
 		flush_user_dcache_range_asm(start,end);
 	else
 		flush_data_cache();
 }
 static inline void
 flush_user_icache_range(unsigned long start, unsigned long end)
 {
 	if ((end - start) < parisc_cache_flush_threshold)
 		flush_user_icache_range_asm(start,end);
 	else
 		flush_instruction_cache();
 }
 extern void flush_dcache_page(struct page *page);
 #define flush_dcache_mmap_lock(mapping) \
 	write_lock_irq(&(mapping)->tree_lock)
 #define flush_dcache_mmap_unlock(mapping) \
 	write_unlock_irq(&(mapping)->tree_lock)
 #define flush_icache_page(vma,page)	do { flush_kernel_dcache_page(page); flush_kernel_icache_page(page_address(page)); } while (0)
 #define flush_icache_range(s,e)		do { flush_kernel_dcache_range_asm(s,e); flush_kernel_icache_range_asm(s,e); } while (0)
 #define copy_to_user_page(vma, page, vaddr, dst, src, len) \
 do { \
 	flush_cache_page(vma, vaddr, page_to_pfn(page)); \
 	memcpy(dst, src, len); \
 	flush_kernel_dcache_range_asm((unsigned long)dst, (unsigned long)dst + len); \
 } while (0)
 #define copy_from_user_page(vma, page, vaddr, dst, src, len) \
 do { \
 	flush_cache_page(vma, vaddr, page_to_pfn(page)); \
 	memcpy(dst, src, len); \
 } while (0)
 static inline void flush_cache_range(struct vm_area_struct *vma,
 		unsigned long start, unsigned long end)
 {
 	int sr3;
 	if (!vma->vm_mm->context) {
 		BUG();
 		return;
 	}
 	sr3 = mfsp(3);
 	if (vma->vm_mm->context == sr3) {
 		flush_user_dcache_range(start,end);
 		flush_user_icache_range(start,end);
 	} else {
 		flush_cache_all();
 	}
 }
 /* Simple function to work out if we have an existing address translation
  * for a user space vma. */
 static inline int translation_exists(struct vm_area_struct *vma,
 				unsigned long addr, unsigned long pfn)
 {
 	pgd_t *pgd = pgd_offset(vma->vm_mm, addr);
 	pmd_t *pmd;
 	pte_t pte;
 	if(pgd_none(*pgd))
 		return 0;
 	pmd = pmd_offset(pgd, addr);
 	if(pmd_none(*pmd) || pmd_bad(*pmd))
 		return 0;
 	/* We cannot take the pte lock here: flush_cache_page is usually
 	 * called with pte lock already held.  Whereas flush_dcache_page
 	 * takes flush_dcache_mmap_lock, which is lower in the hierarchy:
 	 * the vma itself is secure, but the pte might come or go racily.
 	 */
 	pte = *pte_offset_map(pmd, addr);
 	/* But pte_unmap() does nothing on this architecture */
 	/* Filter out coincidental file entries and swap entries */
 	if (!(pte_val(pte) & (_PAGE_FLUSH|_PAGE_PRESENT)))
 		return 0;
 	return pte_pfn(pte) == pfn;
 }
 /* Private function to flush a page from the cache of a non-current
  * process.  cr25 contains the Page Directory of the current user
  * process; we're going to hijack both it and the user space %sr3 to
  * temporarily make the non-current process current.  We have to do
  * this because cache flushing may cause a non-access tlb miss which
  * the handlers have to fill in from the pgd of the non-current
  * process. */
 static inline void
 flush_user_cache_page_non_current(struct vm_area_struct *vma,
 				  unsigned long vmaddr)
 {
 	/* save the current process space and pgd */
 	unsigned long space = mfsp(3), pgd = mfctl(25);
 	/* we don't mind taking interrups since they may not
 	 * do anything with user space, but we can't
 	 * be preempted here */
 	preempt_disable();
 	/* make us current */
 	mtctl(__pa(vma->vm_mm->pgd), 25);
 	mtsp(vma->vm_mm->context, 3);
 	flush_user_dcache_page(vmaddr);
 	if(vma->vm_flags & VM_EXEC)
 		flush_user_icache_page(vmaddr);
 	/* put the old current process back */
 	mtsp(space, 3);
 	mtctl(pgd, 25);
 	preempt_enable();
 }
 static inline void
 __flush_cache_page(struct vm_area_struct *vma, unsigned long vmaddr)
 {
 	if (likely(vma->vm_mm->context == mfsp(3))) {
 		flush_user_dcache_page(vmaddr);
 		if (vma->vm_flags & VM_EXEC)
 			flush_user_icache_page(vmaddr);
 	} else {
 		flush_user_cache_page_non_current(vma, vmaddr);
 	}
 }
 static inline void
 flush_cache_page(struct vm_area_struct *vma, unsigned long vmaddr, unsigned long pfn)
 {
 	BUG_ON(!vma->vm_mm->context);
 	if (likely(translation_exists(vma, vmaddr, pfn)))
 		__flush_cache_page(vma, vmaddr);
 }
 static inline void
 flush_anon_page(struct page *page, unsigned long vmaddr)
 {
 	if (PageAnon(page))
 		flush_user_dcache_page(vmaddr);
 }
 #define ARCH_HAS_FLUSH_ANON_PAGE
 #define ARCH_HAS_FLUSH_KERNEL_DCACHE_PAGE
 void flush_kernel_dcache_page_addr(void *addr);
 static inline void flush_kernel_dcache_page(struct page *page)
 {
 	flush_kernel_dcache_page_addr(page_address(page));
 }
 #ifdef CONFIG_DEBUG_RODATA
 void mark_rodata_ro(void);
 #endif
 #ifdef CONFIG_PA8X00
 /* Only pa8800, pa8900 needs this */
 #define ARCH_HAS_KMAP
 void kunmap_parisc(void *addr);
 static inline void *kmap(struct page *page)
 {
 	might_sleep();
 	return page_address(page);
 }
 #define kunmap(page)			kunmap_parisc(page_address(page))
 #define kmap_atomic(page, idx)		page_address(page)
 #define kunmap_atomic(addr, idx)	kunmap_parisc(addr)
 #define kmap_atomic_pfn(pfn, idx)	page_address(pfn_to_page(pfn))
 #define kmap_atomic_to_page(ptr)	virt_to_page(ptr)
 #endif
 #endif /* _PARISC_CACHEFLUSH_H */

include/asm-powerpc/cacheflush.h

Diff comments View file @ ec8c044

 /*
  *  This program is free software; you can redistribute it and/or
  *  modify it under the terms of the GNU General Public License
  *  as published by the Free Software Foundation; either version
  *  2 of the License, or (at your option) any later version.
  */
 #ifndef _ASM_POWERPC_CACHEFLUSH_H
 #define _ASM_POWERPC_CACHEFLUSH_H
 #ifdef __KERNEL__
 #include <linux/mm.h>
 #include <asm/cputable.h>
 /*
  * No cache flushing is required when address mappings are changed,
  * because the caches on PowerPCs are physically addressed.
  */
 #define flush_cache_all()			do { } while (0)
 #define flush_cache_mm(mm)			do { } while (0)
+#define flush_cache_dup_mm(mm)			do { } while (0)
 #define flush_cache_range(vma, start, end)	do { } while (0)
 #define flush_cache_page(vma, vmaddr, pfn)	do { } while (0)
 #define flush_icache_page(vma, page)		do { } while (0)
 #define flush_cache_vmap(start, end)		do { } while (0)
 #define flush_cache_vunmap(start, end)		do { } while (0)
 extern void flush_dcache_page(struct page *page);
 #define flush_dcache_mmap_lock(mapping)		do { } while (0)
 #define flush_dcache_mmap_unlock(mapping)	do { } while (0)
 extern void __flush_icache_range(unsigned long, unsigned long);
 static inline void flush_icache_range(unsigned long start, unsigned long stop)
 {
 	if (!cpu_has_feature(CPU_FTR_COHERENT_ICACHE))
 		__flush_icache_range(start, stop);
 }
 extern void flush_icache_user_range(struct vm_area_struct *vma,
 				    struct page *page, unsigned long addr,
 				    int len);
 extern void __flush_dcache_icache(void *page_va);
 extern void flush_dcache_icache_page(struct page *page);
 #if defined(CONFIG_PPC32) && !defined(CONFIG_BOOKE)
 extern void __flush_dcache_icache_phys(unsigned long physaddr);
 #endif /* CONFIG_PPC32 && !CONFIG_BOOKE */
 extern void flush_dcache_range(unsigned long start, unsigned long stop);
 #ifdef CONFIG_PPC32
 extern void clean_dcache_range(unsigned long start, unsigned long stop);
 extern void invalidate_dcache_range(unsigned long start, unsigned long stop);
 #endif /* CONFIG_PPC32 */
 #ifdef CONFIG_PPC64
 extern void flush_inval_dcache_range(unsigned long start, unsigned long stop);
 extern void flush_dcache_phys_range(unsigned long start, unsigned long stop);
 #endif
 #define copy_to_user_page(vma, page, vaddr, dst, src, len) \
 	do { \
 		memcpy(dst, src, len); \
 		flush_icache_user_range(vma, page, vaddr, len); \
 	} while (0)
 #define copy_from_user_page(vma, page, vaddr, dst, src, len) \
 	memcpy(dst, src, len)
 #endif /* __KERNEL__ */
 #endif /* _ASM_POWERPC_CACHEFLUSH_H */

include/asm-s390/cacheflush.h

Diff comments View file @ ec8c044

 #ifndef _S390_CACHEFLUSH_H
 #define _S390_CACHEFLUSH_H
 /* Keep includes the same across arches.  */
 #include <linux/mm.h>
 /* Caches aren't brain-dead on the s390. */
 #define flush_cache_all()			do { } while (0)
 #define flush_cache_mm(mm)			do { } while (0)
+#define flush_cache_dup_mm(mm)			do { } while (0)
 #define flush_cache_range(vma, start, end)	do { } while (0)
 #define flush_cache_page(vma, vmaddr, pfn)	do { } while (0)
 #define flush_dcache_page(page)			do { } while (0)
 #define flush_dcache_mmap_lock(mapping)		do { } while (0)
 #define flush_dcache_mmap_unlock(mapping)	do { } while (0)
 #define flush_icache_range(start, end)		do { } while (0)
 #define flush_icache_page(vma,pg)		do { } while (0)
 #define flush_icache_user_range(vma,pg,adr,len)	do { } while (0)
 #define flush_cache_vmap(start, end)		do { } while (0)
 #define flush_cache_vunmap(start, end)		do { } while (0)
 #define copy_to_user_page(vma, page, vaddr, dst, src, len) \
 	memcpy(dst, src, len)
 #define copy_from_user_page(vma, page, vaddr, dst, src, len) \
 	memcpy(dst, src, len)
 #endif /* _S390_CACHEFLUSH_H */

include/asm-sh/cpu-sh2/cacheflush.h

Diff comments View file @ ec8c044

 /*
  * include/asm-sh/cpu-sh2/cacheflush.h
  *
  * Copyright (C) 2003 Paul Mundt
  *
  * This file is subject to the terms and conditions of the GNU General Public
  * License.  See the file "COPYING" in the main directory of this archive
  * for more details.
  */
 #ifndef __ASM_CPU_SH2_CACHEFLUSH_H
 #define __ASM_CPU_SH2_CACHEFLUSH_H
 /*
  * Cache flushing:
  *
  *  - flush_cache_all() flushes entire cache
  *  - flush_cache_mm(mm) flushes the specified mm context's cache lines
+ *  - flush_cache_dup mm(mm) handles cache flushing when forking
  *  - flush_cache_page(mm, vmaddr, pfn) flushes a single page
  *  - flush_cache_range(vma, start, end) flushes a range of pages
  *
  *  - flush_dcache_page(pg) flushes(wback&invalidates) a page for dcache
  *  - flush_icache_range(start, end) flushes(invalidates) a range for icache
  *  - flush_icache_page(vma, pg) flushes(invalidates) a page for icache
  *
  *  Caches are indexed (effectively) by physical address on SH-2, so
  *  we don't need them.
  */
 #define flush_cache_all()			do { } while (0)
 #define flush_cache_mm(mm)			do { } while (0)
+#define flush_cache_dup_mm(mm)			do { } while (0)
 #define flush_cache_range(vma, start, end)	do { } while (0)
 #define flush_cache_page(vma, vmaddr, pfn)	do { } while (0)
 #define flush_dcache_page(page)			do { } while (0)
 #define flush_dcache_mmap_lock(mapping)		do { } while (0)
 #define flush_dcache_mmap_unlock(mapping)	do { } while (0)
 #define flush_icache_range(start, end)		do { } while (0)
 #define flush_icache_page(vma,pg)		do { } while (0)
 #define flush_icache_user_range(vma,pg,adr,len)	do { } while (0)
 #define flush_cache_sigtramp(vaddr)		do { } while (0)
 #define p3_cache_init()				do { } while (0)
 #endif /* __ASM_CPU_SH2_CACHEFLUSH_H */

include/asm-sh/cpu-sh3/cacheflush.h

Diff comments View file @ ec8c044

 /*
  * include/asm-sh/cpu-sh3/cacheflush.h
  *
  * Copyright (C) 1999 Niibe Yutaka
  *
  * This file is subject to the terms and conditions of the GNU General Public
  * License.  See the file "COPYING" in the main directory of this archive
  * for more details.
  */
 #ifndef __ASM_CPU_SH3_CACHEFLUSH_H
 #define __ASM_CPU_SH3_CACHEFLUSH_H
 /*
  * Cache flushing:
  *
  *  - flush_cache_all() flushes entire cache
  *  - flush_cache_mm(mm) flushes the specified mm context's cache lines
+ *  - flush_cache_dup mm(mm) handles cache flushing when forking
  *  - flush_cache_page(mm, vmaddr, pfn) flushes a single page
  *  - flush_cache_range(vma, start, end) flushes a range of pages
  *
  *  - flush_dcache_page(pg) flushes(wback&invalidates) a page for dcache
  *  - flush_icache_range(start, end) flushes(invalidates) a range for icache
  *  - flush_icache_page(vma, pg) flushes(invalidates) a page for icache
  *
  *  Caches are indexed (effectively) by physical address on SH-3, so
  *  we don't need them.
  */
 #if defined(CONFIG_SH7705_CACHE_32KB)
 /* SH7705 is an SH3 processor with 32KB cache. This has alias issues like the
  * SH4. Unlike the SH4 this is a unified cache so we need to do some work
  * in mmap when 'exec'ing a new binary
  */
  /* 32KB cache, 4kb PAGE sizes need to check bit 12 */
 #define CACHE_ALIAS 0x00001000
 #define PG_mapped	PG_arch_1
 void flush_cache_all(void);
 void flush_cache_mm(struct mm_struct *mm);
+#define flush_cache_dup_mm(mm) flush_cache_mm(mm)
 void flush_cache_range(struct vm_area_struct *vma, unsigned long start,
                               unsigned long end);
 void flush_cache_page(struct vm_area_struct *vma, unsigned long addr, unsigned long pfn);
 void flush_dcache_page(struct page *pg);
 void flush_icache_range(unsigned long start, unsigned long end);
 void flush_icache_page(struct vm_area_struct *vma, struct page *page);
 #else
 #define flush_cache_all()			do { } while (0)
 #define flush_cache_mm(mm)			do { } while (0)
+#define flush_cache_dup_mm(mm)			do { } while (0)
 #define flush_cache_range(vma, start, end)	do { } while (0)
 #define flush_cache_page(vma, vmaddr, pfn)	do { } while (0)
 #define flush_dcache_page(page)			do { } while (0)
 #define flush_icache_range(start, end)		do { } while (0)
 #define flush_icache_page(vma,pg)		do { } while (0)
 #endif
 #define flush_dcache_mmap_lock(mapping)		do { } while (0)
 #define flush_dcache_mmap_unlock(mapping)	do { } while (0)
 /* SH3 has unified cache so no special action needed here */
 #define flush_cache_sigtramp(vaddr)		do { } while (0)
 #define flush_icache_user_range(vma,pg,adr,len)	do { } while (0)
 #define p3_cache_init()				do { } while (0)
 #endif /* __ASM_CPU_SH3_CACHEFLUSH_H */

include/asm-sh/cpu-sh4/cacheflush.h

Diff comments View file @ ec8c044

 /*
  * include/asm-sh/cpu-sh4/cacheflush.h
  *
  * Copyright (C) 1999 Niibe Yutaka
  * Copyright (C) 2003 Paul Mundt
  *
  * This file is subject to the terms and conditions of the GNU General Public
  * License.  See the file "COPYING" in the main directory of this archive
  * for more details.
  */
 #ifndef __ASM_CPU_SH4_CACHEFLUSH_H
 #define __ASM_CPU_SH4_CACHEFLUSH_H
 /*
  *  Caches are broken on SH-4 (unless we use write-through
  *  caching; in which case they're only semi-broken),
  *  so we need them.
  */
 void flush_cache_all(void);
 void flush_cache_mm(struct mm_struct *mm);
+#define flush_cache_dup_mm(mm) flush_cache_mm(mm)
 void flush_cache_range(struct vm_area_struct *vma, unsigned long start,
 		       unsigned long end);
 void flush_cache_page(struct vm_area_struct *vma, unsigned long addr,
 		      unsigned long pfn);
 void flush_dcache_page(struct page *pg);
 #define flush_dcache_mmap_lock(mapping)		do { } while (0)
 #define flush_dcache_mmap_unlock(mapping)	do { } while (0)
 void flush_icache_range(unsigned long start, unsigned long end);
 void flush_cache_sigtramp(unsigned long addr);
 void flush_icache_user_range(struct vm_area_struct *vma, struct page *page,
 			     unsigned long addr, int len);
 #define flush_icache_page(vma,pg)		do { } while (0)
 /* Initialization of P3 area for copy_user_page */
 void p3_cache_init(void);
 #define PG_mapped	PG_arch_1
 #ifdef CONFIG_MMU
 extern int remap_area_pages(unsigned long addr, unsigned long phys_addr,
 			    unsigned long size, unsigned long flags);
 #else /* CONFIG_MMU */
 static inline int remap_area_pages(unsigned long addr, unsigned long phys_addr,
 				   unsigned long size, unsigned long flags)
 {
 	return 0;
 }
 #endif /* CONFIG_MMU */
 #endif /* __ASM_CPU_SH4_CACHEFLUSH_H */

include/asm-sh64/cacheflush.h

Diff comments View file @ ec8c044

 #ifndef __ASM_SH64_CACHEFLUSH_H
 #define __ASM_SH64_CACHEFLUSH_H
 #ifndef __ASSEMBLY__
 #include <asm/page.h>
 struct vm_area_struct;
 struct page;
 struct mm_struct;
 extern void flush_cache_all(void);
 extern void flush_cache_mm(struct mm_struct *mm);
 extern void flush_cache_sigtramp(unsigned long start, unsigned long end);
 extern void flush_cache_range(struct vm_area_struct *vma, unsigned long start,
 			      unsigned long end);
 extern void flush_cache_page(struct vm_area_struct *vma, unsigned long addr, unsigned long pfn);
 extern void flush_dcache_page(struct page *pg);
 extern void flush_icache_range(unsigned long start, unsigned long end);
 extern void flush_icache_user_range(struct vm_area_struct *vma,
 				    struct page *page, unsigned long addr,
 				    int len);
+#define flush_cache_dup_mm(mm)	flush_cache_mm(mm)
 #define flush_dcache_mmap_lock(mapping)		do { } while (0)
 #define flush_dcache_mmap_unlock(mapping)	do { } while (0)
 #define flush_cache_vmap(start, end)		flush_cache_all()
 #define flush_cache_vunmap(start, end)		flush_cache_all()
 #define flush_icache_page(vma, page)	do { } while (0)
 #define copy_to_user_page(vma, page, vaddr, dst, src, len) \
 	do {							\
 		flush_cache_page(vma, vaddr, page_to_pfn(page));\
 		memcpy(dst, src, len);				\
 		flush_icache_user_range(vma, page, vaddr, len);	\
 	} while (0)
 #define copy_from_user_page(vma, page, vaddr, dst, src, len) \
 	do {							\
 		flush_cache_page(vma, vaddr, page_to_pfn(page));\
 		memcpy(dst, src, len);				\
 	} while (0)
 #endif /* __ASSEMBLY__ */
 #endif /* __ASM_SH64_CACHEFLUSH_H */

include/asm-sparc/cacheflush.h

Diff comments View file @ ec8c044

 #ifndef _SPARC_CACHEFLUSH_H
 #define _SPARC_CACHEFLUSH_H
 #include <linux/mm.h>		/* Common for other includes */
 // #include <linux/kernel.h> from pgalloc.h
 // #include <linux/sched.h>  from pgalloc.h
 // #include <asm/page.h>
 #include <asm/btfixup.h>
 /*
  * Fine grained cache flushing.
  */
 #ifdef CONFIG_SMP
 BTFIXUPDEF_CALL(void, local_flush_cache_all, void)
 BTFIXUPDEF_CALL(void, local_flush_cache_mm, struct mm_struct *)
 BTFIXUPDEF_CALL(void, local_flush_cache_range, struct vm_area_struct *, unsigned long, unsigned long)
 BTFIXUPDEF_CALL(void, local_flush_cache_page, struct vm_area_struct *, unsigned long)
 #define local_flush_cache_all() BTFIXUP_CALL(local_flush_cache_all)()
 #define local_flush_cache_mm(mm) BTFIXUP_CALL(local_flush_cache_mm)(mm)
 #define local_flush_cache_range(vma,start,end) BTFIXUP_CALL(local_flush_cache_range)(vma,start,end)
 #define local_flush_cache_page(vma,addr) BTFIXUP_CALL(local_flush_cache_page)(vma,addr)
 BTFIXUPDEF_CALL(void, local_flush_page_to_ram, unsigned long)
 BTFIXUPDEF_CALL(void, local_flush_sig_insns, struct mm_struct *, unsigned long)
 #define local_flush_page_to_ram(addr) BTFIXUP_CALL(local_flush_page_to_ram)(addr)
 #define local_flush_sig_insns(mm,insn_addr) BTFIXUP_CALL(local_flush_sig_insns)(mm,insn_addr)
 extern void smp_flush_cache_all(void);
 extern void smp_flush_cache_mm(struct mm_struct *mm);
 extern void smp_flush_cache_range(struct vm_area_struct *vma,
 				  unsigned long start,
 				  unsigned long end);
 extern void smp_flush_cache_page(struct vm_area_struct *vma, unsigned long page);
 extern void smp_flush_page_to_ram(unsigned long page);
 extern void smp_flush_sig_insns(struct mm_struct *mm, unsigned long insn_addr);
 #endif /* CONFIG_SMP */
 BTFIXUPDEF_CALL(void, flush_cache_all, void)
 BTFIXUPDEF_CALL(void, flush_cache_mm, struct mm_struct *)
 BTFIXUPDEF_CALL(void, flush_cache_range, struct vm_area_struct *, unsigned long, unsigned long)
 BTFIXUPDEF_CALL(void, flush_cache_page, struct vm_area_struct *, unsigned long)
 #define flush_cache_all() BTFIXUP_CALL(flush_cache_all)()
 #define flush_cache_mm(mm) BTFIXUP_CALL(flush_cache_mm)(mm)
+#define flush_cache_dup_mm(mm) BTFIXUP_CALL(flush_cache_mm)(mm)
 #define flush_cache_range(vma,start,end) BTFIXUP_CALL(flush_cache_range)(vma,start,end)
 #define flush_cache_page(vma,addr,pfn) BTFIXUP_CALL(flush_cache_page)(vma,addr)
 #define flush_icache_range(start, end)		do { } while (0)
 #define flush_icache_page(vma, pg)		do { } while (0)
 #define flush_icache_user_range(vma,pg,adr,len)	do { } while (0)
 #define copy_to_user_page(vma, page, vaddr, dst, src, len) \
 	do {							\
 		flush_cache_page(vma, vaddr, page_to_pfn(page));\
 		memcpy(dst, src, len);				\
 	} while (0)
 #define copy_from_user_page(vma, page, vaddr, dst, src, len) \
 	do {							\
 		flush_cache_page(vma, vaddr, page_to_pfn(page));\
 		memcpy(dst, src, len);				\
 	} while (0)
 BTFIXUPDEF_CALL(void, __flush_page_to_ram, unsigned long)
 BTFIXUPDEF_CALL(void, flush_sig_insns, struct mm_struct *, unsigned long)
 #define __flush_page_to_ram(addr) BTFIXUP_CALL(__flush_page_to_ram)(addr)
 #define flush_sig_insns(mm,insn_addr) BTFIXUP_CALL(flush_sig_insns)(mm,insn_addr)
 extern void sparc_flush_page_to_ram(struct page *page);
 #define flush_dcache_page(page)			sparc_flush_page_to_ram(page)
 #define flush_dcache_mmap_lock(mapping)		do { } while (0)
 #define flush_dcache_mmap_unlock(mapping)	do { } while (0)
 #define flush_cache_vmap(start, end)		flush_cache_all()
 #define flush_cache_vunmap(start, end)		flush_cache_all()
 #endif /* _SPARC_CACHEFLUSH_H */

include/asm-sparc64/cacheflush.h

Diff comments View file @ ec8c044

 #ifndef _SPARC64_CACHEFLUSH_H
 #define _SPARC64_CACHEFLUSH_H
 #include <asm/page.h>
 #ifndef __ASSEMBLY__
 #include <linux/mm.h>
 /* Cache flush operations. */
 /* These are the same regardless of whether this is an SMP kernel or not. */
 #define flush_cache_mm(__mm) \
 	do { if ((__mm) == current->mm) flushw_user(); } while(0)
+#define flush_cache_dup_mm(mm) flush_cache_mm(mm)
 #define flush_cache_range(vma, start, end) \
 	flush_cache_mm((vma)->vm_mm)
 #define flush_cache_page(vma, page, pfn) \
 	flush_cache_mm((vma)->vm_mm)
 /*
  * On spitfire, the icache doesn't snoop local stores and we don't
  * use block commit stores (which invalidate icache lines) during
  * module load, so we need this.
  */
 extern void flush_icache_range(unsigned long start, unsigned long end);
 extern void __flush_icache_page(unsigned long);
 extern void __flush_dcache_page(void *addr, int flush_icache);
 extern void flush_dcache_page_impl(struct page *page);
 #ifdef CONFIG_SMP
 extern void smp_flush_dcache_page_impl(struct page *page, int cpu);
 extern void flush_dcache_page_all(struct mm_struct *mm, struct page *page);
 #else
 #define smp_flush_dcache_page_impl(page,cpu) flush_dcache_page_impl(page)
 #define flush_dcache_page_all(mm,page) flush_dcache_page_impl(page)
 #endif
 extern void __flush_dcache_range(unsigned long start, unsigned long end);
 extern void flush_dcache_page(struct page *page);
 #define flush_icache_page(vma, pg)	do { } while(0)
 #define flush_icache_user_range(vma,pg,adr,len)	do { } while (0)
 extern void flush_ptrace_access(struct vm_area_struct *, struct page *,
 				unsigned long uaddr, void *kaddr,
 				unsigned long len, int write);
 #define copy_to_user_page(vma, page, vaddr, dst, src, len)		\
 	do {								\
 		flush_cache_page(vma, vaddr, page_to_pfn(page));	\
 		memcpy(dst, src, len);					\
 		flush_ptrace_access(vma, page, vaddr, src, len, 0);	\
 	} while (0)
 #define copy_from_user_page(vma, page, vaddr, dst, src, len) 		\
 	do {								\
 		flush_cache_page(vma, vaddr, page_to_pfn(page));	\
 		memcpy(dst, src, len);					\
 		flush_ptrace_access(vma, page, vaddr, dst, len, 1);	\
 	} while (0)
 #define flush_dcache_mmap_lock(mapping)		do { } while (0)
 #define flush_dcache_mmap_unlock(mapping)	do { } while (0)
 #define flush_cache_vmap(start, end)		do { } while (0)
 #define flush_cache_vunmap(start, end)		do { } while (0)
 #ifdef CONFIG_DEBUG_PAGEALLOC
 /* internal debugging function */
 void kernel_map_pages(struct page *page, int numpages, int enable);
 #endif
 #endif /* !__ASSEMBLY__ */
 #endif /* _SPARC64_CACHEFLUSH_H */

include/asm-v850/cacheflush.h

Diff comments View file @ ec8c044

 /*
  * include/asm-v850/cacheflush.h
  *
  *  Copyright (C) 2001,02,03  NEC Electronics Corporation
  *  Copyright (C) 2001,02,03  Miles Bader <miles@gnu.org>
  *
  * This file is subject to the terms and conditions of the GNU General
  * Public License.  See the file COPYING in the main directory of this
  * archive for more details.
  *
  * Written by Miles Bader <miles@gnu.org>
  */
 #ifndef __V850_CACHEFLUSH_H__
 #define __V850_CACHEFLUSH_H__
 /* Somebody depends on this; sigh...  */
 #include <linux/mm.h>
 #include <asm/machdep.h>
 /* The following are all used by the kernel in ways that only affect
    systems with MMUs, so we don't need them.  */
 #define flush_cache_all()			((void)0)
 #define flush_cache_mm(mm)			((void)0)
+#define flush_cache_dup_mm(mm)			((void)0)
 #define flush_cache_range(vma, start, end)	((void)0)
 #define flush_cache_page(vma, vmaddr, pfn)	((void)0)
 #define flush_dcache_page(page)			((void)0)
 #define flush_dcache_mmap_lock(mapping)		((void)0)
 #define flush_dcache_mmap_unlock(mapping)	((void)0)
 #define flush_cache_vmap(start, end)		((void)0)
 #define flush_cache_vunmap(start, end)		((void)0)
 #ifdef CONFIG_NO_CACHE
 /* Some systems have no cache at all, in which case we don't need these
    either.  */
 #define flush_icache()				((void)0)
 #define flush_icache_range(start, end)		((void)0)
 #define flush_icache_page(vma,pg)		((void)0)
 #define flush_icache_user_range(vma,pg,adr,len)	((void)0)
 #define flush_cache_sigtramp(vaddr)		((void)0)
 #else /* !CONFIG_NO_CACHE */
 struct page;
 struct mm_struct;
 struct vm_area_struct;
 /* Otherwise, somebody had better define them.  */
 extern void flush_icache (void);
 extern void flush_icache_range (unsigned long start, unsigned long end);
 extern void flush_icache_page (struct vm_area_struct *vma, struct page *page);
 extern void flush_icache_user_range (struct vm_area_struct *vma,
 				     struct page *page,
 				     unsigned long adr, int len);
 extern void flush_cache_sigtramp (unsigned long addr);
 #endif /* CONFIG_NO_CACHE */
 #define copy_to_user_page(vma, page, vaddr, dst, src, len) \
 do { memcpy(dst, src, len); \
      flush_icache_user_range(vma, page, vaddr, len); \
 } while (0)
 #define copy_from_user_page(vma, page, vaddr, dst, src, len) \
 	memcpy(dst, src, len)
 #endif /* __V850_CACHEFLUSH_H__ */

include/asm-x86_64/cacheflush.h

Diff comments View file @ ec8c044

 #ifndef _X8664_CACHEFLUSH_H
 #define _X8664_CACHEFLUSH_H
 /* Keep includes the same across arches.  */
 #include <linux/mm.h>
 /* Caches aren't brain-dead on the intel. */
 #define flush_cache_all()			do { } while (0)
 #define flush_cache_mm(mm)			do { } while (0)
+#define flush_cache_dup_mm(mm)			do { } while (0)
 #define flush_cache_range(vma, start, end)	do { } while (0)
 #define flush_cache_page(vma, vmaddr, pfn)	do { } while (0)
 #define flush_dcache_page(page)			do { } while (0)
 #define flush_dcache_mmap_lock(mapping)		do { } while (0)
 #define flush_dcache_mmap_unlock(mapping)	do { } while (0)
 #define flush_icache_range(start, end)		do { } while (0)
 #define flush_icache_page(vma,pg)		do { } while (0)
 #define flush_icache_user_range(vma,pg,adr,len)	do { } while (0)
 #define flush_cache_vmap(start, end)		do { } while (0)
 #define flush_cache_vunmap(start, end)		do { } while (0)
 #define copy_to_user_page(vma, page, vaddr, dst, src, len) \
 	memcpy(dst, src, len)
 #define copy_from_user_page(vma, page, vaddr, dst, src, len) \
 	memcpy(dst, src, len)
 void global_flush_tlb(void);
 int change_page_attr(struct page *page, int numpages, pgprot_t prot);
 int change_page_attr_addr(unsigned long addr, int numpages, pgprot_t prot);
 #ifdef CONFIG_DEBUG_RODATA
 void mark_rodata_ro(void);
 #endif
 #endif /* _X8664_CACHEFLUSH_H */

include/asm-xtensa/cacheflush.h

Diff comments View file @ ec8c044

 /*
  * include/asm-xtensa/cacheflush.h
  *
  * This file is subject to the terms and conditions of the GNU General Public
  * License.  See the file "COPYING" in the main directory of this archive
  * for more details.
  *
  * (C) 2001 - 2006 Tensilica Inc.
  */
 #ifndef _XTENSA_CACHEFLUSH_H
 #define _XTENSA_CACHEFLUSH_H
 #ifdef __KERNEL__
 #include <linux/mm.h>
 #include <asm/processor.h>
 #include <asm/page.h>
 /*
  * flush and invalidate data cache, invalidate instruction cache:
  *
  * __flush_invalidate_cache_all()
  * __flush_invalidate_cache_range(from,sze)
  *
  * invalidate data or instruction cache:
  *
  * __invalidate_icache_all()
  * __invalidate_icache_page(adr)
  * __invalidate_dcache_page(adr)
  * __invalidate_icache_range(from,size)
  * __invalidate_dcache_range(from,size)
  *
  * flush data cache:
  *
  * __flush_dcache_page(adr)
  *
  * flush and invalidate data cache:
  *
  * __flush_invalidate_dcache_all()
  * __flush_invalidate_dcache_page(adr)
  * __flush_invalidate_dcache_range(from,size)
  */
 extern void __flush_invalidate_cache_all(void);
 extern void __flush_invalidate_cache_range(unsigned long, unsigned long);
 extern void __flush_invalidate_dcache_all(void);
 extern void __invalidate_icache_all(void);
 extern void __invalidate_dcache_page(unsigned long);
 extern void __invalidate_icache_page(unsigned long);
 extern void __invalidate_icache_range(unsigned long, unsigned long);
 extern void __invalidate_dcache_range(unsigned long, unsigned long);
 #if XCHAL_DCACHE_IS_WRITEBACK
 extern void __flush_dcache_page(unsigned long);
 extern void __flush_invalidate_dcache_page(unsigned long);
 extern void __flush_invalidate_dcache_range(unsigned long, unsigned long);
 #else
 # define __flush_dcache_page(p)				do { } while(0)
 # define __flush_invalidate_dcache_page(p) 		do { } while(0)
 # define __flush_invalidate_dcache_range(p,s)		do { } while(0)
 #endif
 /*
  * We have physically tagged caches - nothing to do here -
  * unless we have cache aliasing.
  *
  * Pages can get remapped. Because this might change the 'color' of that page,
  * we have to flush the cache before the PTE is changed.
  * (see also Documentation/cachetlb.txt)
  */
 #if (DCACHE_WAY_SIZE > PAGE_SIZE) && XCHAL_DCACHE_IS_WRITEBACK
 #define flush_cache_all()		__flush_invalidate_cache_all();
 #define flush_cache_mm(mm)		__flush_invalidate_cache_all();
+#define flush_cache_dup_mm(mm)		__flush_invalidate_cache_all();
 #define flush_cache_vmap(start,end)	__flush_invalidate_cache_all();
 #define flush_cache_vunmap(start,end)	__flush_invalidate_cache_all();
 extern void flush_dcache_page(struct page*);
 extern void flush_cache_range(struct vm_area_struct*, ulong, ulong);
 extern void flush_cache_page(struct vm_area_struct*, unsigned long, unsigned long);
 #else
 #define flush_cache_all()				do { } while (0)
 #define flush_cache_mm(mm)				do { } while (0)
+#define flush_cache_dup_mm(mm)				do { } while (0)
 #define flush_cache_vmap(start,end)			do { } while (0)
 #define flush_cache_vunmap(start,end)			do { } while (0)
 #define flush_dcache_page(page)				do { } while (0)
 #define flush_cache_page(vma,addr,pfn)			do { } while (0)
 #define flush_cache_range(vma,start,end)		do { } while (0)
 #endif
 #define flush_icache_range(start,end) 					\
 	__invalidate_icache_range(start,(end)-(start))
 /* This is not required, see Documentation/cachetlb.txt */
 #define	flush_icache_page(vma,page)			do { } while(0)
 #define flush_dcache_mmap_lock(mapping)			do { } while (0)
 #define flush_dcache_mmap_unlock(mapping)		do { } while (0)
 #define copy_to_user_page(vma, page, vaddr, dst, src, len) \
 	memcpy(dst, src, len)
 #define copy_from_user_page(vma, page, vaddr, dst, src, len) \
 	memcpy(dst, src, len)
 #endif /* __KERNEL__ */
 #endif /* _XTENSA_CACHEFLUSH_H */

kernel/fork.c

Diff comments View file @ ec8c044

1	/*	1	/*
2	* linux/kernel/fork.c	2	* linux/kernel/fork.c
3	*	3	*
4	* Copyright (C) 1991, 1992 Linus Torvalds	4	* Copyright (C) 1991, 1992 Linus Torvalds
5	*/	5	*/
6		6
7	/*	7	/*
8	* 'fork.c' contains the help-routines for the 'fork' system call	8	* 'fork.c' contains the help-routines for the 'fork' system call
9	* (see also entry.S and others).	9	* (see also entry.S and others).
10	* Fork is rather simple, once you get the hang of it, but the memory	10	* Fork is rather simple, once you get the hang of it, but the memory
11	* management can be a bitch. See 'mm/memory.c': 'copy_page_range()'	11	* management can be a bitch. See 'mm/memory.c': 'copy_page_range()'
12	*/	12	*/
13		13
14	#include <linux/slab.h>	14	#include <linux/slab.h>
15	#include <linux/init.h>	15	#include <linux/init.h>
16	#include <linux/unistd.h>	16	#include <linux/unistd.h>
17	#include <linux/smp_lock.h>	17	#include <linux/smp_lock.h>
18	#include <linux/module.h>	18	#include <linux/module.h>
19	#include <linux/vmalloc.h>	19	#include <linux/vmalloc.h>
20	#include <linux/completion.h>	20	#include <linux/completion.h>
21	#include <linux/mnt_namespace.h>	21	#include <linux/mnt_namespace.h>
22	#include <linux/personality.h>	22	#include <linux/personality.h>
23	#include <linux/mempolicy.h>	23	#include <linux/mempolicy.h>
24	#include <linux/sem.h>	24	#include <linux/sem.h>
25	#include <linux/file.h>	25	#include <linux/file.h>
26	#include <linux/key.h>	26	#include <linux/key.h>
27	#include <linux/binfmts.h>	27	#include <linux/binfmts.h>
28	#include <linux/mman.h>	28	#include <linux/mman.h>
29	#include <linux/fs.h>	29	#include <linux/fs.h>
30	#include <linux/nsproxy.h>	30	#include <linux/nsproxy.h>
31	#include <linux/capability.h>	31	#include <linux/capability.h>
32	#include <linux/cpu.h>	32	#include <linux/cpu.h>
33	#include <linux/cpuset.h>	33	#include <linux/cpuset.h>
34	#include <linux/security.h>	34	#include <linux/security.h>
35	#include <linux/swap.h>	35	#include <linux/swap.h>
36	#include <linux/syscalls.h>	36	#include <linux/syscalls.h>
37	#include <linux/jiffies.h>	37	#include <linux/jiffies.h>
38	#include <linux/futex.h>	38	#include <linux/futex.h>
39	#include <linux/task_io_accounting_ops.h>	39	#include <linux/task_io_accounting_ops.h>
40	#include <linux/rcupdate.h>	40	#include <linux/rcupdate.h>
41	#include <linux/ptrace.h>	41	#include <linux/ptrace.h>
42	#include <linux/mount.h>	42	#include <linux/mount.h>
43	#include <linux/audit.h>	43	#include <linux/audit.h>
44	#include <linux/profile.h>	44	#include <linux/profile.h>
45	#include <linux/rmap.h>	45	#include <linux/rmap.h>
46	#include <linux/acct.h>	46	#include <linux/acct.h>
47	#include <linux/tsacct_kern.h>	47	#include <linux/tsacct_kern.h>
48	#include <linux/cn_proc.h>	48	#include <linux/cn_proc.h>
49	#include <linux/delayacct.h>	49	#include <linux/delayacct.h>
50	#include <linux/taskstats_kern.h>	50	#include <linux/taskstats_kern.h>
51	#include <linux/random.h>	51	#include <linux/random.h>
52		52
53	#include <asm/pgtable.h>	53	#include <asm/pgtable.h>
54	#include <asm/pgalloc.h>	54	#include <asm/pgalloc.h>
55	#include <asm/uaccess.h>	55	#include <asm/uaccess.h>
56	#include <asm/mmu_context.h>	56	#include <asm/mmu_context.h>
57	#include <asm/cacheflush.h>	57	#include <asm/cacheflush.h>
58	#include <asm/tlbflush.h>	58	#include <asm/tlbflush.h>
59		59
60	/*	60	/*
61	* Protected counters by write_lock_irq(&tasklist_lock)	61	* Protected counters by write_lock_irq(&tasklist_lock)
62	*/	62	*/
63	unsigned long total_forks; /* Handle normal Linux uptimes. */	63	unsigned long total_forks; /* Handle normal Linux uptimes. */
64	int nr_threads; /* The idle threads do not count.. */	64	int nr_threads; /* The idle threads do not count.. */
65		65
66	int max_threads; /* tunable limit on nr_threads */	66	int max_threads; /* tunable limit on nr_threads */
67		67
68	DEFINE_PER_CPU(unsigned long, process_counts) = 0;	68	DEFINE_PER_CPU(unsigned long, process_counts) = 0;
69		69
70	__cacheline_aligned DEFINE_RWLOCK(tasklist_lock); /* outer */	70	__cacheline_aligned DEFINE_RWLOCK(tasklist_lock); /* outer */
71		71
72	int nr_processes(void)	72	int nr_processes(void)
73	{	73	{
74	int cpu;	74	int cpu;
75	int total = 0;	75	int total = 0;
76		76
77	for_each_online_cpu(cpu)	77	for_each_online_cpu(cpu)
78	total += per_cpu(process_counts, cpu);	78	total += per_cpu(process_counts, cpu);
79		79
80	return total;	80	return total;
81	}	81	}
82		82
83	#ifndef __HAVE_ARCH_TASK_STRUCT_ALLOCATOR	83	#ifndef __HAVE_ARCH_TASK_STRUCT_ALLOCATOR
84	# define alloc_task_struct() kmem_cache_alloc(task_struct_cachep, GFP_KERNEL)	84	# define alloc_task_struct() kmem_cache_alloc(task_struct_cachep, GFP_KERNEL)
85	# define free_task_struct(tsk) kmem_cache_free(task_struct_cachep, (tsk))	85	# define free_task_struct(tsk) kmem_cache_free(task_struct_cachep, (tsk))
86	static struct kmem_cache *task_struct_cachep;	86	static struct kmem_cache *task_struct_cachep;
87	#endif	87	#endif
88		88
89	/* SLAB cache for signal_struct structures (tsk->signal) */	89	/* SLAB cache for signal_struct structures (tsk->signal) */
90	static struct kmem_cache *signal_cachep;	90	static struct kmem_cache *signal_cachep;
91		91
92	/* SLAB cache for sighand_struct structures (tsk->sighand) */	92	/* SLAB cache for sighand_struct structures (tsk->sighand) */
93	struct kmem_cache *sighand_cachep;	93	struct kmem_cache *sighand_cachep;
94		94
95	/* SLAB cache for files_struct structures (tsk->files) */	95	/* SLAB cache for files_struct structures (tsk->files) */
96	struct kmem_cache *files_cachep;	96	struct kmem_cache *files_cachep;
97		97
98	/* SLAB cache for fs_struct structures (tsk->fs) */	98	/* SLAB cache for fs_struct structures (tsk->fs) */
99	struct kmem_cache *fs_cachep;	99	struct kmem_cache *fs_cachep;
100		100
101	/* SLAB cache for vm_area_struct structures */	101	/* SLAB cache for vm_area_struct structures */
102	struct kmem_cache *vm_area_cachep;	102	struct kmem_cache *vm_area_cachep;
103		103
104	/* SLAB cache for mm_struct structures (tsk->mm) */	104	/* SLAB cache for mm_struct structures (tsk->mm) */
105	static struct kmem_cache *mm_cachep;	105	static struct kmem_cache *mm_cachep;
106		106
107	void free_task(struct task_struct *tsk)	107	void free_task(struct task_struct *tsk)
108	{	108	{
109	free_thread_info(tsk->thread_info);	109	free_thread_info(tsk->thread_info);
110	rt_mutex_debug_task_free(tsk);	110	rt_mutex_debug_task_free(tsk);
111	free_task_struct(tsk);	111	free_task_struct(tsk);
112	}	112	}
113	EXPORT_SYMBOL(free_task);	113	EXPORT_SYMBOL(free_task);
114		114
115	void __put_task_struct(struct task_struct *tsk)	115	void __put_task_struct(struct task_struct *tsk)
116	{	116	{
117	WARN_ON(!(tsk->exit_state & (EXIT_DEAD \| EXIT_ZOMBIE)));	117	WARN_ON(!(tsk->exit_state & (EXIT_DEAD \| EXIT_ZOMBIE)));
118	WARN_ON(atomic_read(&tsk->usage));	118	WARN_ON(atomic_read(&tsk->usage));
119	WARN_ON(tsk == current);	119	WARN_ON(tsk == current);
120		120
121	security_task_free(tsk);	121	security_task_free(tsk);
122	free_uid(tsk->user);	122	free_uid(tsk->user);
123	put_group_info(tsk->group_info);	123	put_group_info(tsk->group_info);
124	delayacct_tsk_free(tsk);	124	delayacct_tsk_free(tsk);
125		125
126	if (!profile_handoff_task(tsk))	126	if (!profile_handoff_task(tsk))
127	free_task(tsk);	127	free_task(tsk);
128	}	128	}
129		129
130	void __init fork_init(unsigned long mempages)	130	void __init fork_init(unsigned long mempages)
131	{	131	{
132	#ifndef __HAVE_ARCH_TASK_STRUCT_ALLOCATOR	132	#ifndef __HAVE_ARCH_TASK_STRUCT_ALLOCATOR
133	#ifndef ARCH_MIN_TASKALIGN	133	#ifndef ARCH_MIN_TASKALIGN
134	#define ARCH_MIN_TASKALIGN L1_CACHE_BYTES	134	#define ARCH_MIN_TASKALIGN L1_CACHE_BYTES
135	#endif	135	#endif
136	/* create a slab on which task_structs can be allocated */	136	/* create a slab on which task_structs can be allocated */
137	task_struct_cachep =	137	task_struct_cachep =
138	kmem_cache_create("task_struct", sizeof(struct task_struct),	138	kmem_cache_create("task_struct", sizeof(struct task_struct),
139	ARCH_MIN_TASKALIGN, SLAB_PANIC, NULL, NULL);	139	ARCH_MIN_TASKALIGN, SLAB_PANIC, NULL, NULL);
140	#endif	140	#endif
141		141
142	/*	142	/*
143	* The default maximum number of threads is set to a safe	143	* The default maximum number of threads is set to a safe
144	* value: the thread structures can take up at most half	144	* value: the thread structures can take up at most half
145	* of memory.	145	* of memory.
146	*/	146	*/
147	max_threads = mempages / (8 * THREAD_SIZE / PAGE_SIZE);	147	max_threads = mempages / (8 * THREAD_SIZE / PAGE_SIZE);
148		148
149	/*	149	/*
150	* we need to allow at least 20 threads to boot a system	150	* we need to allow at least 20 threads to boot a system
151	*/	151	*/
152	if(max_threads < 20)	152	if(max_threads < 20)
153	max_threads = 20;	153	max_threads = 20;
154		154
155	init_task.signal->rlim[RLIMIT_NPROC].rlim_cur = max_threads/2;	155	init_task.signal->rlim[RLIMIT_NPROC].rlim_cur = max_threads/2;
156	init_task.signal->rlim[RLIMIT_NPROC].rlim_max = max_threads/2;	156	init_task.signal->rlim[RLIMIT_NPROC].rlim_max = max_threads/2;
157	init_task.signal->rlim[RLIMIT_SIGPENDING] =	157	init_task.signal->rlim[RLIMIT_SIGPENDING] =
158	init_task.signal->rlim[RLIMIT_NPROC];	158	init_task.signal->rlim[RLIMIT_NPROC];
159	}	159	}
160		160
161	static struct task_struct dup_task_struct(struct task_struct orig)	161	static struct task_struct dup_task_struct(struct task_struct orig)
162	{	162	{
163	struct task_struct *tsk;	163	struct task_struct *tsk;
164	struct thread_info *ti;	164	struct thread_info *ti;
165		165
166	prepare_to_copy(orig);	166	prepare_to_copy(orig);
167		167
168	tsk = alloc_task_struct();	168	tsk = alloc_task_struct();
169	if (!tsk)	169	if (!tsk)
170	return NULL;	170	return NULL;
171		171
172	ti = alloc_thread_info(tsk);	172	ti = alloc_thread_info(tsk);
173	if (!ti) {	173	if (!ti) {
174	free_task_struct(tsk);	174	free_task_struct(tsk);
175	return NULL;	175	return NULL;
176	}	176	}
177		177
178	tsk = orig;	178	tsk = orig;
179	tsk->thread_info = ti;	179	tsk->thread_info = ti;
180	setup_thread_stack(tsk, orig);	180	setup_thread_stack(tsk, orig);
181		181
182	#ifdef CONFIG_CC_STACKPROTECTOR	182	#ifdef CONFIG_CC_STACKPROTECTOR
183	tsk->stack_canary = get_random_int();	183	tsk->stack_canary = get_random_int();
184	#endif	184	#endif
185		185
186	/* One for us, one for whoever does the "release_task()" (usually parent) */	186	/* One for us, one for whoever does the "release_task()" (usually parent) */
187	atomic_set(&tsk->usage,2);	187	atomic_set(&tsk->usage,2);
188	atomic_set(&tsk->fs_excl, 0);	188	atomic_set(&tsk->fs_excl, 0);
189	#ifdef CONFIG_BLK_DEV_IO_TRACE	189	#ifdef CONFIG_BLK_DEV_IO_TRACE
190	tsk->btrace_seq = 0;	190	tsk->btrace_seq = 0;
191	#endif	191	#endif
192	tsk->splice_pipe = NULL;	192	tsk->splice_pipe = NULL;
193	return tsk;	193	return tsk;
194	}	194	}
195		195
196	#ifdef CONFIG_MMU	196	#ifdef CONFIG_MMU
197	static inline int dup_mmap(struct mm_struct mm, struct mm_struct oldmm)	197	static inline int dup_mmap(struct mm_struct mm, struct mm_struct oldmm)
198	{	198	{
199	struct vm_area_struct mpnt, tmp, **pprev;	199	struct vm_area_struct mpnt, tmp, **pprev;
200	struct rb_node *rb_link, rb_parent;	200	struct rb_node *rb_link, rb_parent;
201	int retval;	201	int retval;
202	unsigned long charge;	202	unsigned long charge;
203	struct mempolicy *pol;	203	struct mempolicy *pol;
204		204
205	down_write(&oldmm->mmap_sem);	205	down_write(&oldmm->mmap_sem);
206	flush_cache_mm(oldmm);	206	flush_cache_dup_mm(oldmm);
207	/*	207	/*
208	* Not linked in yet - no deadlock potential:	208	* Not linked in yet - no deadlock potential:
209	*/	209	*/
210	down_write_nested(&mm->mmap_sem, SINGLE_DEPTH_NESTING);	210	down_write_nested(&mm->mmap_sem, SINGLE_DEPTH_NESTING);
211		211
212	mm->locked_vm = 0;	212	mm->locked_vm = 0;
213	mm->mmap = NULL;	213	mm->mmap = NULL;
214	mm->mmap_cache = NULL;	214	mm->mmap_cache = NULL;
215	mm->free_area_cache = oldmm->mmap_base;	215	mm->free_area_cache = oldmm->mmap_base;
216	mm->cached_hole_size = ~0UL;	216	mm->cached_hole_size = ~0UL;
217	mm->map_count = 0;	217	mm->map_count = 0;
218	cpus_clear(mm->cpu_vm_mask);	218	cpus_clear(mm->cpu_vm_mask);
219	mm->mm_rb = RB_ROOT;	219	mm->mm_rb = RB_ROOT;
220	rb_link = &mm->mm_rb.rb_node;	220	rb_link = &mm->mm_rb.rb_node;
221	rb_parent = NULL;	221	rb_parent = NULL;
222	pprev = &mm->mmap;	222	pprev = &mm->mmap;
223		223
224	for (mpnt = oldmm->mmap; mpnt; mpnt = mpnt->vm_next) {	224	for (mpnt = oldmm->mmap; mpnt; mpnt = mpnt->vm_next) {
225	struct file *file;	225	struct file *file;
226		226
227	if (mpnt->vm_flags & VM_DONTCOPY) {	227	if (mpnt->vm_flags & VM_DONTCOPY) {
228	long pages = vma_pages(mpnt);	228	long pages = vma_pages(mpnt);
229	mm->total_vm -= pages;	229	mm->total_vm -= pages;
230	vm_stat_account(mm, mpnt->vm_flags, mpnt->vm_file,	230	vm_stat_account(mm, mpnt->vm_flags, mpnt->vm_file,
231	-pages);	231	-pages);
232	continue;	232	continue;
233	}	233	}
234	charge = 0;	234	charge = 0;
235	if (mpnt->vm_flags & VM_ACCOUNT) {	235	if (mpnt->vm_flags & VM_ACCOUNT) {
236	unsigned int len = (mpnt->vm_end - mpnt->vm_start) >> PAGE_SHIFT;	236	unsigned int len = (mpnt->vm_end - mpnt->vm_start) >> PAGE_SHIFT;
237	if (security_vm_enough_memory(len))	237	if (security_vm_enough_memory(len))
238	goto fail_nomem;	238	goto fail_nomem;
239	charge = len;	239	charge = len;
240	}	240	}
241	tmp = kmem_cache_alloc(vm_area_cachep, GFP_KERNEL);	241	tmp = kmem_cache_alloc(vm_area_cachep, GFP_KERNEL);
242	if (!tmp)	242	if (!tmp)
243	goto fail_nomem;	243	goto fail_nomem;
244	tmp = mpnt;	244	tmp = mpnt;
245	pol = mpol_copy(vma_policy(mpnt));	245	pol = mpol_copy(vma_policy(mpnt));
246	retval = PTR_ERR(pol);	246	retval = PTR_ERR(pol);
247	if (IS_ERR(pol))	247	if (IS_ERR(pol))
248	goto fail_nomem_policy;	248	goto fail_nomem_policy;
249	vma_set_policy(tmp, pol);	249	vma_set_policy(tmp, pol);
250	tmp->vm_flags &= ~VM_LOCKED;	250	tmp->vm_flags &= ~VM_LOCKED;
251	tmp->vm_mm = mm;	251	tmp->vm_mm = mm;
252	tmp->vm_next = NULL;	252	tmp->vm_next = NULL;
253	anon_vma_link(tmp);	253	anon_vma_link(tmp);
254	file = tmp->vm_file;	254	file = tmp->vm_file;
255	if (file) {	255	if (file) {
256	struct inode *inode = file->f_path.dentry->d_inode;	256	struct inode *inode = file->f_path.dentry->d_inode;
257	get_file(file);	257	get_file(file);
258	if (tmp->vm_flags & VM_DENYWRITE)	258	if (tmp->vm_flags & VM_DENYWRITE)
259	atomic_dec(&inode->i_writecount);	259	atomic_dec(&inode->i_writecount);
260		260
261	/* insert tmp into the share list, just after mpnt */	261	/* insert tmp into the share list, just after mpnt */
262	spin_lock(&file->f_mapping->i_mmap_lock);	262	spin_lock(&file->f_mapping->i_mmap_lock);
263	tmp->vm_truncate_count = mpnt->vm_truncate_count;	263	tmp->vm_truncate_count = mpnt->vm_truncate_count;
264	flush_dcache_mmap_lock(file->f_mapping);	264	flush_dcache_mmap_lock(file->f_mapping);
265	vma_prio_tree_add(tmp, mpnt);	265	vma_prio_tree_add(tmp, mpnt);
266	flush_dcache_mmap_unlock(file->f_mapping);	266	flush_dcache_mmap_unlock(file->f_mapping);
267	spin_unlock(&file->f_mapping->i_mmap_lock);	267	spin_unlock(&file->f_mapping->i_mmap_lock);
268	}	268	}
269		269
270	/*	270	/*
271	* Link in the new vma and copy the page table entries.	271	* Link in the new vma and copy the page table entries.
272	*/	272	*/
273	*pprev = tmp;	273	*pprev = tmp;
274	pprev = &tmp->vm_next;	274	pprev = &tmp->vm_next;
275		275
276	__vma_link_rb(mm, tmp, rb_link, rb_parent);	276	__vma_link_rb(mm, tmp, rb_link, rb_parent);
277	rb_link = &tmp->vm_rb.rb_right;	277	rb_link = &tmp->vm_rb.rb_right;
278	rb_parent = &tmp->vm_rb;	278	rb_parent = &tmp->vm_rb;
279		279
280	mm->map_count++;	280	mm->map_count++;
281	retval = copy_page_range(mm, oldmm, mpnt);	281	retval = copy_page_range(mm, oldmm, mpnt);
282		282
283	if (tmp->vm_ops && tmp->vm_ops->open)	283	if (tmp->vm_ops && tmp->vm_ops->open)
284	tmp->vm_ops->open(tmp);	284	tmp->vm_ops->open(tmp);
285		285
286	if (retval)	286	if (retval)
287	goto out;	287	goto out;
288	}	288	}
289	retval = 0;	289	retval = 0;
290	out:	290	out:
291	up_write(&mm->mmap_sem);	291	up_write(&mm->mmap_sem);
292	flush_tlb_mm(oldmm);	292	flush_tlb_mm(oldmm);
293	up_write(&oldmm->mmap_sem);	293	up_write(&oldmm->mmap_sem);
294	return retval;	294	return retval;
295	fail_nomem_policy:	295	fail_nomem_policy:
296	kmem_cache_free(vm_area_cachep, tmp);	296	kmem_cache_free(vm_area_cachep, tmp);
297	fail_nomem:	297	fail_nomem:
298	retval = -ENOMEM;	298	retval = -ENOMEM;
299	vm_unacct_memory(charge);	299	vm_unacct_memory(charge);
300	goto out;	300	goto out;
301	}	301	}
302		302
303	static inline int mm_alloc_pgd(struct mm_struct * mm)	303	static inline int mm_alloc_pgd(struct mm_struct * mm)
304	{	304	{
305	mm->pgd = pgd_alloc(mm);	305	mm->pgd = pgd_alloc(mm);
306	if (unlikely(!mm->pgd))	306	if (unlikely(!mm->pgd))
307	return -ENOMEM;	307	return -ENOMEM;
308	return 0;	308	return 0;
309	}	309	}
310		310
311	static inline void mm_free_pgd(struct mm_struct * mm)	311	static inline void mm_free_pgd(struct mm_struct * mm)
312	{	312	{
313	pgd_free(mm->pgd);	313	pgd_free(mm->pgd);
314	}	314	}
315	#else	315	#else
316	#define dup_mmap(mm, oldmm) (0)	316	#define dup_mmap(mm, oldmm) (0)
317	#define mm_alloc_pgd(mm) (0)	317	#define mm_alloc_pgd(mm) (0)
318	#define mm_free_pgd(mm)	318	#define mm_free_pgd(mm)
319	#endif /* CONFIG_MMU */	319	#endif /* CONFIG_MMU */
320		320
321	__cacheline_aligned_in_smp DEFINE_SPINLOCK(mmlist_lock);	321	__cacheline_aligned_in_smp DEFINE_SPINLOCK(mmlist_lock);
322		322
323	#define allocate_mm() (kmem_cache_alloc(mm_cachep, GFP_KERNEL))	323	#define allocate_mm() (kmem_cache_alloc(mm_cachep, GFP_KERNEL))
324	#define free_mm(mm) (kmem_cache_free(mm_cachep, (mm)))	324	#define free_mm(mm) (kmem_cache_free(mm_cachep, (mm)))
325		325
326	#include <linux/init_task.h>	326	#include <linux/init_task.h>
327		327
328	static struct mm_struct * mm_init(struct mm_struct * mm)	328	static struct mm_struct * mm_init(struct mm_struct * mm)
329	{	329	{
330	atomic_set(&mm->mm_users, 1);	330	atomic_set(&mm->mm_users, 1);
331	atomic_set(&mm->mm_count, 1);	331	atomic_set(&mm->mm_count, 1);
332	init_rwsem(&mm->mmap_sem);	332	init_rwsem(&mm->mmap_sem);
333	INIT_LIST_HEAD(&mm->mmlist);	333	INIT_LIST_HEAD(&mm->mmlist);
334	mm->core_waiters = 0;	334	mm->core_waiters = 0;
335	mm->nr_ptes = 0;	335	mm->nr_ptes = 0;
336	set_mm_counter(mm, file_rss, 0);	336	set_mm_counter(mm, file_rss, 0);
337	set_mm_counter(mm, anon_rss, 0);	337	set_mm_counter(mm, anon_rss, 0);
338	spin_lock_init(&mm->page_table_lock);	338	spin_lock_init(&mm->page_table_lock);
339	rwlock_init(&mm->ioctx_list_lock);	339	rwlock_init(&mm->ioctx_list_lock);
340	mm->ioctx_list = NULL;	340	mm->ioctx_list = NULL;
341	mm->free_area_cache = TASK_UNMAPPED_BASE;	341	mm->free_area_cache = TASK_UNMAPPED_BASE;
342	mm->cached_hole_size = ~0UL;	342	mm->cached_hole_size = ~0UL;
343		343
344	if (likely(!mm_alloc_pgd(mm))) {	344	if (likely(!mm_alloc_pgd(mm))) {
345	mm->def_flags = 0;	345	mm->def_flags = 0;
346	return mm;	346	return mm;
347	}	347	}
348	free_mm(mm);	348	free_mm(mm);
349	return NULL;	349	return NULL;
350	}	350	}
351		351
352	/*	352	/*
353	* Allocate and initialize an mm_struct.	353	* Allocate and initialize an mm_struct.
354	*/	354	*/
355	struct mm_struct * mm_alloc(void)	355	struct mm_struct * mm_alloc(void)
356	{	356	{
357	struct mm_struct * mm;	357	struct mm_struct * mm;
358		358
359	mm = allocate_mm();	359	mm = allocate_mm();
360	if (mm) {	360	if (mm) {
361	memset(mm, 0, sizeof(*mm));	361	memset(mm, 0, sizeof(*mm));
362	mm = mm_init(mm);	362	mm = mm_init(mm);
363	}	363	}
364	return mm;	364	return mm;
365	}	365	}
366		366
367	/*	367	/*
368	* Called when the last reference to the mm	368	* Called when the last reference to the mm
369	* is dropped: either by a lazy thread or by	369	* is dropped: either by a lazy thread or by
370	* mmput. Free the page directory and the mm.	370	* mmput. Free the page directory and the mm.
371	*/	371	*/
372	void fastcall __mmdrop(struct mm_struct *mm)	372	void fastcall __mmdrop(struct mm_struct *mm)
373	{	373	{
374	BUG_ON(mm == &init_mm);	374	BUG_ON(mm == &init_mm);
375	mm_free_pgd(mm);	375	mm_free_pgd(mm);
376	destroy_context(mm);	376	destroy_context(mm);
377	free_mm(mm);	377	free_mm(mm);
378	}	378	}
379		379
380	/*	380	/*
381	* Decrement the use count and release all resources for an mm.	381	* Decrement the use count and release all resources for an mm.
382	*/	382	*/
383	void mmput(struct mm_struct *mm)	383	void mmput(struct mm_struct *mm)
384	{	384	{
385	might_sleep();	385	might_sleep();
386		386
387	if (atomic_dec_and_test(&mm->mm_users)) {	387	if (atomic_dec_and_test(&mm->mm_users)) {
388	exit_aio(mm);	388	exit_aio(mm);
389	exit_mmap(mm);	389	exit_mmap(mm);
390	if (!list_empty(&mm->mmlist)) {	390	if (!list_empty(&mm->mmlist)) {
391	spin_lock(&mmlist_lock);	391	spin_lock(&mmlist_lock);
392	list_del(&mm->mmlist);	392	list_del(&mm->mmlist);
393	spin_unlock(&mmlist_lock);	393	spin_unlock(&mmlist_lock);
394	}	394	}
395	put_swap_token(mm);	395	put_swap_token(mm);
396	mmdrop(mm);	396	mmdrop(mm);
397	}	397	}
398	}	398	}
399	EXPORT_SYMBOL_GPL(mmput);	399	EXPORT_SYMBOL_GPL(mmput);
400		400
401	/**	401	/**
402	* get_task_mm - acquire a reference to the task's mm	402	* get_task_mm - acquire a reference to the task's mm
403	*	403	*
404	* Returns %NULL if the task has no mm. Checks PF_BORROWED_MM (meaning	404	* Returns %NULL if the task has no mm. Checks PF_BORROWED_MM (meaning
405	* this kernel workthread has transiently adopted a user mm with use_mm,	405	* this kernel workthread has transiently adopted a user mm with use_mm,
406	* to do its AIO) is not set and if so returns a reference to it, after	406	* to do its AIO) is not set and if so returns a reference to it, after
407	* bumping up the use count. User must release the mm via mmput()	407	* bumping up the use count. User must release the mm via mmput()
408	* after use. Typically used by /proc and ptrace.	408	* after use. Typically used by /proc and ptrace.
409	*/	409	*/
410	struct mm_struct get_task_mm(struct task_struct task)	410	struct mm_struct get_task_mm(struct task_struct task)
411	{	411	{
412	struct mm_struct *mm;	412	struct mm_struct *mm;
413		413
414	task_lock(task);	414	task_lock(task);
415	mm = task->mm;	415	mm = task->mm;
416	if (mm) {	416	if (mm) {
417	if (task->flags & PF_BORROWED_MM)	417	if (task->flags & PF_BORROWED_MM)
418	mm = NULL;	418	mm = NULL;
419	else	419	else
420	atomic_inc(&mm->mm_users);	420	atomic_inc(&mm->mm_users);
421	}	421	}
422	task_unlock(task);	422	task_unlock(task);
423	return mm;	423	return mm;
424	}	424	}
425	EXPORT_SYMBOL_GPL(get_task_mm);	425	EXPORT_SYMBOL_GPL(get_task_mm);
426		426
427	/* Please note the differences between mmput and mm_release.	427	/* Please note the differences between mmput and mm_release.
428	* mmput is called whenever we stop holding onto a mm_struct,	428	* mmput is called whenever we stop holding onto a mm_struct,
429	* error success whatever.	429	* error success whatever.
430	*	430	*
431	* mm_release is called after a mm_struct has been removed	431	* mm_release is called after a mm_struct has been removed
432	* from the current process.	432	* from the current process.
433	*	433	*
434	* This difference is important for error handling, when we	434	* This difference is important for error handling, when we
435	* only half set up a mm_struct for a new process and need to restore	435	* only half set up a mm_struct for a new process and need to restore
436	* the old one. Because we mmput the new mm_struct before	436	* the old one. Because we mmput the new mm_struct before
437	* restoring the old one. . .	437	* restoring the old one. . .
438	* Eric Biederman 10 January 1998	438	* Eric Biederman 10 January 1998
439	*/	439	*/
440	void mm_release(struct task_struct tsk, struct mm_struct mm)	440	void mm_release(struct task_struct tsk, struct mm_struct mm)
441	{	441	{
442	struct completion *vfork_done = tsk->vfork_done;	442	struct completion *vfork_done = tsk->vfork_done;
443		443
444	/* Get rid of any cached register state */	444	/* Get rid of any cached register state */
445	deactivate_mm(tsk, mm);	445	deactivate_mm(tsk, mm);
446		446
447	/* notify parent sleeping on vfork() */	447	/* notify parent sleeping on vfork() */
448	if (vfork_done) {	448	if (vfork_done) {
449	tsk->vfork_done = NULL;	449	tsk->vfork_done = NULL;
450	complete(vfork_done);	450	complete(vfork_done);
451	}	451	}
452		452
453	/*	453	/*
454	* If we're exiting normally, clear a user-space tid field if	454	* If we're exiting normally, clear a user-space tid field if
455	* requested. We leave this alone when dying by signal, to leave	455	* requested. We leave this alone when dying by signal, to leave
456	* the value intact in a core dump, and to save the unnecessary	456	* the value intact in a core dump, and to save the unnecessary
457	* trouble otherwise. Userland only wants this done for a sys_exit.	457	* trouble otherwise. Userland only wants this done for a sys_exit.
458	*/	458	*/
459	if (tsk->clear_child_tid	459	if (tsk->clear_child_tid
460	&& !(tsk->flags & PF_SIGNALED)	460	&& !(tsk->flags & PF_SIGNALED)
461	&& atomic_read(&mm->mm_users) > 1) {	461	&& atomic_read(&mm->mm_users) > 1) {
462	u32 __user * tidptr = tsk->clear_child_tid;	462	u32 __user * tidptr = tsk->clear_child_tid;
463	tsk->clear_child_tid = NULL;	463	tsk->clear_child_tid = NULL;
464		464
465	/*	465	/*
466	* We don't check the error code - if userspace has	466	* We don't check the error code - if userspace has
467	* not set up a proper pointer then tough luck.	467	* not set up a proper pointer then tough luck.
468	*/	468	*/
469	put_user(0, tidptr);	469	put_user(0, tidptr);
470	sys_futex(tidptr, FUTEX_WAKE, 1, NULL, NULL, 0);	470	sys_futex(tidptr, FUTEX_WAKE, 1, NULL, NULL, 0);
471	}	471	}
472	}	472	}
473		473
474	/*	474	/*
475	* Allocate a new mm structure and copy contents from the	475	* Allocate a new mm structure and copy contents from the
476	* mm structure of the passed in task structure.	476	* mm structure of the passed in task structure.
477	*/	477	*/
478	static struct mm_struct dup_mm(struct task_struct tsk)	478	static struct mm_struct dup_mm(struct task_struct tsk)
479	{	479	{
480	struct mm_struct mm, oldmm = current->mm;	480	struct mm_struct mm, oldmm = current->mm;
481	int err;	481	int err;
482		482
483	if (!oldmm)	483	if (!oldmm)
484	return NULL;	484	return NULL;
485		485
486	mm = allocate_mm();	486	mm = allocate_mm();
487	if (!mm)	487	if (!mm)
488	goto fail_nomem;	488	goto fail_nomem;
489		489
490	memcpy(mm, oldmm, sizeof(*mm));	490	memcpy(mm, oldmm, sizeof(*mm));
491		491
492	/* Initializing for Swap token stuff */	492	/* Initializing for Swap token stuff */
493	mm->token_priority = 0;	493	mm->token_priority = 0;
494	mm->last_interval = 0;	494	mm->last_interval = 0;
495		495
496	if (!mm_init(mm))	496	if (!mm_init(mm))
497	goto fail_nomem;	497	goto fail_nomem;
498		498
499	if (init_new_context(tsk, mm))	499	if (init_new_context(tsk, mm))
500	goto fail_nocontext;	500	goto fail_nocontext;
501		501
502	err = dup_mmap(mm, oldmm);	502	err = dup_mmap(mm, oldmm);
503	if (err)	503	if (err)
504	goto free_pt;	504	goto free_pt;
505		505
506	mm->hiwater_rss = get_mm_rss(mm);	506	mm->hiwater_rss = get_mm_rss(mm);
507	mm->hiwater_vm = mm->total_vm;	507	mm->hiwater_vm = mm->total_vm;
508		508
509	return mm;	509	return mm;
510		510
511	free_pt:	511	free_pt:
512	mmput(mm);	512	mmput(mm);
513		513
514	fail_nomem:	514	fail_nomem:
515	return NULL;	515	return NULL;
516		516
517	fail_nocontext:	517	fail_nocontext:
518	/*	518	/*
519	* If init_new_context() failed, we cannot use mmput() to free the mm	519	* If init_new_context() failed, we cannot use mmput() to free the mm
520	* because it calls destroy_context()	520	* because it calls destroy_context()
521	*/	521	*/
522	mm_free_pgd(mm);	522	mm_free_pgd(mm);
523	free_mm(mm);	523	free_mm(mm);
524	return NULL;	524	return NULL;
525	}	525	}
526		526
527	static int copy_mm(unsigned long clone_flags, struct task_struct * tsk)	527	static int copy_mm(unsigned long clone_flags, struct task_struct * tsk)
528	{	528	{
529	struct mm_struct * mm, *oldmm;	529	struct mm_struct * mm, *oldmm;
530	int retval;	530	int retval;
531		531
532	tsk->min_flt = tsk->maj_flt = 0;	532	tsk->min_flt = tsk->maj_flt = 0;
533	tsk->nvcsw = tsk->nivcsw = 0;	533	tsk->nvcsw = tsk->nivcsw = 0;
534		534
535	tsk->mm = NULL;	535	tsk->mm = NULL;
536	tsk->active_mm = NULL;	536	tsk->active_mm = NULL;
537		537
538	/*	538	/*
539	* Are we cloning a kernel thread?	539	* Are we cloning a kernel thread?
540	*	540	*
541	* We need to steal a active VM for that..	541	* We need to steal a active VM for that..
542	*/	542	*/
543	oldmm = current->mm;	543	oldmm = current->mm;
544	if (!oldmm)	544	if (!oldmm)
545	return 0;	545	return 0;
546		546
547	if (clone_flags & CLONE_VM) {	547	if (clone_flags & CLONE_VM) {
548	atomic_inc(&oldmm->mm_users);	548	atomic_inc(&oldmm->mm_users);
549	mm = oldmm;	549	mm = oldmm;
550	goto good_mm;	550	goto good_mm;
551	}	551	}
552		552
553	retval = -ENOMEM;	553	retval = -ENOMEM;
554	mm = dup_mm(tsk);	554	mm = dup_mm(tsk);
555	if (!mm)	555	if (!mm)
556	goto fail_nomem;	556	goto fail_nomem;
557		557
558	good_mm:	558	good_mm:
559	/* Initializing for Swap token stuff */	559	/* Initializing for Swap token stuff */
560	mm->token_priority = 0;	560	mm->token_priority = 0;
561	mm->last_interval = 0;	561	mm->last_interval = 0;
562		562
563	tsk->mm = mm;	563	tsk->mm = mm;
564	tsk->active_mm = mm;	564	tsk->active_mm = mm;
565	return 0;	565	return 0;
566		566
567	fail_nomem:	567	fail_nomem:
568	return retval;	568	return retval;
569	}	569	}
570		570
571	static inline struct fs_struct __copy_fs_struct(struct fs_struct old)	571	static inline struct fs_struct __copy_fs_struct(struct fs_struct old)
572	{	572	{
573	struct fs_struct *fs = kmem_cache_alloc(fs_cachep, GFP_KERNEL);	573	struct fs_struct *fs = kmem_cache_alloc(fs_cachep, GFP_KERNEL);
574	/* We don't need to lock fs - think why ;-) */	574	/* We don't need to lock fs - think why ;-) */
575	if (fs) {	575	if (fs) {
576	atomic_set(&fs->count, 1);	576	atomic_set(&fs->count, 1);
577	rwlock_init(&fs->lock);	577	rwlock_init(&fs->lock);
578	fs->umask = old->umask;	578	fs->umask = old->umask;
579	read_lock(&old->lock);	579	read_lock(&old->lock);
580	fs->rootmnt = mntget(old->rootmnt);	580	fs->rootmnt = mntget(old->rootmnt);
581	fs->root = dget(old->root);	581	fs->root = dget(old->root);
582	fs->pwdmnt = mntget(old->pwdmnt);	582	fs->pwdmnt = mntget(old->pwdmnt);
583	fs->pwd = dget(old->pwd);	583	fs->pwd = dget(old->pwd);
584	if (old->altroot) {	584	if (old->altroot) {
585	fs->altrootmnt = mntget(old->altrootmnt);	585	fs->altrootmnt = mntget(old->altrootmnt);
586	fs->altroot = dget(old->altroot);	586	fs->altroot = dget(old->altroot);
587	} else {	587	} else {
588	fs->altrootmnt = NULL;	588	fs->altrootmnt = NULL;
589	fs->altroot = NULL;	589	fs->altroot = NULL;
590	}	590	}
591	read_unlock(&old->lock);	591	read_unlock(&old->lock);
592	}	592	}
593	return fs;	593	return fs;
594	}	594	}
595		595
596	struct fs_struct copy_fs_struct(struct fs_struct old)	596	struct fs_struct copy_fs_struct(struct fs_struct old)
597	{	597	{
598	return __copy_fs_struct(old);	598	return __copy_fs_struct(old);
599	}	599	}
600		600
601	EXPORT_SYMBOL_GPL(copy_fs_struct);	601	EXPORT_SYMBOL_GPL(copy_fs_struct);
602		602
603	static inline int copy_fs(unsigned long clone_flags, struct task_struct * tsk)	603	static inline int copy_fs(unsigned long clone_flags, struct task_struct * tsk)
604	{	604	{
605	if (clone_flags & CLONE_FS) {	605	if (clone_flags & CLONE_FS) {
606	atomic_inc(&current->fs->count);	606	atomic_inc(&current->fs->count);
607	return 0;	607	return 0;
608	}	608	}
609	tsk->fs = __copy_fs_struct(current->fs);	609	tsk->fs = __copy_fs_struct(current->fs);
610	if (!tsk->fs)	610	if (!tsk->fs)
611	return -ENOMEM;	611	return -ENOMEM;
612	return 0;	612	return 0;
613	}	613	}
614		614
615	static int count_open_files(struct fdtable *fdt)	615	static int count_open_files(struct fdtable *fdt)
616	{	616	{
617	int size = fdt->max_fds;	617	int size = fdt->max_fds;
618	int i;	618	int i;
619		619
620	/* Find the last open fd */	620	/* Find the last open fd */
621	for (i = size/(8*sizeof(long)); i > 0; ) {	621	for (i = size/(8*sizeof(long)); i > 0; ) {
622	if (fdt->open_fds->fds_bits[--i])	622	if (fdt->open_fds->fds_bits[--i])
623	break;	623	break;
624	}	624	}
625	i = (i+1) * 8 * sizeof(long);	625	i = (i+1) * 8 * sizeof(long);
626	return i;	626	return i;
627	}	627	}
628		628
629	static struct files_struct *alloc_files(void)	629	static struct files_struct *alloc_files(void)
630	{	630	{
631	struct files_struct *newf;	631	struct files_struct *newf;
632	struct fdtable *fdt;	632	struct fdtable *fdt;
633		633
634	newf = kmem_cache_alloc(files_cachep, GFP_KERNEL);	634	newf = kmem_cache_alloc(files_cachep, GFP_KERNEL);
635	if (!newf)	635	if (!newf)
636	goto out;	636	goto out;
637		637
638	atomic_set(&newf->count, 1);	638	atomic_set(&newf->count, 1);
639		639
640	spin_lock_init(&newf->file_lock);	640	spin_lock_init(&newf->file_lock);
641	newf->next_fd = 0;	641	newf->next_fd = 0;
642	fdt = &newf->fdtab;	642	fdt = &newf->fdtab;
643	fdt->max_fds = NR_OPEN_DEFAULT;	643	fdt->max_fds = NR_OPEN_DEFAULT;
644	fdt->close_on_exec = (fd_set *)&newf->close_on_exec_init;	644	fdt->close_on_exec = (fd_set *)&newf->close_on_exec_init;
645	fdt->open_fds = (fd_set *)&newf->open_fds_init;	645	fdt->open_fds = (fd_set *)&newf->open_fds_init;
646	fdt->fd = &newf->fd_array[0];	646	fdt->fd = &newf->fd_array[0];
647	INIT_RCU_HEAD(&fdt->rcu);	647	INIT_RCU_HEAD(&fdt->rcu);
648	fdt->next = NULL;	648	fdt->next = NULL;
649	rcu_assign_pointer(newf->fdt, fdt);	649	rcu_assign_pointer(newf->fdt, fdt);
650	out:	650	out:
651	return newf;	651	return newf;
652	}	652	}
653		653
654	/*	654	/*
655	* Allocate a new files structure and copy contents from the	655	* Allocate a new files structure and copy contents from the
656	* passed in files structure.	656	* passed in files structure.
657	* errorp will be valid only when the returned files_struct is NULL.	657	* errorp will be valid only when the returned files_struct is NULL.
658	*/	658	*/
659	static struct files_struct dup_fd(struct files_struct oldf, int *errorp)	659	static struct files_struct dup_fd(struct files_struct oldf, int *errorp)
660	{	660	{
661	struct files_struct *newf;	661	struct files_struct *newf;
662	struct file old_fds, new_fds;	662	struct file old_fds, new_fds;
663	int open_files, size, i;	663	int open_files, size, i;
664	struct fdtable old_fdt, new_fdt;	664	struct fdtable old_fdt, new_fdt;
665		665
666	*errorp = -ENOMEM;	666	*errorp = -ENOMEM;
667	newf = alloc_files();	667	newf = alloc_files();
668	if (!newf)	668	if (!newf)
669	goto out;	669	goto out;
670		670
671	spin_lock(&oldf->file_lock);	671	spin_lock(&oldf->file_lock);
672	old_fdt = files_fdtable(oldf);	672	old_fdt = files_fdtable(oldf);
673	new_fdt = files_fdtable(newf);	673	new_fdt = files_fdtable(newf);
674	open_files = count_open_files(old_fdt);	674	open_files = count_open_files(old_fdt);
675		675
676	/*	676	/*
677	* Check whether we need to allocate a larger fd array and fd set.	677	* Check whether we need to allocate a larger fd array and fd set.
678	* Note: we're not a clone task, so the open count won't change.	678	* Note: we're not a clone task, so the open count won't change.
679	*/	679	*/
680	if (open_files > new_fdt->max_fds) {	680	if (open_files > new_fdt->max_fds) {
681	new_fdt->max_fds = 0;	681	new_fdt->max_fds = 0;
682	spin_unlock(&oldf->file_lock);	682	spin_unlock(&oldf->file_lock);
683	spin_lock(&newf->file_lock);	683	spin_lock(&newf->file_lock);
684	*errorp = expand_files(newf, open_files-1);	684	*errorp = expand_files(newf, open_files-1);
685	spin_unlock(&newf->file_lock);	685	spin_unlock(&newf->file_lock);
686	if (*errorp < 0)	686	if (*errorp < 0)
687	goto out_release;	687	goto out_release;
688	new_fdt = files_fdtable(newf);	688	new_fdt = files_fdtable(newf);
689	/*	689	/*
690	* Reacquire the oldf lock and a pointer to its fd table	690	* Reacquire the oldf lock and a pointer to its fd table
691	* who knows it may have a new bigger fd table. We need	691	* who knows it may have a new bigger fd table. We need
692	* the latest pointer.	692	* the latest pointer.
693	*/	693	*/
694	spin_lock(&oldf->file_lock);	694	spin_lock(&oldf->file_lock);
695	old_fdt = files_fdtable(oldf);	695	old_fdt = files_fdtable(oldf);
696	}	696	}
697		697
698	old_fds = old_fdt->fd;	698	old_fds = old_fdt->fd;
699	new_fds = new_fdt->fd;	699	new_fds = new_fdt->fd;
700		700
701	memcpy(new_fdt->open_fds->fds_bits,	701	memcpy(new_fdt->open_fds->fds_bits,
702	old_fdt->open_fds->fds_bits, open_files/8);	702	old_fdt->open_fds->fds_bits, open_files/8);
703	memcpy(new_fdt->close_on_exec->fds_bits,	703	memcpy(new_fdt->close_on_exec->fds_bits,
704	old_fdt->close_on_exec->fds_bits, open_files/8);	704	old_fdt->close_on_exec->fds_bits, open_files/8);
705		705
706	for (i = open_files; i != 0; i--) {	706	for (i = open_files; i != 0; i--) {
707	struct file f = old_fds++;	707	struct file f = old_fds++;
708	if (f) {	708	if (f) {
709	get_file(f);	709	get_file(f);
710	} else {	710	} else {
711	/*	711	/*
712	* The fd may be claimed in the fd bitmap but not yet	712	* The fd may be claimed in the fd bitmap but not yet
713	* instantiated in the files array if a sibling thread	713	* instantiated in the files array if a sibling thread
714	* is partway through open(). So make sure that this	714	* is partway through open(). So make sure that this
715	* fd is available to the new process.	715	* fd is available to the new process.
716	*/	716	*/
717	FD_CLR(open_files - i, new_fdt->open_fds);	717	FD_CLR(open_files - i, new_fdt->open_fds);
718	}	718	}
719	rcu_assign_pointer(*new_fds++, f);	719	rcu_assign_pointer(*new_fds++, f);
720	}	720	}
721	spin_unlock(&oldf->file_lock);	721	spin_unlock(&oldf->file_lock);
722		722
723	/* compute the remainder to be cleared */	723	/* compute the remainder to be cleared */
724	size = (new_fdt->max_fds - open_files) * sizeof(struct file *);	724	size = (new_fdt->max_fds - open_files) * sizeof(struct file *);
725		725
726	/* This is long word aligned thus could use a optimized version */	726	/* This is long word aligned thus could use a optimized version */
727	memset(new_fds, 0, size);	727	memset(new_fds, 0, size);
728		728
729	if (new_fdt->max_fds > open_files) {	729	if (new_fdt->max_fds > open_files) {
730	int left = (new_fdt->max_fds-open_files)/8;	730	int left = (new_fdt->max_fds-open_files)/8;
731	int start = open_files / (8 * sizeof(unsigned long));	731	int start = open_files / (8 * sizeof(unsigned long));
732		732
733	memset(&new_fdt->open_fds->fds_bits[start], 0, left);	733	memset(&new_fdt->open_fds->fds_bits[start], 0, left);
734	memset(&new_fdt->close_on_exec->fds_bits[start], 0, left);	734	memset(&new_fdt->close_on_exec->fds_bits[start], 0, left);
735	}	735	}
736		736
737	return newf;	737	return newf;
738		738
739	out_release:	739	out_release:
740	kmem_cache_free(files_cachep, newf);	740	kmem_cache_free(files_cachep, newf);
741	out:	741	out:
742	return NULL;	742	return NULL;
743	}	743	}
744		744
745	static int copy_files(unsigned long clone_flags, struct task_struct * tsk)	745	static int copy_files(unsigned long clone_flags, struct task_struct * tsk)
746	{	746	{
747	struct files_struct oldf, newf;	747	struct files_struct oldf, newf;
748	int error = 0;	748	int error = 0;
749		749
750	/*	750	/*
751	* A background process may not have any files ...	751	* A background process may not have any files ...
752	*/	752	*/
753	oldf = current->files;	753	oldf = current->files;
754	if (!oldf)	754	if (!oldf)
755	goto out;	755	goto out;
756		756
757	if (clone_flags & CLONE_FILES) {	757	if (clone_flags & CLONE_FILES) {
758	atomic_inc(&oldf->count);	758	atomic_inc(&oldf->count);
759	goto out;	759	goto out;
760	}	760	}
761		761
762	/*	762	/*
763	* Note: we may be using current for both targets (See exec.c)	763	* Note: we may be using current for both targets (See exec.c)
764	* This works because we cache current->files (old) as oldf. Don't	764	* This works because we cache current->files (old) as oldf. Don't
765	* break this.	765	* break this.
766	*/	766	*/
767	tsk->files = NULL;	767	tsk->files = NULL;
768	newf = dup_fd(oldf, &error);	768	newf = dup_fd(oldf, &error);
769	if (!newf)	769	if (!newf)
770	goto out;	770	goto out;
771		771
772	tsk->files = newf;	772	tsk->files = newf;
773	error = 0;	773	error = 0;
774	out:	774	out:
775	return error;	775	return error;
776	}	776	}
777		777
778	/*	778	/*
779	* Helper to unshare the files of the current task.	779	* Helper to unshare the files of the current task.
780	* We don't want to expose copy_files internals to	780	* We don't want to expose copy_files internals to
781	* the exec layer of the kernel.	781	* the exec layer of the kernel.
782	*/	782	*/
783		783
784	int unshare_files(void)	784	int unshare_files(void)
785	{	785	{
786	struct files_struct *files = current->files;	786	struct files_struct *files = current->files;
787	int rc;	787	int rc;
788		788
789	BUG_ON(!files);	789	BUG_ON(!files);
790		790
791	/* This can race but the race causes us to copy when we don't	791	/* This can race but the race causes us to copy when we don't
792	need to and drop the copy */	792	need to and drop the copy */
793	if(atomic_read(&files->count) == 1)	793	if(atomic_read(&files->count) == 1)
794	{	794	{
795	atomic_inc(&files->count);	795	atomic_inc(&files->count);
796	return 0;	796	return 0;
797	}	797	}
798	rc = copy_files(0, current);	798	rc = copy_files(0, current);
799	if(rc)	799	if(rc)
800	current->files = files;	800	current->files = files;
801	return rc;	801	return rc;
802	}	802	}
803		803
804	EXPORT_SYMBOL(unshare_files);	804	EXPORT_SYMBOL(unshare_files);
805		805
806	static inline int copy_sighand(unsigned long clone_flags, struct task_struct * tsk)	806	static inline int copy_sighand(unsigned long clone_flags, struct task_struct * tsk)
807	{	807	{
808	struct sighand_struct *sig;	808	struct sighand_struct *sig;
809		809
810	if (clone_flags & (CLONE_SIGHAND \| CLONE_THREAD)) {	810	if (clone_flags & (CLONE_SIGHAND \| CLONE_THREAD)) {
811	atomic_inc(&current->sighand->count);	811	atomic_inc(&current->sighand->count);
812	return 0;	812	return 0;
813	}	813	}
814	sig = kmem_cache_alloc(sighand_cachep, GFP_KERNEL);	814	sig = kmem_cache_alloc(sighand_cachep, GFP_KERNEL);
815	rcu_assign_pointer(tsk->sighand, sig);	815	rcu_assign_pointer(tsk->sighand, sig);
816	if (!sig)	816	if (!sig)
817	return -ENOMEM;	817	return -ENOMEM;
818	atomic_set(&sig->count, 1);	818	atomic_set(&sig->count, 1);
819	memcpy(sig->action, current->sighand->action, sizeof(sig->action));	819	memcpy(sig->action, current->sighand->action, sizeof(sig->action));
820	return 0;	820	return 0;
821	}	821	}
822		822
823	void __cleanup_sighand(struct sighand_struct *sighand)	823	void __cleanup_sighand(struct sighand_struct *sighand)
824	{	824	{
825	if (atomic_dec_and_test(&sighand->count))	825	if (atomic_dec_and_test(&sighand->count))
826	kmem_cache_free(sighand_cachep, sighand);	826	kmem_cache_free(sighand_cachep, sighand);
827	}	827	}
828		828
829	static inline int copy_signal(unsigned long clone_flags, struct task_struct * tsk)	829	static inline int copy_signal(unsigned long clone_flags, struct task_struct * tsk)
830	{	830	{
831	struct signal_struct *sig;	831	struct signal_struct *sig;
832	int ret;	832	int ret;
833		833
834	if (clone_flags & CLONE_THREAD) {	834	if (clone_flags & CLONE_THREAD) {
835	atomic_inc(&current->signal->count);	835	atomic_inc(&current->signal->count);
836	atomic_inc(&current->signal->live);	836	atomic_inc(&current->signal->live);
837	return 0;	837	return 0;
838	}	838	}
839	sig = kmem_cache_alloc(signal_cachep, GFP_KERNEL);	839	sig = kmem_cache_alloc(signal_cachep, GFP_KERNEL);
840	tsk->signal = sig;	840	tsk->signal = sig;
841	if (!sig)	841	if (!sig)
842	return -ENOMEM;	842	return -ENOMEM;
843		843
844	ret = copy_thread_group_keys(tsk);	844	ret = copy_thread_group_keys(tsk);
845	if (ret < 0) {	845	if (ret < 0) {
846	kmem_cache_free(signal_cachep, sig);	846	kmem_cache_free(signal_cachep, sig);
847	return ret;	847	return ret;
848	}	848	}
849		849
850	atomic_set(&sig->count, 1);	850	atomic_set(&sig->count, 1);
851	atomic_set(&sig->live, 1);	851	atomic_set(&sig->live, 1);
852	init_waitqueue_head(&sig->wait_chldexit);	852	init_waitqueue_head(&sig->wait_chldexit);
853	sig->flags = 0;	853	sig->flags = 0;
854	sig->group_exit_code = 0;	854	sig->group_exit_code = 0;
855	sig->group_exit_task = NULL;	855	sig->group_exit_task = NULL;
856	sig->group_stop_count = 0;	856	sig->group_stop_count = 0;
857	sig->curr_target = NULL;	857	sig->curr_target = NULL;
858	init_sigpending(&sig->shared_pending);	858	init_sigpending(&sig->shared_pending);
859	INIT_LIST_HEAD(&sig->posix_timers);	859	INIT_LIST_HEAD(&sig->posix_timers);
860		860
861	hrtimer_init(&sig->real_timer, CLOCK_MONOTONIC, HRTIMER_REL);	861	hrtimer_init(&sig->real_timer, CLOCK_MONOTONIC, HRTIMER_REL);
862	sig->it_real_incr.tv64 = 0;	862	sig->it_real_incr.tv64 = 0;
863	sig->real_timer.function = it_real_fn;	863	sig->real_timer.function = it_real_fn;
864	sig->tsk = tsk;	864	sig->tsk = tsk;
865		865
866	sig->it_virt_expires = cputime_zero;	866	sig->it_virt_expires = cputime_zero;
867	sig->it_virt_incr = cputime_zero;	867	sig->it_virt_incr = cputime_zero;
868	sig->it_prof_expires = cputime_zero;	868	sig->it_prof_expires = cputime_zero;
869	sig->it_prof_incr = cputime_zero;	869	sig->it_prof_incr = cputime_zero;
870		870
871	sig->leader = 0; /* session leadership doesn't inherit */	871	sig->leader = 0; /* session leadership doesn't inherit */
872	sig->tty_old_pgrp = 0;	872	sig->tty_old_pgrp = 0;
873		873
874	sig->utime = sig->stime = sig->cutime = sig->cstime = cputime_zero;	874	sig->utime = sig->stime = sig->cutime = sig->cstime = cputime_zero;
875	sig->nvcsw = sig->nivcsw = sig->cnvcsw = sig->cnivcsw = 0;	875	sig->nvcsw = sig->nivcsw = sig->cnvcsw = sig->cnivcsw = 0;
876	sig->min_flt = sig->maj_flt = sig->cmin_flt = sig->cmaj_flt = 0;	876	sig->min_flt = sig->maj_flt = sig->cmin_flt = sig->cmaj_flt = 0;
877	sig->sched_time = 0;	877	sig->sched_time = 0;
878	INIT_LIST_HEAD(&sig->cpu_timers[0]);	878	INIT_LIST_HEAD(&sig->cpu_timers[0]);
879	INIT_LIST_HEAD(&sig->cpu_timers[1]);	879	INIT_LIST_HEAD(&sig->cpu_timers[1]);
880	INIT_LIST_HEAD(&sig->cpu_timers[2]);	880	INIT_LIST_HEAD(&sig->cpu_timers[2]);
881	taskstats_tgid_init(sig);	881	taskstats_tgid_init(sig);
882		882
883	task_lock(current->group_leader);	883	task_lock(current->group_leader);
884	memcpy(sig->rlim, current->signal->rlim, sizeof sig->rlim);	884	memcpy(sig->rlim, current->signal->rlim, sizeof sig->rlim);
885	task_unlock(current->group_leader);	885	task_unlock(current->group_leader);
886		886
887	if (sig->rlim[RLIMIT_CPU].rlim_cur != RLIM_INFINITY) {	887	if (sig->rlim[RLIMIT_CPU].rlim_cur != RLIM_INFINITY) {
888	/*	888	/*
889	* New sole thread in the process gets an expiry time	889	* New sole thread in the process gets an expiry time
890	* of the whole CPU time limit.	890	* of the whole CPU time limit.
891	*/	891	*/
892	tsk->it_prof_expires =	892	tsk->it_prof_expires =
893	secs_to_cputime(sig->rlim[RLIMIT_CPU].rlim_cur);	893	secs_to_cputime(sig->rlim[RLIMIT_CPU].rlim_cur);
894	}	894	}
895	acct_init_pacct(&sig->pacct);	895	acct_init_pacct(&sig->pacct);
896		896
897	return 0;	897	return 0;
898	}	898	}
899		899
900	void __cleanup_signal(struct signal_struct *sig)	900	void __cleanup_signal(struct signal_struct *sig)
901	{	901	{
902	exit_thread_group_keys(sig);	902	exit_thread_group_keys(sig);
903	kmem_cache_free(signal_cachep, sig);	903	kmem_cache_free(signal_cachep, sig);
904	}	904	}
905		905
906	static inline void cleanup_signal(struct task_struct *tsk)	906	static inline void cleanup_signal(struct task_struct *tsk)
907	{	907	{
908	struct signal_struct *sig = tsk->signal;	908	struct signal_struct *sig = tsk->signal;
909		909
910	atomic_dec(&sig->live);	910	atomic_dec(&sig->live);
911		911
912	if (atomic_dec_and_test(&sig->count))	912	if (atomic_dec_and_test(&sig->count))
913	__cleanup_signal(sig);	913	__cleanup_signal(sig);
914	}	914	}
915		915
916	static inline void copy_flags(unsigned long clone_flags, struct task_struct *p)	916	static inline void copy_flags(unsigned long clone_flags, struct task_struct *p)
917	{	917	{
918	unsigned long new_flags = p->flags;	918	unsigned long new_flags = p->flags;
919		919
920	new_flags &= ~(PF_SUPERPRIV \| PF_NOFREEZE);	920	new_flags &= ~(PF_SUPERPRIV \| PF_NOFREEZE);
921	new_flags \|= PF_FORKNOEXEC;	921	new_flags \|= PF_FORKNOEXEC;
922	if (!(clone_flags & CLONE_PTRACE))	922	if (!(clone_flags & CLONE_PTRACE))
923	p->ptrace = 0;	923	p->ptrace = 0;
924	p->flags = new_flags;	924	p->flags = new_flags;
925	}	925	}
926		926
927	asmlinkage long sys_set_tid_address(int __user *tidptr)	927	asmlinkage long sys_set_tid_address(int __user *tidptr)
928	{	928	{
929	current->clear_child_tid = tidptr;	929	current->clear_child_tid = tidptr;
930		930
931	return current->pid;	931	return current->pid;
932	}	932	}
933		933
934	static inline void rt_mutex_init_task(struct task_struct *p)	934	static inline void rt_mutex_init_task(struct task_struct *p)
935	{	935	{
936	#ifdef CONFIG_RT_MUTEXES	936	#ifdef CONFIG_RT_MUTEXES
937	spin_lock_init(&p->pi_lock);	937	spin_lock_init(&p->pi_lock);
938	plist_head_init(&p->pi_waiters, &p->pi_lock);	938	plist_head_init(&p->pi_waiters, &p->pi_lock);
939	p->pi_blocked_on = NULL;	939	p->pi_blocked_on = NULL;
940	#endif	940	#endif
941	}	941	}
942		942
943	/*	943	/*
944	* This creates a new process as a copy of the old one,	944	* This creates a new process as a copy of the old one,
945	* but does not actually start it yet.	945	* but does not actually start it yet.
946	*	946	*
947	* It copies the registers, and all the appropriate	947	* It copies the registers, and all the appropriate
948	* parts of the process environment (as per the clone	948	* parts of the process environment (as per the clone
949	* flags). The actual kick-off is left to the caller.	949	* flags). The actual kick-off is left to the caller.
950	*/	950	*/
951	static struct task_struct *copy_process(unsigned long clone_flags,	951	static struct task_struct *copy_process(unsigned long clone_flags,
952	unsigned long stack_start,	952	unsigned long stack_start,
953	struct pt_regs *regs,	953	struct pt_regs *regs,
954	unsigned long stack_size,	954	unsigned long stack_size,
955	int __user *parent_tidptr,	955	int __user *parent_tidptr,
956	int __user *child_tidptr,	956	int __user *child_tidptr,
957	int pid)	957	int pid)
958	{	958	{
959	int retval;	959	int retval;
960	struct task_struct *p = NULL;	960	struct task_struct *p = NULL;
961		961
962	if ((clone_flags & (CLONE_NEWNS\|CLONE_FS)) == (CLONE_NEWNS\|CLONE_FS))	962	if ((clone_flags & (CLONE_NEWNS\|CLONE_FS)) == (CLONE_NEWNS\|CLONE_FS))
963	return ERR_PTR(-EINVAL);	963	return ERR_PTR(-EINVAL);
964		964
965	/*	965	/*
966	* Thread groups must share signals as well, and detached threads	966	* Thread groups must share signals as well, and detached threads
967	* can only be started up within the thread group.	967	* can only be started up within the thread group.
968	*/	968	*/
969	if ((clone_flags & CLONE_THREAD) && !(clone_flags & CLONE_SIGHAND))	969	if ((clone_flags & CLONE_THREAD) && !(clone_flags & CLONE_SIGHAND))
970	return ERR_PTR(-EINVAL);	970	return ERR_PTR(-EINVAL);
971		971
972	/*	972	/*
973	* Shared signal handlers imply shared VM. By way of the above,	973	* Shared signal handlers imply shared VM. By way of the above,
974	* thread groups also imply shared VM. Blocking this case allows	974	* thread groups also imply shared VM. Blocking this case allows
975	* for various simplifications in other code.	975	* for various simplifications in other code.
976	*/	976	*/
977	if ((clone_flags & CLONE_SIGHAND) && !(clone_flags & CLONE_VM))	977	if ((clone_flags & CLONE_SIGHAND) && !(clone_flags & CLONE_VM))
978	return ERR_PTR(-EINVAL);	978	return ERR_PTR(-EINVAL);
979		979
980	retval = security_task_create(clone_flags);	980	retval = security_task_create(clone_flags);
981	if (retval)	981	if (retval)
982	goto fork_out;	982	goto fork_out;
983		983
984	retval = -ENOMEM;	984	retval = -ENOMEM;
985	p = dup_task_struct(current);	985	p = dup_task_struct(current);
986	if (!p)	986	if (!p)
987	goto fork_out;	987	goto fork_out;
988		988
989	rt_mutex_init_task(p);	989	rt_mutex_init_task(p);
990		990
991	#ifdef CONFIG_TRACE_IRQFLAGS	991	#ifdef CONFIG_TRACE_IRQFLAGS
992	DEBUG_LOCKS_WARN_ON(!p->hardirqs_enabled);	992	DEBUG_LOCKS_WARN_ON(!p->hardirqs_enabled);
993	DEBUG_LOCKS_WARN_ON(!p->softirqs_enabled);	993	DEBUG_LOCKS_WARN_ON(!p->softirqs_enabled);
994	#endif	994	#endif
995	retval = -EAGAIN;	995	retval = -EAGAIN;
996	if (atomic_read(&p->user->processes) >=	996	if (atomic_read(&p->user->processes) >=
997	p->signal->rlim[RLIMIT_NPROC].rlim_cur) {	997	p->signal->rlim[RLIMIT_NPROC].rlim_cur) {
998	if (!capable(CAP_SYS_ADMIN) && !capable(CAP_SYS_RESOURCE) &&	998	if (!capable(CAP_SYS_ADMIN) && !capable(CAP_SYS_RESOURCE) &&
999	p->user != &root_user)	999	p->user != &root_user)
1000	goto bad_fork_free;	1000	goto bad_fork_free;
1001	}	1001	}
1002		1002
1003	atomic_inc(&p->user->__count);	1003	atomic_inc(&p->user->__count);
1004	atomic_inc(&p->user->processes);	1004	atomic_inc(&p->user->processes);
1005	get_group_info(p->group_info);	1005	get_group_info(p->group_info);
1006		1006
1007	/*	1007	/*
1008	* If multiple threads are within copy_process(), then this check	1008	* If multiple threads are within copy_process(), then this check
1009	* triggers too late. This doesn't hurt, the check is only there	1009	* triggers too late. This doesn't hurt, the check is only there
1010	* to stop root fork bombs.	1010	* to stop root fork bombs.
1011	*/	1011	*/
1012	if (nr_threads >= max_threads)	1012	if (nr_threads >= max_threads)
1013	goto bad_fork_cleanup_count;	1013	goto bad_fork_cleanup_count;
1014		1014
1015	if (!try_module_get(task_thread_info(p)->exec_domain->module))	1015	if (!try_module_get(task_thread_info(p)->exec_domain->module))
1016	goto bad_fork_cleanup_count;	1016	goto bad_fork_cleanup_count;
1017		1017
1018	if (p->binfmt && !try_module_get(p->binfmt->module))	1018	if (p->binfmt && !try_module_get(p->binfmt->module))
1019	goto bad_fork_cleanup_put_domain;	1019	goto bad_fork_cleanup_put_domain;
1020		1020
1021	p->did_exec = 0;	1021	p->did_exec = 0;
1022	delayacct_tsk_init(p); /* Must remain after dup_task_struct() */	1022	delayacct_tsk_init(p); /* Must remain after dup_task_struct() */
1023	copy_flags(clone_flags, p);	1023	copy_flags(clone_flags, p);
1024	p->pid = pid;	1024	p->pid = pid;
1025	retval = -EFAULT;	1025	retval = -EFAULT;
1026	if (clone_flags & CLONE_PARENT_SETTID)	1026	if (clone_flags & CLONE_PARENT_SETTID)
1027	if (put_user(p->pid, parent_tidptr))	1027	if (put_user(p->pid, parent_tidptr))
1028	goto bad_fork_cleanup_delays_binfmt;	1028	goto bad_fork_cleanup_delays_binfmt;
1029		1029
1030	INIT_LIST_HEAD(&p->children);	1030	INIT_LIST_HEAD(&p->children);
1031	INIT_LIST_HEAD(&p->sibling);	1031	INIT_LIST_HEAD(&p->sibling);
1032	p->vfork_done = NULL;	1032	p->vfork_done = NULL;
1033	spin_lock_init(&p->alloc_lock);	1033	spin_lock_init(&p->alloc_lock);
1034		1034
1035	clear_tsk_thread_flag(p, TIF_SIGPENDING);	1035	clear_tsk_thread_flag(p, TIF_SIGPENDING);
1036	init_sigpending(&p->pending);	1036	init_sigpending(&p->pending);
1037		1037
1038	p->utime = cputime_zero;	1038	p->utime = cputime_zero;
1039	p->stime = cputime_zero;	1039	p->stime = cputime_zero;
1040	p->sched_time = 0;	1040	p->sched_time = 0;
1041	p->rchar = 0; /* I/O counter: bytes read */	1041	p->rchar = 0; /* I/O counter: bytes read */
1042	p->wchar = 0; /* I/O counter: bytes written */	1042	p->wchar = 0; /* I/O counter: bytes written */
1043	p->syscr = 0; /* I/O counter: read syscalls */	1043	p->syscr = 0; /* I/O counter: read syscalls */
1044	p->syscw = 0; /* I/O counter: write syscalls */	1044	p->syscw = 0; /* I/O counter: write syscalls */
1045	task_io_accounting_init(p);	1045	task_io_accounting_init(p);
1046	acct_clear_integrals(p);	1046	acct_clear_integrals(p);
1047		1047
1048	p->it_virt_expires = cputime_zero;	1048	p->it_virt_expires = cputime_zero;
1049	p->it_prof_expires = cputime_zero;	1049	p->it_prof_expires = cputime_zero;
1050	p->it_sched_expires = 0;	1050	p->it_sched_expires = 0;
1051	INIT_LIST_HEAD(&p->cpu_timers[0]);	1051	INIT_LIST_HEAD(&p->cpu_timers[0]);
1052	INIT_LIST_HEAD(&p->cpu_timers[1]);	1052	INIT_LIST_HEAD(&p->cpu_timers[1]);
1053	INIT_LIST_HEAD(&p->cpu_timers[2]);	1053	INIT_LIST_HEAD(&p->cpu_timers[2]);
1054		1054
1055	p->lock_depth = -1; /* -1 = no lock */	1055	p->lock_depth = -1; /* -1 = no lock */
1056	do_posix_clock_monotonic_gettime(&p->start_time);	1056	do_posix_clock_monotonic_gettime(&p->start_time);
1057	p->security = NULL;	1057	p->security = NULL;
1058	p->io_context = NULL;	1058	p->io_context = NULL;
1059	p->io_wait = NULL;	1059	p->io_wait = NULL;
1060	p->audit_context = NULL;	1060	p->audit_context = NULL;
1061	cpuset_fork(p);	1061	cpuset_fork(p);
1062	#ifdef CONFIG_NUMA	1062	#ifdef CONFIG_NUMA
1063	p->mempolicy = mpol_copy(p->mempolicy);	1063	p->mempolicy = mpol_copy(p->mempolicy);
1064	if (IS_ERR(p->mempolicy)) {	1064	if (IS_ERR(p->mempolicy)) {
1065	retval = PTR_ERR(p->mempolicy);	1065	retval = PTR_ERR(p->mempolicy);
1066	p->mempolicy = NULL;	1066	p->mempolicy = NULL;
1067	goto bad_fork_cleanup_cpuset;	1067	goto bad_fork_cleanup_cpuset;
1068	}	1068	}
1069	mpol_fix_fork_child_flag(p);	1069	mpol_fix_fork_child_flag(p);
1070	#endif	1070	#endif
1071	#ifdef CONFIG_TRACE_IRQFLAGS	1071	#ifdef CONFIG_TRACE_IRQFLAGS
1072	p->irq_events = 0;	1072	p->irq_events = 0;
1073	#ifdef __ARCH_WANT_INTERRUPTS_ON_CTXSW	1073	#ifdef __ARCH_WANT_INTERRUPTS_ON_CTXSW
1074	p->hardirqs_enabled = 1;	1074	p->hardirqs_enabled = 1;
1075	#else	1075	#else
1076	p->hardirqs_enabled = 0;	1076	p->hardirqs_enabled = 0;
1077	#endif	1077	#endif
1078	p->hardirq_enable_ip = 0;	1078	p->hardirq_enable_ip = 0;
1079	p->hardirq_enable_event = 0;	1079	p->hardirq_enable_event = 0;
1080	p->hardirq_disable_ip = _THIS_IP_;	1080	p->hardirq_disable_ip = _THIS_IP_;
1081	p->hardirq_disable_event = 0;	1081	p->hardirq_disable_event = 0;
1082	p->softirqs_enabled = 1;	1082	p->softirqs_enabled = 1;
1083	p->softirq_enable_ip = _THIS_IP_;	1083	p->softirq_enable_ip = _THIS_IP_;
1084	p->softirq_enable_event = 0;	1084	p->softirq_enable_event = 0;
1085	p->softirq_disable_ip = 0;	1085	p->softirq_disable_ip = 0;
1086	p->softirq_disable_event = 0;	1086	p->softirq_disable_event = 0;
1087	p->hardirq_context = 0;	1087	p->hardirq_context = 0;
1088	p->softirq_context = 0;	1088	p->softirq_context = 0;
1089	#endif	1089	#endif
1090	#ifdef CONFIG_LOCKDEP	1090	#ifdef CONFIG_LOCKDEP
1091	p->lockdep_depth = 0; /* no locks held yet */	1091	p->lockdep_depth = 0; /* no locks held yet */
1092	p->curr_chain_key = 0;	1092	p->curr_chain_key = 0;
1093	p->lockdep_recursion = 0;	1093	p->lockdep_recursion = 0;
1094	#endif	1094	#endif
1095		1095
1096	#ifdef CONFIG_DEBUG_MUTEXES	1096	#ifdef CONFIG_DEBUG_MUTEXES
1097	p->blocked_on = NULL; /* not blocked yet */	1097	p->blocked_on = NULL; /* not blocked yet */
1098	#endif	1098	#endif
1099		1099
1100	p->tgid = p->pid;	1100	p->tgid = p->pid;
1101	if (clone_flags & CLONE_THREAD)	1101	if (clone_flags & CLONE_THREAD)
1102	p->tgid = current->tgid;	1102	p->tgid = current->tgid;
1103		1103
1104	if ((retval = security_task_alloc(p)))	1104	if ((retval = security_task_alloc(p)))
1105	goto bad_fork_cleanup_policy;	1105	goto bad_fork_cleanup_policy;
1106	if ((retval = audit_alloc(p)))	1106	if ((retval = audit_alloc(p)))
1107	goto bad_fork_cleanup_security;	1107	goto bad_fork_cleanup_security;
1108	/* copy all the process information */	1108	/* copy all the process information */
1109	if ((retval = copy_semundo(clone_flags, p)))	1109	if ((retval = copy_semundo(clone_flags, p)))
1110	goto bad_fork_cleanup_audit;	1110	goto bad_fork_cleanup_audit;
1111	if ((retval = copy_files(clone_flags, p)))	1111	if ((retval = copy_files(clone_flags, p)))
1112	goto bad_fork_cleanup_semundo;	1112	goto bad_fork_cleanup_semundo;
1113	if ((retval = copy_fs(clone_flags, p)))	1113	if ((retval = copy_fs(clone_flags, p)))
1114	goto bad_fork_cleanup_files;	1114	goto bad_fork_cleanup_files;
1115	if ((retval = copy_sighand(clone_flags, p)))	1115	if ((retval = copy_sighand(clone_flags, p)))
1116	goto bad_fork_cleanup_fs;	1116	goto bad_fork_cleanup_fs;
1117	if ((retval = copy_signal(clone_flags, p)))	1117	if ((retval = copy_signal(clone_flags, p)))
1118	goto bad_fork_cleanup_sighand;	1118	goto bad_fork_cleanup_sighand;
1119	if ((retval = copy_mm(clone_flags, p)))	1119	if ((retval = copy_mm(clone_flags, p)))
1120	goto bad_fork_cleanup_signal;	1120	goto bad_fork_cleanup_signal;
1121	if ((retval = copy_keys(clone_flags, p)))	1121	if ((retval = copy_keys(clone_flags, p)))
1122	goto bad_fork_cleanup_mm;	1122	goto bad_fork_cleanup_mm;
1123	if ((retval = copy_namespaces(clone_flags, p)))	1123	if ((retval = copy_namespaces(clone_flags, p)))
1124	goto bad_fork_cleanup_keys;	1124	goto bad_fork_cleanup_keys;
1125	retval = copy_thread(0, clone_flags, stack_start, stack_size, p, regs);	1125	retval = copy_thread(0, clone_flags, stack_start, stack_size, p, regs);
1126	if (retval)	1126	if (retval)
1127	goto bad_fork_cleanup_namespaces;	1127	goto bad_fork_cleanup_namespaces;
1128		1128
1129	p->set_child_tid = (clone_flags & CLONE_CHILD_SETTID) ? child_tidptr : NULL;	1129	p->set_child_tid = (clone_flags & CLONE_CHILD_SETTID) ? child_tidptr : NULL;
1130	/*	1130	/*
1131	* Clear TID on mm_release()?	1131	* Clear TID on mm_release()?
1132	*/	1132	*/
1133	p->clear_child_tid = (clone_flags & CLONE_CHILD_CLEARTID) ? child_tidptr: NULL;	1133	p->clear_child_tid = (clone_flags & CLONE_CHILD_CLEARTID) ? child_tidptr: NULL;
1134	p->robust_list = NULL;	1134	p->robust_list = NULL;
1135	#ifdef CONFIG_COMPAT	1135	#ifdef CONFIG_COMPAT
1136	p->compat_robust_list = NULL;	1136	p->compat_robust_list = NULL;
1137	#endif	1137	#endif
1138	INIT_LIST_HEAD(&p->pi_state_list);	1138	INIT_LIST_HEAD(&p->pi_state_list);
1139	p->pi_state_cache = NULL;	1139	p->pi_state_cache = NULL;
1140		1140
1141	/*	1141	/*
1142	* sigaltstack should be cleared when sharing the same VM	1142	* sigaltstack should be cleared when sharing the same VM
1143	*/	1143	*/
1144	if ((clone_flags & (CLONE_VM\|CLONE_VFORK)) == CLONE_VM)	1144	if ((clone_flags & (CLONE_VM\|CLONE_VFORK)) == CLONE_VM)
1145	p->sas_ss_sp = p->sas_ss_size = 0;	1145	p->sas_ss_sp = p->sas_ss_size = 0;
1146		1146
1147	/*	1147	/*
1148	* Syscall tracing should be turned off in the child regardless	1148	* Syscall tracing should be turned off in the child regardless
1149	* of CLONE_PTRACE.	1149	* of CLONE_PTRACE.
1150	*/	1150	*/
1151	clear_tsk_thread_flag(p, TIF_SYSCALL_TRACE);	1151	clear_tsk_thread_flag(p, TIF_SYSCALL_TRACE);
1152	#ifdef TIF_SYSCALL_EMU	1152	#ifdef TIF_SYSCALL_EMU
1153	clear_tsk_thread_flag(p, TIF_SYSCALL_EMU);	1153	clear_tsk_thread_flag(p, TIF_SYSCALL_EMU);
1154	#endif	1154	#endif
1155		1155
1156	/* Our parent execution domain becomes current domain	1156	/* Our parent execution domain becomes current domain
1157	These must match for thread signalling to apply */	1157	These must match for thread signalling to apply */
1158	p->parent_exec_id = p->self_exec_id;	1158	p->parent_exec_id = p->self_exec_id;
1159		1159
1160	/* ok, now we should be set up.. */	1160	/* ok, now we should be set up.. */
1161	p->exit_signal = (clone_flags & CLONE_THREAD) ? -1 : (clone_flags & CSIGNAL);	1161	p->exit_signal = (clone_flags & CLONE_THREAD) ? -1 : (clone_flags & CSIGNAL);
1162	p->pdeath_signal = 0;	1162	p->pdeath_signal = 0;
1163	p->exit_state = 0;	1163	p->exit_state = 0;
1164		1164
1165	/*	1165	/*
1166	* Ok, make it visible to the rest of the system.	1166	* Ok, make it visible to the rest of the system.
1167	* We dont wake it up yet.	1167	* We dont wake it up yet.
1168	*/	1168	*/
1169	p->group_leader = p;	1169	p->group_leader = p;
1170	INIT_LIST_HEAD(&p->thread_group);	1170	INIT_LIST_HEAD(&p->thread_group);
1171	INIT_LIST_HEAD(&p->ptrace_children);	1171	INIT_LIST_HEAD(&p->ptrace_children);
1172	INIT_LIST_HEAD(&p->ptrace_list);	1172	INIT_LIST_HEAD(&p->ptrace_list);
1173		1173
1174	/* Perform scheduler related setup. Assign this task to a CPU. */	1174	/* Perform scheduler related setup. Assign this task to a CPU. */
1175	sched_fork(p, clone_flags);	1175	sched_fork(p, clone_flags);
1176		1176
1177	/* Need tasklist lock for parent etc handling! */	1177	/* Need tasklist lock for parent etc handling! */
1178	write_lock_irq(&tasklist_lock);	1178	write_lock_irq(&tasklist_lock);
1179		1179
1180	/* for sys_ioprio_set(IOPRIO_WHO_PGRP) */	1180	/* for sys_ioprio_set(IOPRIO_WHO_PGRP) */
1181	p->ioprio = current->ioprio;	1181	p->ioprio = current->ioprio;
1182		1182
1183	/*	1183	/*
1184	* The task hasn't been attached yet, so its cpus_allowed mask will	1184	* The task hasn't been attached yet, so its cpus_allowed mask will
1185	* not be changed, nor will its assigned CPU.	1185	* not be changed, nor will its assigned CPU.
1186	*	1186	*
1187	* The cpus_allowed mask of the parent may have changed after it was	1187	* The cpus_allowed mask of the parent may have changed after it was
1188	* copied first time - so re-copy it here, then check the child's CPU	1188	* copied first time - so re-copy it here, then check the child's CPU
1189	* to ensure it is on a valid CPU (and if not, just force it back to	1189	* to ensure it is on a valid CPU (and if not, just force it back to
1190	* parent's CPU). This avoids alot of nasty races.	1190	* parent's CPU). This avoids alot of nasty races.
1191	*/	1191	*/
1192	p->cpus_allowed = current->cpus_allowed;	1192	p->cpus_allowed = current->cpus_allowed;
1193	if (unlikely(!cpu_isset(task_cpu(p), p->cpus_allowed) \|\|	1193	if (unlikely(!cpu_isset(task_cpu(p), p->cpus_allowed) \|\|
1194	!cpu_online(task_cpu(p))))	1194	!cpu_online(task_cpu(p))))
1195	set_task_cpu(p, smp_processor_id());	1195	set_task_cpu(p, smp_processor_id());
1196		1196
1197	/* CLONE_PARENT re-uses the old parent */	1197	/* CLONE_PARENT re-uses the old parent */
1198	if (clone_flags & (CLONE_PARENT\|CLONE_THREAD))	1198	if (clone_flags & (CLONE_PARENT\|CLONE_THREAD))
1199	p->real_parent = current->real_parent;	1199	p->real_parent = current->real_parent;
1200	else	1200	else
1201	p->real_parent = current;	1201	p->real_parent = current;
1202	p->parent = p->real_parent;	1202	p->parent = p->real_parent;
1203		1203
1204	spin_lock(&current->sighand->siglock);	1204	spin_lock(&current->sighand->siglock);
1205		1205
1206	/*	1206	/*
1207	* Process group and session signals need to be delivered to just the	1207	* Process group and session signals need to be delivered to just the
1208	* parent before the fork or both the parent and the child after the	1208	* parent before the fork or both the parent and the child after the
1209	* fork. Restart if a signal comes in before we add the new process to	1209	* fork. Restart if a signal comes in before we add the new process to
1210	* it's process group.	1210	* it's process group.
1211	* A fatal signal pending means that current will exit, so the new	1211	* A fatal signal pending means that current will exit, so the new
1212	* thread can't slip out of an OOM kill (or normal SIGKILL).	1212	* thread can't slip out of an OOM kill (or normal SIGKILL).
1213	*/	1213	*/
1214	recalc_sigpending();	1214	recalc_sigpending();
1215	if (signal_pending(current)) {	1215	if (signal_pending(current)) {
1216	spin_unlock(&current->sighand->siglock);	1216	spin_unlock(&current->sighand->siglock);
1217	write_unlock_irq(&tasklist_lock);	1217	write_unlock_irq(&tasklist_lock);
1218	retval = -ERESTARTNOINTR;	1218	retval = -ERESTARTNOINTR;
1219	goto bad_fork_cleanup_namespaces;	1219	goto bad_fork_cleanup_namespaces;
1220	}	1220	}
1221		1221
1222	if (clone_flags & CLONE_THREAD) {	1222	if (clone_flags & CLONE_THREAD) {
1223	p->group_leader = current->group_leader;	1223	p->group_leader = current->group_leader;
1224	list_add_tail_rcu(&p->thread_group, &p->group_leader->thread_group);	1224	list_add_tail_rcu(&p->thread_group, &p->group_leader->thread_group);
1225		1225
1226	if (!cputime_eq(current->signal->it_virt_expires,	1226	if (!cputime_eq(current->signal->it_virt_expires,
1227	cputime_zero) \|\|	1227	cputime_zero) \|\|
1228	!cputime_eq(current->signal->it_prof_expires,	1228	!cputime_eq(current->signal->it_prof_expires,
1229	cputime_zero) \|\|	1229	cputime_zero) \|\|
1230	current->signal->rlim[RLIMIT_CPU].rlim_cur != RLIM_INFINITY \|\|	1230	current->signal->rlim[RLIMIT_CPU].rlim_cur != RLIM_INFINITY \|\|
1231	!list_empty(&current->signal->cpu_timers[0]) \|\|	1231	!list_empty(&current->signal->cpu_timers[0]) \|\|
1232	!list_empty(&current->signal->cpu_timers[1]) \|\|	1232	!list_empty(&current->signal->cpu_timers[1]) \|\|
1233	!list_empty(&current->signal->cpu_timers[2])) {	1233	!list_empty(&current->signal->cpu_timers[2])) {
1234	/*	1234	/*
1235	* Have child wake up on its first tick to check	1235	* Have child wake up on its first tick to check
1236	* for process CPU timers.	1236	* for process CPU timers.
1237	*/	1237	*/
1238	p->it_prof_expires = jiffies_to_cputime(1);	1238	p->it_prof_expires = jiffies_to_cputime(1);
1239	}	1239	}
1240	}	1240	}
1241		1241
1242	if (likely(p->pid)) {	1242	if (likely(p->pid)) {
1243	add_parent(p);	1243	add_parent(p);
1244	if (unlikely(p->ptrace & PT_PTRACED))	1244	if (unlikely(p->ptrace & PT_PTRACED))
1245	__ptrace_link(p, current->parent);	1245	__ptrace_link(p, current->parent);
1246		1246
1247	if (thread_group_leader(p)) {	1247	if (thread_group_leader(p)) {
1248	p->signal->tty = current->signal->tty;	1248	p->signal->tty = current->signal->tty;
1249	p->signal->pgrp = process_group(current);	1249	p->signal->pgrp = process_group(current);
1250	set_signal_session(p->signal, process_session(current));	1250	set_signal_session(p->signal, process_session(current));
1251	attach_pid(p, PIDTYPE_PGID, process_group(p));	1251	attach_pid(p, PIDTYPE_PGID, process_group(p));
1252	attach_pid(p, PIDTYPE_SID, process_session(p));	1252	attach_pid(p, PIDTYPE_SID, process_session(p));
1253		1253
1254	list_add_tail_rcu(&p->tasks, &init_task.tasks);	1254	list_add_tail_rcu(&p->tasks, &init_task.tasks);
1255	__get_cpu_var(process_counts)++;	1255	__get_cpu_var(process_counts)++;
1256	}	1256	}
1257	attach_pid(p, PIDTYPE_PID, p->pid);	1257	attach_pid(p, PIDTYPE_PID, p->pid);
1258	nr_threads++;	1258	nr_threads++;
1259	}	1259	}
1260		1260
1261	total_forks++;	1261	total_forks++;
1262	spin_unlock(&current->sighand->siglock);	1262	spin_unlock(&current->sighand->siglock);
1263	write_unlock_irq(&tasklist_lock);	1263	write_unlock_irq(&tasklist_lock);
1264	proc_fork_connector(p);	1264	proc_fork_connector(p);
1265	return p;	1265	return p;
1266		1266
1267	bad_fork_cleanup_namespaces:	1267	bad_fork_cleanup_namespaces:
1268	exit_task_namespaces(p);	1268	exit_task_namespaces(p);
1269	bad_fork_cleanup_keys:	1269	bad_fork_cleanup_keys:
1270	exit_keys(p);	1270	exit_keys(p);
1271	bad_fork_cleanup_mm:	1271	bad_fork_cleanup_mm:
1272	if (p->mm)	1272	if (p->mm)
1273	mmput(p->mm);	1273	mmput(p->mm);
1274	bad_fork_cleanup_signal:	1274	bad_fork_cleanup_signal:
1275	cleanup_signal(p);	1275	cleanup_signal(p);
1276	bad_fork_cleanup_sighand:	1276	bad_fork_cleanup_sighand:
1277	__cleanup_sighand(p->sighand);	1277	__cleanup_sighand(p->sighand);
1278	bad_fork_cleanup_fs:	1278	bad_fork_cleanup_fs:
1279	exit_fs(p); /* blocking */	1279	exit_fs(p); /* blocking */
1280	bad_fork_cleanup_files:	1280	bad_fork_cleanup_files:
1281	exit_files(p); /* blocking */	1281	exit_files(p); /* blocking */
1282	bad_fork_cleanup_semundo:	1282	bad_fork_cleanup_semundo:
1283	exit_sem(p);	1283	exit_sem(p);
1284	bad_fork_cleanup_audit:	1284	bad_fork_cleanup_audit:
1285	audit_free(p);	1285	audit_free(p);
1286	bad_fork_cleanup_security:	1286	bad_fork_cleanup_security:
1287	security_task_free(p);	1287	security_task_free(p);
1288	bad_fork_cleanup_policy:	1288	bad_fork_cleanup_policy:
1289	#ifdef CONFIG_NUMA	1289	#ifdef CONFIG_NUMA
1290	mpol_free(p->mempolicy);	1290	mpol_free(p->mempolicy);
1291	bad_fork_cleanup_cpuset:	1291	bad_fork_cleanup_cpuset:
1292	#endif	1292	#endif
1293	cpuset_exit(p);	1293	cpuset_exit(p);
1294	bad_fork_cleanup_delays_binfmt:	1294	bad_fork_cleanup_delays_binfmt:
1295	delayacct_tsk_free(p);	1295	delayacct_tsk_free(p);
1296	if (p->binfmt)	1296	if (p->binfmt)
1297	module_put(p->binfmt->module);	1297	module_put(p->binfmt->module);
1298	bad_fork_cleanup_put_domain:	1298	bad_fork_cleanup_put_domain:
1299	module_put(task_thread_info(p)->exec_domain->module);	1299	module_put(task_thread_info(p)->exec_domain->module);
1300	bad_fork_cleanup_count:	1300	bad_fork_cleanup_count:
1301	put_group_info(p->group_info);	1301	put_group_info(p->group_info);
1302	atomic_dec(&p->user->processes);	1302	atomic_dec(&p->user->processes);
1303	free_uid(p->user);	1303	free_uid(p->user);
1304	bad_fork_free:	1304	bad_fork_free:
1305	free_task(p);	1305	free_task(p);
1306	fork_out:	1306	fork_out:
1307	return ERR_PTR(retval);	1307	return ERR_PTR(retval);
1308	}	1308	}
1309		1309
1310	noinline struct pt_regs * __devinit __attribute__((weak)) idle_regs(struct pt_regs *regs)	1310	noinline struct pt_regs * __devinit __attribute__((weak)) idle_regs(struct pt_regs *regs)
1311	{	1311	{
1312	memset(regs, 0, sizeof(struct pt_regs));	1312	memset(regs, 0, sizeof(struct pt_regs));
1313	return regs;	1313	return regs;
1314	}	1314	}
1315		1315
1316	struct task_struct * __devinit fork_idle(int cpu)	1316	struct task_struct * __devinit fork_idle(int cpu)
1317	{	1317	{
1318	struct task_struct *task;	1318	struct task_struct *task;
1319	struct pt_regs regs;	1319	struct pt_regs regs;
1320		1320
1321	task = copy_process(CLONE_VM, 0, idle_regs(&regs), 0, NULL, NULL, 0);	1321	task = copy_process(CLONE_VM, 0, idle_regs(&regs), 0, NULL, NULL, 0);
1322	if (!IS_ERR(task))	1322	if (!IS_ERR(task))
1323	init_idle(task, cpu);	1323	init_idle(task, cpu);
1324		1324
1325	return task;	1325	return task;
1326	}	1326	}
1327		1327
1328	static inline int fork_traceflag (unsigned clone_flags)	1328	static inline int fork_traceflag (unsigned clone_flags)
1329	{	1329	{
1330	if (clone_flags & CLONE_UNTRACED)	1330	if (clone_flags & CLONE_UNTRACED)
1331	return 0;	1331	return 0;
1332	else if (clone_flags & CLONE_VFORK) {	1332	else if (clone_flags & CLONE_VFORK) {
1333	if (current->ptrace & PT_TRACE_VFORK)	1333	if (current->ptrace & PT_TRACE_VFORK)
1334	return PTRACE_EVENT_VFORK;	1334	return PTRACE_EVENT_VFORK;
1335	} else if ((clone_flags & CSIGNAL) != SIGCHLD) {	1335	} else if ((clone_flags & CSIGNAL) != SIGCHLD) {
1336	if (current->ptrace & PT_TRACE_CLONE)	1336	if (current->ptrace & PT_TRACE_CLONE)
1337	return PTRACE_EVENT_CLONE;	1337	return PTRACE_EVENT_CLONE;
1338	} else if (current->ptrace & PT_TRACE_FORK)	1338	} else if (current->ptrace & PT_TRACE_FORK)
1339	return PTRACE_EVENT_FORK;	1339	return PTRACE_EVENT_FORK;
1340		1340
1341	return 0;	1341	return 0;
1342	}	1342	}
1343		1343
1344	/*	1344	/*
1345	* Ok, this is the main fork-routine.	1345	* Ok, this is the main fork-routine.
1346	*	1346	*
1347	* It copies the process, and if successful kick-starts	1347	* It copies the process, and if successful kick-starts
1348	* it and waits for it to finish using the VM if required.	1348	* it and waits for it to finish using the VM if required.
1349	*/	1349	*/
1350	long do_fork(unsigned long clone_flags,	1350	long do_fork(unsigned long clone_flags,
1351	unsigned long stack_start,	1351	unsigned long stack_start,
1352	struct pt_regs *regs,	1352	struct pt_regs *regs,
1353	unsigned long stack_size,	1353	unsigned long stack_size,
1354	int __user *parent_tidptr,	1354	int __user *parent_tidptr,
1355	int __user *child_tidptr)	1355	int __user *child_tidptr)
1356	{	1356	{
1357	struct task_struct *p;	1357	struct task_struct *p;
1358	int trace = 0;	1358	int trace = 0;
1359	struct pid *pid = alloc_pid();	1359	struct pid *pid = alloc_pid();
1360	long nr;	1360	long nr;
1361		1361
1362	if (!pid)	1362	if (!pid)
1363	return -EAGAIN;	1363	return -EAGAIN;
1364	nr = pid->nr;	1364	nr = pid->nr;
1365	if (unlikely(current->ptrace)) {	1365	if (unlikely(current->ptrace)) {
1366	trace = fork_traceflag (clone_flags);	1366	trace = fork_traceflag (clone_flags);
1367	if (trace)	1367	if (trace)
1368	clone_flags \|= CLONE_PTRACE;	1368	clone_flags \|= CLONE_PTRACE;
1369	}	1369	}
1370		1370
1371	p = copy_process(clone_flags, stack_start, regs, stack_size, parent_tidptr, child_tidptr, nr);	1371	p = copy_process(clone_flags, stack_start, regs, stack_size, parent_tidptr, child_tidptr, nr);
1372	/*	1372	/*
1373	* Do this prior waking up the new thread - the thread pointer	1373	* Do this prior waking up the new thread - the thread pointer
1374	* might get invalid after that point, if the thread exits quickly.	1374	* might get invalid after that point, if the thread exits quickly.
1375	*/	1375	*/
1376	if (!IS_ERR(p)) {	1376	if (!IS_ERR(p)) {
1377	struct completion vfork;	1377	struct completion vfork;
1378		1378
1379	if (clone_flags & CLONE_VFORK) {	1379	if (clone_flags & CLONE_VFORK) {
1380	p->vfork_done = &vfork;	1380	p->vfork_done = &vfork;
1381	init_completion(&vfork);	1381	init_completion(&vfork);
1382	}	1382	}
1383		1383
1384	if ((p->ptrace & PT_PTRACED) \|\| (clone_flags & CLONE_STOPPED)) {	1384	if ((p->ptrace & PT_PTRACED) \|\| (clone_flags & CLONE_STOPPED)) {
1385	/*	1385	/*
1386	* We'll start up with an immediate SIGSTOP.	1386	* We'll start up with an immediate SIGSTOP.
1387	*/	1387	*/
1388	sigaddset(&p->pending.signal, SIGSTOP);	1388	sigaddset(&p->pending.signal, SIGSTOP);
1389	set_tsk_thread_flag(p, TIF_SIGPENDING);	1389	set_tsk_thread_flag(p, TIF_SIGPENDING);
1390	}	1390	}
1391		1391
1392	if (!(clone_flags & CLONE_STOPPED))	1392	if (!(clone_flags & CLONE_STOPPED))
1393	wake_up_new_task(p, clone_flags);	1393	wake_up_new_task(p, clone_flags);
1394	else	1394	else
1395	p->state = TASK_STOPPED;	1395	p->state = TASK_STOPPED;
1396		1396
1397	if (unlikely (trace)) {	1397	if (unlikely (trace)) {
1398	current->ptrace_message = nr;	1398	current->ptrace_message = nr;
1399	ptrace_notify ((trace << 8) \| SIGTRAP);	1399	ptrace_notify ((trace << 8) \| SIGTRAP);
1400	}	1400	}
1401		1401
1402	if (clone_flags & CLONE_VFORK) {	1402	if (clone_flags & CLONE_VFORK) {
1403	wait_for_completion(&vfork);	1403	wait_for_completion(&vfork);
1404	if (unlikely (current->ptrace & PT_TRACE_VFORK_DONE)) {	1404	if (unlikely (current->ptrace & PT_TRACE_VFORK_DONE)) {
1405	current->ptrace_message = nr;	1405	current->ptrace_message = nr;
1406	ptrace_notify ((PTRACE_EVENT_VFORK_DONE << 8) \| SIGTRAP);	1406	ptrace_notify ((PTRACE_EVENT_VFORK_DONE << 8) \| SIGTRAP);
1407	}	1407	}
1408	}	1408	}
1409	} else {	1409	} else {
1410	free_pid(pid);	1410	free_pid(pid);
1411	nr = PTR_ERR(p);	1411	nr = PTR_ERR(p);
1412	}	1412	}
1413	return nr;	1413	return nr;
1414	}	1414	}
1415		1415
1416	#ifndef ARCH_MIN_MMSTRUCT_ALIGN	1416	#ifndef ARCH_MIN_MMSTRUCT_ALIGN
1417	#define ARCH_MIN_MMSTRUCT_ALIGN 0	1417	#define ARCH_MIN_MMSTRUCT_ALIGN 0
1418	#endif	1418	#endif
1419		1419
1420	static void sighand_ctor(void data, struct kmem_cache cachep, unsigned long flags)	1420	static void sighand_ctor(void data, struct kmem_cache cachep, unsigned long flags)
1421	{	1421	{
1422	struct sighand_struct *sighand = data;	1422	struct sighand_struct *sighand = data;
1423		1423
1424	if ((flags & (SLAB_CTOR_VERIFY \| SLAB_CTOR_CONSTRUCTOR)) ==	1424	if ((flags & (SLAB_CTOR_VERIFY \| SLAB_CTOR_CONSTRUCTOR)) ==
1425	SLAB_CTOR_CONSTRUCTOR)	1425	SLAB_CTOR_CONSTRUCTOR)
1426	spin_lock_init(&sighand->siglock);	1426	spin_lock_init(&sighand->siglock);
1427	}	1427	}
1428		1428
1429	void __init proc_caches_init(void)	1429	void __init proc_caches_init(void)
1430	{	1430	{
1431	sighand_cachep = kmem_cache_create("sighand_cache",	1431	sighand_cachep = kmem_cache_create("sighand_cache",
1432	sizeof(struct sighand_struct), 0,	1432	sizeof(struct sighand_struct), 0,
1433	SLAB_HWCACHE_ALIGN\|SLAB_PANIC\|SLAB_DESTROY_BY_RCU,	1433	SLAB_HWCACHE_ALIGN\|SLAB_PANIC\|SLAB_DESTROY_BY_RCU,
1434	sighand_ctor, NULL);	1434	sighand_ctor, NULL);
1435	signal_cachep = kmem_cache_create("signal_cache",	1435	signal_cachep = kmem_cache_create("signal_cache",
1436	sizeof(struct signal_struct), 0,	1436	sizeof(struct signal_struct), 0,
1437	SLAB_HWCACHE_ALIGN\|SLAB_PANIC, NULL, NULL);	1437	SLAB_HWCACHE_ALIGN\|SLAB_PANIC, NULL, NULL);
1438	files_cachep = kmem_cache_create("files_cache",	1438	files_cachep = kmem_cache_create("files_cache",
1439	sizeof(struct files_struct), 0,	1439	sizeof(struct files_struct), 0,
1440	SLAB_HWCACHE_ALIGN\|SLAB_PANIC, NULL, NULL);	1440	SLAB_HWCACHE_ALIGN\|SLAB_PANIC, NULL, NULL);
1441	fs_cachep = kmem_cache_create("fs_cache",	1441	fs_cachep = kmem_cache_create("fs_cache",
1442	sizeof(struct fs_struct), 0,	1442	sizeof(struct fs_struct), 0,
1443	SLAB_HWCACHE_ALIGN\|SLAB_PANIC, NULL, NULL);	1443	SLAB_HWCACHE_ALIGN\|SLAB_PANIC, NULL, NULL);
1444	vm_area_cachep = kmem_cache_create("vm_area_struct",	1444	vm_area_cachep = kmem_cache_create("vm_area_struct",
1445	sizeof(struct vm_area_struct), 0,	1445	sizeof(struct vm_area_struct), 0,
1446	SLAB_PANIC, NULL, NULL);	1446	SLAB_PANIC, NULL, NULL);
1447	mm_cachep = kmem_cache_create("mm_struct",	1447	mm_cachep = kmem_cache_create("mm_struct",
1448	sizeof(struct mm_struct), ARCH_MIN_MMSTRUCT_ALIGN,	1448	sizeof(struct mm_struct), ARCH_MIN_MMSTRUCT_ALIGN,
1449	SLAB_HWCACHE_ALIGN\|SLAB_PANIC, NULL, NULL);	1449	SLAB_HWCACHE_ALIGN\|SLAB_PANIC, NULL, NULL);
1450	}	1450	}
1451		1451
1452		1452
1453	/*	1453	/*
1454	* Check constraints on flags passed to the unshare system call and	1454	* Check constraints on flags passed to the unshare system call and
1455	* force unsharing of additional process context as appropriate.	1455	* force unsharing of additional process context as appropriate.
1456	*/	1456	*/
1457	static inline void check_unshare_flags(unsigned long *flags_ptr)	1457	static inline void check_unshare_flags(unsigned long *flags_ptr)
1458	{	1458	{
1459	/*	1459	/*
1460	* If unsharing a thread from a thread group, must also	1460	* If unsharing a thread from a thread group, must also
1461	* unshare vm.	1461	* unshare vm.
1462	*/	1462	*/
1463	if (*flags_ptr & CLONE_THREAD)	1463	if (*flags_ptr & CLONE_THREAD)
1464	*flags_ptr \|= CLONE_VM;	1464	*flags_ptr \|= CLONE_VM;
1465		1465
1466	/*	1466	/*
1467	* If unsharing vm, must also unshare signal handlers.	1467	* If unsharing vm, must also unshare signal handlers.
1468	*/	1468	*/
1469	if (*flags_ptr & CLONE_VM)	1469	if (*flags_ptr & CLONE_VM)
1470	*flags_ptr \|= CLONE_SIGHAND;	1470	*flags_ptr \|= CLONE_SIGHAND;
1471		1471
1472	/*	1472	/*
1473	* If unsharing signal handlers and the task was created	1473	* If unsharing signal handlers and the task was created
1474	* using CLONE_THREAD, then must unshare the thread	1474	* using CLONE_THREAD, then must unshare the thread
1475	*/	1475	*/
1476	if ((*flags_ptr & CLONE_SIGHAND) &&	1476	if ((*flags_ptr & CLONE_SIGHAND) &&
1477	(atomic_read(&current->signal->count) > 1))	1477	(atomic_read(&current->signal->count) > 1))
1478	*flags_ptr \|= CLONE_THREAD;	1478	*flags_ptr \|= CLONE_THREAD;
1479		1479
1480	/*	1480	/*
1481	* If unsharing namespace, must also unshare filesystem information.	1481	* If unsharing namespace, must also unshare filesystem information.
1482	*/	1482	*/
1483	if (*flags_ptr & CLONE_NEWNS)	1483	if (*flags_ptr & CLONE_NEWNS)
1484	*flags_ptr \|= CLONE_FS;	1484	*flags_ptr \|= CLONE_FS;
1485	}	1485	}
1486		1486
1487	/*	1487	/*
1488	* Unsharing of tasks created with CLONE_THREAD is not supported yet	1488	* Unsharing of tasks created with CLONE_THREAD is not supported yet
1489	*/	1489	*/
1490	static int unshare_thread(unsigned long unshare_flags)	1490	static int unshare_thread(unsigned long unshare_flags)
1491	{	1491	{
1492	if (unshare_flags & CLONE_THREAD)	1492	if (unshare_flags & CLONE_THREAD)
1493	return -EINVAL;	1493	return -EINVAL;
1494		1494
1495	return 0;	1495	return 0;
1496	}	1496	}
1497		1497
1498	/*	1498	/*
1499	* Unshare the filesystem structure if it is being shared	1499	* Unshare the filesystem structure if it is being shared
1500	*/	1500	*/
1501	static int unshare_fs(unsigned long unshare_flags, struct fs_struct **new_fsp)	1501	static int unshare_fs(unsigned long unshare_flags, struct fs_struct **new_fsp)
1502	{	1502	{
1503	struct fs_struct *fs = current->fs;	1503	struct fs_struct *fs = current->fs;
1504		1504
1505	if ((unshare_flags & CLONE_FS) &&	1505	if ((unshare_flags & CLONE_FS) &&
1506	(fs && atomic_read(&fs->count) > 1)) {	1506	(fs && atomic_read(&fs->count) > 1)) {
1507	*new_fsp = __copy_fs_struct(current->fs);	1507	*new_fsp = __copy_fs_struct(current->fs);
1508	if (!*new_fsp)	1508	if (!*new_fsp)
1509	return -ENOMEM;	1509	return -ENOMEM;
1510	}	1510	}
1511		1511
1512	return 0;	1512	return 0;
1513	}	1513	}
1514		1514
1515	/*	1515	/*
1516	* Unshare the mnt_namespace structure if it is being shared	1516	* Unshare the mnt_namespace structure if it is being shared
1517	*/	1517	*/
1518	static int unshare_mnt_namespace(unsigned long unshare_flags,	1518	static int unshare_mnt_namespace(unsigned long unshare_flags,
1519	struct mnt_namespace *new_nsp, struct fs_struct new_fs)	1519	struct mnt_namespace *new_nsp, struct fs_struct new_fs)
1520	{	1520	{
1521	struct mnt_namespace *ns = current->nsproxy->mnt_ns;	1521	struct mnt_namespace *ns = current->nsproxy->mnt_ns;
1522		1522
1523	if ((unshare_flags & CLONE_NEWNS) && ns) {	1523	if ((unshare_flags & CLONE_NEWNS) && ns) {
1524	if (!capable(CAP_SYS_ADMIN))	1524	if (!capable(CAP_SYS_ADMIN))
1525	return -EPERM;	1525	return -EPERM;
1526		1526
1527	*new_nsp = dup_mnt_ns(current, new_fs ? new_fs : current->fs);	1527	*new_nsp = dup_mnt_ns(current, new_fs ? new_fs : current->fs);
1528	if (!*new_nsp)	1528	if (!*new_nsp)
1529	return -ENOMEM;	1529	return -ENOMEM;
1530	}	1530	}
1531		1531
1532	return 0;	1532	return 0;
1533	}	1533	}
1534		1534
1535	/*	1535	/*
1536	* Unsharing of sighand is not supported yet	1536	* Unsharing of sighand is not supported yet
1537	*/	1537	*/
1538	static int unshare_sighand(unsigned long unshare_flags, struct sighand_struct **new_sighp)	1538	static int unshare_sighand(unsigned long unshare_flags, struct sighand_struct **new_sighp)
1539	{	1539	{
1540	struct sighand_struct *sigh = current->sighand;	1540	struct sighand_struct *sigh = current->sighand;
1541		1541
1542	if ((unshare_flags & CLONE_SIGHAND) && atomic_read(&sigh->count) > 1)	1542	if ((unshare_flags & CLONE_SIGHAND) && atomic_read(&sigh->count) > 1)
1543	return -EINVAL;	1543	return -EINVAL;
1544	else	1544	else
1545	return 0;	1545	return 0;
1546	}	1546	}
1547		1547
1548	/*	1548	/*
1549	* Unshare vm if it is being shared	1549	* Unshare vm if it is being shared
1550	*/	1550	*/
1551	static int unshare_vm(unsigned long unshare_flags, struct mm_struct **new_mmp)	1551	static int unshare_vm(unsigned long unshare_flags, struct mm_struct **new_mmp)
1552	{	1552	{
1553	struct mm_struct *mm = current->mm;	1553	struct mm_struct *mm = current->mm;
1554		1554
1555	if ((unshare_flags & CLONE_VM) &&	1555	if ((unshare_flags & CLONE_VM) &&
1556	(mm && atomic_read(&mm->mm_users) > 1)) {	1556	(mm && atomic_read(&mm->mm_users) > 1)) {
1557	return -EINVAL;	1557	return -EINVAL;
1558	}	1558	}
1559		1559
1560	return 0;	1560	return 0;
1561	}	1561	}
1562		1562
1563	/*	1563	/*
1564	* Unshare file descriptor table if it is being shared	1564	* Unshare file descriptor table if it is being shared
1565	*/	1565	*/
1566	static int unshare_fd(unsigned long unshare_flags, struct files_struct **new_fdp)	1566	static int unshare_fd(unsigned long unshare_flags, struct files_struct **new_fdp)
1567	{	1567	{
1568	struct files_struct *fd = current->files;	1568	struct files_struct *fd = current->files;
1569	int error = 0;	1569	int error = 0;
1570		1570
1571	if ((unshare_flags & CLONE_FILES) &&	1571	if ((unshare_flags & CLONE_FILES) &&
1572	(fd && atomic_read(&fd->count) > 1)) {	1572	(fd && atomic_read(&fd->count) > 1)) {
1573	*new_fdp = dup_fd(fd, &error);	1573	*new_fdp = dup_fd(fd, &error);
1574	if (!*new_fdp)	1574	if (!*new_fdp)
1575	return error;	1575	return error;
1576	}	1576	}
1577		1577
1578	return 0;	1578	return 0;
1579	}	1579	}
1580		1580
1581	/*	1581	/*
1582	* Unsharing of semundo for tasks created with CLONE_SYSVSEM is not	1582	* Unsharing of semundo for tasks created with CLONE_SYSVSEM is not
1583	* supported yet	1583	* supported yet
1584	*/	1584	*/
1585	static int unshare_semundo(unsigned long unshare_flags, struct sem_undo_list **new_ulistp)	1585	static int unshare_semundo(unsigned long unshare_flags, struct sem_undo_list **new_ulistp)
1586	{	1586	{
1587	if (unshare_flags & CLONE_SYSVSEM)	1587	if (unshare_flags & CLONE_SYSVSEM)
1588	return -EINVAL;	1588	return -EINVAL;
1589		1589
1590	return 0;	1590	return 0;
1591	}	1591	}
1592		1592
1593	#ifndef CONFIG_IPC_NS	1593	#ifndef CONFIG_IPC_NS
1594	static inline int unshare_ipcs(unsigned long flags, struct ipc_namespace **ns)	1594	static inline int unshare_ipcs(unsigned long flags, struct ipc_namespace **ns)
1595	{	1595	{
1596	if (flags & CLONE_NEWIPC)	1596	if (flags & CLONE_NEWIPC)
1597	return -EINVAL;	1597	return -EINVAL;
1598		1598
1599	return 0;	1599	return 0;
1600	}	1600	}
1601	#endif	1601	#endif
1602		1602
1603	/*	1603	/*
1604	* unshare allows a process to 'unshare' part of the process	1604	* unshare allows a process to 'unshare' part of the process
1605	* context which was originally shared using clone. copy_*	1605	* context which was originally shared using clone. copy_*
1606	* functions used by do_fork() cannot be used here directly	1606	* functions used by do_fork() cannot be used here directly
1607	* because they modify an inactive task_struct that is being	1607	* because they modify an inactive task_struct that is being
1608	* constructed. Here we are modifying the current, active,	1608	* constructed. Here we are modifying the current, active,
1609	* task_struct.	1609	* task_struct.
1610	*/	1610	*/
1611	asmlinkage long sys_unshare(unsigned long unshare_flags)	1611	asmlinkage long sys_unshare(unsigned long unshare_flags)
1612	{	1612	{
1613	int err = 0;	1613	int err = 0;
1614	struct fs_struct fs, new_fs = NULL;	1614	struct fs_struct fs, new_fs = NULL;
1615	struct mnt_namespace ns, new_ns = NULL;	1615	struct mnt_namespace ns, new_ns = NULL;
1616	struct sighand_struct *new_sigh = NULL;	1616	struct sighand_struct *new_sigh = NULL;
1617	struct mm_struct mm, new_mm = NULL, *active_mm = NULL;	1617	struct mm_struct mm, new_mm = NULL, *active_mm = NULL;
1618	struct files_struct fd, new_fd = NULL;	1618	struct files_struct fd, new_fd = NULL;
1619	struct sem_undo_list *new_ulist = NULL;	1619	struct sem_undo_list *new_ulist = NULL;
1620	struct nsproxy new_nsproxy = NULL, old_nsproxy = NULL;	1620	struct nsproxy new_nsproxy = NULL, old_nsproxy = NULL;
1621	struct uts_namespace uts, new_uts = NULL;	1621	struct uts_namespace uts, new_uts = NULL;
1622	struct ipc_namespace ipc, new_ipc = NULL;	1622	struct ipc_namespace ipc, new_ipc = NULL;
1623		1623
1624	check_unshare_flags(&unshare_flags);	1624	check_unshare_flags(&unshare_flags);
1625		1625
1626	/* Return -EINVAL for all unsupported flags */	1626	/* Return -EINVAL for all unsupported flags */
1627	err = -EINVAL;	1627	err = -EINVAL;
1628	if (unshare_flags & ~(CLONE_THREAD\|CLONE_FS\|CLONE_NEWNS\|CLONE_SIGHAND\|	1628	if (unshare_flags & ~(CLONE_THREAD\|CLONE_FS\|CLONE_NEWNS\|CLONE_SIGHAND\|
1629	CLONE_VM\|CLONE_FILES\|CLONE_SYSVSEM\|	1629	CLONE_VM\|CLONE_FILES\|CLONE_SYSVSEM\|
1630	CLONE_NEWUTS\|CLONE_NEWIPC))	1630	CLONE_NEWUTS\|CLONE_NEWIPC))
1631	goto bad_unshare_out;	1631	goto bad_unshare_out;
1632		1632
1633	if ((err = unshare_thread(unshare_flags)))	1633	if ((err = unshare_thread(unshare_flags)))
1634	goto bad_unshare_out;	1634	goto bad_unshare_out;
1635	if ((err = unshare_fs(unshare_flags, &new_fs)))	1635	if ((err = unshare_fs(unshare_flags, &new_fs)))
1636	goto bad_unshare_cleanup_thread;	1636	goto bad_unshare_cleanup_thread;
1637	if ((err = unshare_mnt_namespace(unshare_flags, &new_ns, new_fs)))	1637	if ((err = unshare_mnt_namespace(unshare_flags, &new_ns, new_fs)))
1638	goto bad_unshare_cleanup_fs;	1638	goto bad_unshare_cleanup_fs;
1639	if ((err = unshare_sighand(unshare_flags, &new_sigh)))	1639	if ((err = unshare_sighand(unshare_flags, &new_sigh)))
1640	goto bad_unshare_cleanup_ns;	1640	goto bad_unshare_cleanup_ns;
1641	if ((err = unshare_vm(unshare_flags, &new_mm)))	1641	if ((err = unshare_vm(unshare_flags, &new_mm)))
1642	goto bad_unshare_cleanup_sigh;	1642	goto bad_unshare_cleanup_sigh;
1643	if ((err = unshare_fd(unshare_flags, &new_fd)))	1643	if ((err = unshare_fd(unshare_flags, &new_fd)))
1644	goto bad_unshare_cleanup_vm;	1644	goto bad_unshare_cleanup_vm;
1645	if ((err = unshare_semundo(unshare_flags, &new_ulist)))	1645	if ((err = unshare_semundo(unshare_flags, &new_ulist)))
1646	goto bad_unshare_cleanup_fd;	1646	goto bad_unshare_cleanup_fd;
1647	if ((err = unshare_utsname(unshare_flags, &new_uts)))	1647	if ((err = unshare_utsname(unshare_flags, &new_uts)))
1648	goto bad_unshare_cleanup_semundo;	1648	goto bad_unshare_cleanup_semundo;
1649	if ((err = unshare_ipcs(unshare_flags, &new_ipc)))	1649	if ((err = unshare_ipcs(unshare_flags, &new_ipc)))
1650	goto bad_unshare_cleanup_uts;	1650	goto bad_unshare_cleanup_uts;
1651		1651
1652	if (new_ns \|\| new_uts \|\| new_ipc) {	1652	if (new_ns \|\| new_uts \|\| new_ipc) {
1653	old_nsproxy = current->nsproxy;	1653	old_nsproxy = current->nsproxy;
1654	new_nsproxy = dup_namespaces(old_nsproxy);	1654	new_nsproxy = dup_namespaces(old_nsproxy);
1655	if (!new_nsproxy) {	1655	if (!new_nsproxy) {
1656	err = -ENOMEM;	1656	err = -ENOMEM;
1657	goto bad_unshare_cleanup_ipc;	1657	goto bad_unshare_cleanup_ipc;
1658	}	1658	}
1659	}	1659	}
1660		1660
1661	if (new_fs \|\| new_ns \|\| new_mm \|\| new_fd \|\| new_ulist \|\|	1661	if (new_fs \|\| new_ns \|\| new_mm \|\| new_fd \|\| new_ulist \|\|
1662	new_uts \|\| new_ipc) {	1662	new_uts \|\| new_ipc) {
1663		1663
1664	task_lock(current);	1664	task_lock(current);
1665		1665
1666	if (new_nsproxy) {	1666	if (new_nsproxy) {
1667	current->nsproxy = new_nsproxy;	1667	current->nsproxy = new_nsproxy;
1668	new_nsproxy = old_nsproxy;	1668	new_nsproxy = old_nsproxy;
1669	}	1669	}
1670		1670
1671	if (new_fs) {	1671	if (new_fs) {
1672	fs = current->fs;	1672	fs = current->fs;
1673	current->fs = new_fs;	1673	current->fs = new_fs;
1674	new_fs = fs;	1674	new_fs = fs;
1675	}	1675	}
1676		1676
1677	if (new_ns) {	1677	if (new_ns) {
1678	ns = current->nsproxy->mnt_ns;	1678	ns = current->nsproxy->mnt_ns;
1679	current->nsproxy->mnt_ns = new_ns;	1679	current->nsproxy->mnt_ns = new_ns;
1680	new_ns = ns;	1680	new_ns = ns;
1681	}	1681	}
1682		1682
1683	if (new_mm) {	1683	if (new_mm) {
1684	mm = current->mm;	1684	mm = current->mm;
1685	active_mm = current->active_mm;	1685	active_mm = current->active_mm;
1686	current->mm = new_mm;	1686	current->mm = new_mm;
1687	current->active_mm = new_mm;	1687	current->active_mm = new_mm;
1688	activate_mm(active_mm, new_mm);	1688	activate_mm(active_mm, new_mm);
1689	new_mm = mm;	1689	new_mm = mm;
1690	}	1690	}
1691		1691
1692	if (new_fd) {	1692	if (new_fd) {
1693	fd = current->files;	1693	fd = current->files;
1694	current->files = new_fd;	1694	current->files = new_fd;
1695	new_fd = fd;	1695	new_fd = fd;
1696	}	1696	}
1697		1697
1698	if (new_uts) {	1698	if (new_uts) {
1699	uts = current->nsproxy->uts_ns;	1699	uts = current->nsproxy->uts_ns;
1700	current->nsproxy->uts_ns = new_uts;	1700	current->nsproxy->uts_ns = new_uts;
1701	new_uts = uts;	1701	new_uts = uts;
1702	}	1702	}
1703		1703
1704	if (new_ipc) {	1704	if (new_ipc) {
1705	ipc = current->nsproxy->ipc_ns;	1705	ipc = current->nsproxy->ipc_ns;
1706	current->nsproxy->ipc_ns = new_ipc;	1706	current->nsproxy->ipc_ns = new_ipc;
1707	new_ipc = ipc;	1707	new_ipc = ipc;
1708	}	1708	}
1709		1709
1710	task_unlock(current);	1710	task_unlock(current);
1711	}	1711	}
1712		1712
1713	if (new_nsproxy)	1713	if (new_nsproxy)
1714	put_nsproxy(new_nsproxy);	1714	put_nsproxy(new_nsproxy);
1715		1715
1716	bad_unshare_cleanup_ipc:	1716	bad_unshare_cleanup_ipc:
1717	if (new_ipc)	1717	if (new_ipc)
1718	put_ipc_ns(new_ipc);	1718	put_ipc_ns(new_ipc);
1719		1719
1720	bad_unshare_cleanup_uts:	1720	bad_unshare_cleanup_uts:
1721	if (new_uts)	1721	if (new_uts)
1722	put_uts_ns(new_uts);	1722	put_uts_ns(new_uts);
1723		1723
1724	bad_unshare_cleanup_semundo:	1724	bad_unshare_cleanup_semundo:
1725	bad_unshare_cleanup_fd:	1725	bad_unshare_cleanup_fd:
1726	if (new_fd)	1726	if (new_fd)
1727	put_files_struct(new_fd);	1727	put_files_struct(new_fd);
1728		1728
1729	bad_unshare_cleanup_vm:	1729	bad_unshare_cleanup_vm:
1730	if (new_mm)	1730	if (new_mm)
1731	mmput(new_mm);	1731	mmput(new_mm);
1732		1732
1733	bad_unshare_cleanup_sigh:	1733	bad_unshare_cleanup_sigh:
1734	if (new_sigh)	1734	if (new_sigh)
1735	if (atomic_dec_and_test(&new_sigh->count))	1735	if (atomic_dec_and_test(&new_sigh->count))
1736	kmem_cache_free(sighand_cachep, new_sigh);	1736	kmem_cache_free(sighand_cachep, new_sigh);
1737		1737
1738	bad_unshare_cleanup_ns:	1738	bad_unshare_cleanup_ns:
1739	if (new_ns)	1739	if (new_ns)
1740	put_mnt_ns(new_ns);	1740	put_mnt_ns(new_ns);
1741		1741
1742	bad_unshare_cleanup_fs:	1742	bad_unshare_cleanup_fs:
1743	if (new_fs)	1743	if (new_fs)
1744	put_fs_struct(new_fs);	1744	put_fs_struct(new_fs);
1745		1745
1746	bad_unshare_cleanup_thread:	1746	bad_unshare_cleanup_thread:
1747	bad_unshare_out:	1747	bad_unshare_out:
1748	return err;	1748	return err;
1749	}	1749	}
1750		1750