Eric Lee / linux-smarc-t335x-v3.2

Commit a1f4d39500ad8ed61825eff061debff42386ab5b

Authored by Avi Kivity 2010-06-21 16:44:20 +0800

Exists in master and in 4 other branches

KVM: Remove memory alias support

As advertised in feature-removal-schedule.txt.  Equivalent support is provided
by overlapping memory regions.

Signed-off-by: Avi Kivity <avi@redhat.com>

Showing 13 changed files with 11 additions and 225 deletions Inline Diff

Documentation/feature-removal-schedule.txt
Documentation/kvm/api.txt
arch/ia64/kvm/kvm-ia64.c
arch/powerpc/kvm/powerpc.c
arch/s390/kvm/kvm-s390.c
arch/x86/include/asm/kvm_host.h
arch/x86/kvm/mmu.c
arch/x86/kvm/paging_tmpl.h
arch/x86/kvm/x86.c
arch/x86/kvm/x86.h
include/linux/kvm.h
include/linux/kvm_host.h
virt/kvm/kvm_main.c

Documentation/feature-removal-schedule.txt

Diff comments View file @ a1f4d39

 The following is a list of files and features that are going to be
 removed in the kernel source tree.  Every entry should contain what
 exactly is going away, why it is happening, and who is going to be doing
 the work.  When the feature is removed from the kernel, it should also
 be removed from this file.
 ---------------------------
 What:	PRISM54
 When:	2.6.34
 Why:	prism54 FullMAC PCI / Cardbus devices used to be supported only by the
 	prism54 wireless driver. After Intersil stopped selling these
 	devices in preference for the newer more flexible SoftMAC devices
 	a SoftMAC device driver was required and prism54 did not support
 	them. The p54pci driver now exists and has been present in the kernel for
 	a while. This driver supports both SoftMAC devices and FullMAC devices.
 	The main difference between these devices was the amount of memory which
 	could be used for the firmware. The SoftMAC devices support a smaller
 	amount of memory. Because of this the SoftMAC firmware fits into FullMAC
 	devices's memory. p54pci supports not only PCI / Cardbus but also USB
 	and SPI. Since p54pci supports all devices prism54 supports
 	you will have a conflict. I'm not quite sure how distributions are
 	handling this conflict right now. prism54 was kept around due to
 	claims users may experience issues when using the SoftMAC driver.
 	Time has passed users have not reported issues. If you use prism54
 	and for whatever reason you cannot use p54pci please let us know!
 	E-mail us at: linux-wireless@vger.kernel.org
 	For more information see the p54 wiki page:
 	http://wireless.kernel.org/en/users/Drivers/p54
 Who:	Luis R. Rodriguez <lrodriguez@atheros.com>
 ---------------------------
 What:	IRQF_SAMPLE_RANDOM
 Check:	IRQF_SAMPLE_RANDOM
 When:	July 2009
 Why:	Many of IRQF_SAMPLE_RANDOM users are technically bogus as entropy
 	sources in the kernel's current entropy model. To resolve this, every
 	input point to the kernel's entropy pool needs to better document the
 	type of entropy source it actually is. This will be replaced with
 	additional add_*_randomness functions in drivers/char/random.c
 Who:	Robin Getz <rgetz@blackfin.uclinux.org> & Matt Mackall <mpm@selenic.com>
 ---------------------------
 What:	Deprecated snapshot ioctls
 When:	2.6.36
 Why:	The ioctls in kernel/power/user.c were marked as deprecated long time
 	ago. Now they notify users about that so that they need to replace
 	their userspace. After some more time, remove them completely.
 Who:	Jiri Slaby <jirislaby@gmail.com>
 ---------------------------
 What:	The ieee80211_regdom module parameter
 When:	March 2010 / desktop catchup
 Why:	This was inherited by the CONFIG_WIRELESS_OLD_REGULATORY code,
 	and currently serves as an option for users to define an
 	ISO / IEC 3166 alpha2 code for the country they are currently
 	present in. Although there are userspace API replacements for this
 	through nl80211 distributions haven't yet caught up with implementing
 	decent alternatives through standard GUIs. Although available as an
 	option through iw or wpa_supplicant its just a matter of time before
 	distributions pick up good GUI options for this. The ideal solution
 	would actually consist of intelligent designs which would do this for
 	the user automatically even when travelling through different countries.
 	Until then we leave this module parameter as a compromise.
 	When userspace improves with reasonable widely-available alternatives for
 	this we will no longer need this module parameter. This entry hopes that
 	by the super-futuristically looking date of "March 2010" we will have
 	such replacements widely available.
 Who:	Luis R. Rodriguez <lrodriguez@atheros.com>
 ---------------------------
 What:	dev->power.power_state
 When:	July 2007
 Why:	Broken design for runtime control over driver power states, confusing
 	driver-internal runtime power management with:  mechanisms to support
 	system-wide sleep state transitions; event codes that distinguish
 	different phases of swsusp "sleep" transitions; and userspace policy
 	inputs.  This framework was never widely used, and most attempts to
 	use it were broken.  Drivers should instead be exposing domain-specific
 	interfaces either to kernel or to userspace.
 Who:	Pavel Machek <pavel@suse.cz>
 ---------------------------
 What:	Video4Linux API 1 ioctls and from Video devices.
 When:	July 2009
 Files:	include/linux/videodev.h
 Check:	include/linux/videodev.h
 Why:	V4L1 AP1 was replaced by V4L2 API during migration from 2.4 to 2.6
 	series. The old API have lots of drawbacks and don't provide enough
 	means to work with all video and audio standards. The newer API is
 	already available on the main drivers and should be used instead.
 	Newer drivers should use v4l_compat_translate_ioctl function to handle
 	old calls, replacing to newer ones.
 	Decoder iocts are using internally to allow video drivers to
 	communicate with video decoders. This should also be improved to allow
 	V4L2 calls being translated into compatible internal ioctls.
 	Compatibility ioctls will be provided, for a while, via
 	v4l1-compat module.
 Who:	Mauro Carvalho Chehab <mchehab@infradead.org>
 ---------------------------
 What:	PCMCIA control ioctl (needed for pcmcia-cs [cardmgr, cardctl])
 When:	2.6.35/2.6.36
 Files:	drivers/pcmcia/: pcmcia_ioctl.c
 Why:	With the 16-bit PCMCIA subsystem now behaving (almost) like a
 	normal hotpluggable bus, and with it using the default kernel
 	infrastructure (hotplug, driver core, sysfs) keeping the PCMCIA
 	control ioctl needed by cardmgr and cardctl from pcmcia-cs is
 	unnecessary and potentially harmful (it does not provide for
 	proper locking), and makes further cleanups and integration of the
 	PCMCIA subsystem into the Linux kernel device driver model more
 	difficult. The features provided by cardmgr and cardctl are either
 	handled by the kernel itself now or are available in the new
 	pcmciautils package available at
 	http://kernel.org/pub/linux/utils/kernel/pcmcia/
 	For all architectures except ARM, the associated config symbol
 	has been removed from kernel 2.6.34; for ARM, it will be likely
 	be removed from kernel 2.6.35. The actual code will then likely
 	be removed from kernel 2.6.36.
 Who:	Dominik Brodowski <linux@dominikbrodowski.net>
 ---------------------------
 What:	sys_sysctl
 When:	September 2010
 Option: CONFIG_SYSCTL_SYSCALL
 Why:	The same information is available in a more convenient from
 	/proc/sys, and none of the sysctl variables appear to be
 	important performance wise.
 	Binary sysctls are a long standing source of subtle kernel
 	bugs and security issues.
 	When I looked several months ago all I could find after
 	searching several distributions were 5 user space programs and
 	glibc (which falls back to /proc/sys) using this syscall.
 	The man page for sysctl(2) documents it as unusable for user
 	space programs.
 	sysctl(2) is not generally ABI compatible to a 32bit user
 	space application on a 64bit and a 32bit kernel.
 	For the last several months the policy has been no new binary
 	sysctls and no one has put forward an argument to use them.
 	Binary sysctls issues seem to keep happening appearing so
 	properly deprecating them (with a warning to user space) and a
 	2 year grace warning period will mean eventually we can kill
 	them and end the pain.
 	In the mean time individual binary sysctls can be dealt with
 	in a piecewise fashion.
 Who:	Eric Biederman <ebiederm@xmission.com>
 ---------------------------
 What:	remove EXPORT_SYMBOL(kernel_thread)
 When:	August 2006
 Files:	arch/*/kernel/*_ksyms.c
 Check:	kernel_thread
 Why:	kernel_thread is a low-level implementation detail.  Drivers should
         use the <linux/kthread.h> API instead which shields them from
 	implementation details and provides a higherlevel interface that
 	prevents bugs and code duplication
 Who:	Christoph Hellwig <hch@lst.de>
 ---------------------------
 What:	Unused EXPORT_SYMBOL/EXPORT_SYMBOL_GPL exports
 	(temporary transition config option provided until then)
 	The transition config option will also be removed at the same time.
 When:	before 2.6.19
 Why:	Unused symbols are both increasing the size of the kernel binary
 	and are often a sign of "wrong API"
 Who:	Arjan van de Ven <arjan@linux.intel.com>
 ---------------------------
 What:	PHYSDEVPATH, PHYSDEVBUS, PHYSDEVDRIVER in the uevent environment
 When:	October 2008
 Why:	The stacking of class devices makes these values misleading and
 	inconsistent.
 	Class devices should not carry any of these properties, and bus
 	devices have SUBSYTEM and DRIVER as a replacement.
 Who:	Kay Sievers <kay.sievers@suse.de>
 ---------------------------
 What:	ACPI procfs interface
 When:	July 2008
 Why:	ACPI sysfs conversion should be finished by January 2008.
 	ACPI procfs interface will be removed in July 2008 so that
 	there is enough time for the user space to catch up.
 Who:	Zhang Rui <rui.zhang@intel.com>
 ---------------------------
 What:	/proc/acpi/button
 When:	August 2007
 Why:	/proc/acpi/button has been replaced by events to the input layer
 	since 2.6.20.
 Who:	Len Brown <len.brown@intel.com>
 ---------------------------
 What:	/proc/acpi/event
 When:	February 2008
 Why:	/proc/acpi/event has been replaced by events via the input layer
 	and netlink since 2.6.23.
 Who:	Len Brown <len.brown@intel.com>
 ---------------------------
 What:	i386/x86_64 bzImage symlinks
 When:	April 2010
 Why:	The i386/x86_64 merge provides a symlink to the old bzImage
 	location so not yet updated user space tools, e.g. package
 	scripts, do not break.
 Who:	Thomas Gleixner <tglx@linutronix.de>
 ---------------------------
 What:	GPIO autorequest on gpio_direction_{input,output}() in gpiolib
 When:	February 2010
 Why:	All callers should use explicit gpio_request()/gpio_free().
 	The autorequest mechanism in gpiolib was provided mostly as a
 	migration aid for legacy GPIO interfaces (for SOC based GPIOs).
 	Those users have now largely migrated.  Platforms implementing
 	the GPIO interfaces without using gpiolib will see no changes.
 Who:	David Brownell <dbrownell@users.sourceforge.net>
 ---------------------------
 What:	b43 support for firmware revision < 410
 When:	The schedule was July 2008, but it was decided that we are going to keep the
         code as long as there are no major maintanance headaches.
 	So it _could_ be removed _any_ time now, if it conflicts with something new.
 Why:	The support code for the old firmware hurts code readability/maintainability
 	and slightly hurts runtime performance. Bugfixes for the old firmware
 	are not provided by Broadcom anymore.
 Who:	Michael Buesch <mb@bu3sch.de>
 ---------------------------
 What:	/sys/o2cb symlink
 When:	January 2010
 Why:	/sys/fs/o2cb is the proper location for this information - /sys/o2cb
 	exists as a symlink for backwards compatibility for old versions of
 	ocfs2-tools. 2 years should be sufficient time to phase in new versions
 	which know to look in /sys/fs/o2cb.
 Who:	ocfs2-devel@oss.oracle.com
 ---------------------------
 What:	Ability for non root users to shm_get hugetlb pages based on mlock
 	resource limits
 When:	2.6.31
 Why:	Non root users need to be part of /proc/sys/vm/hugetlb_shm_group or
 	have CAP_IPC_LOCK to be able to allocate shm segments backed by
 	huge pages.  The mlock based rlimit check to allow shm hugetlb is
 	inconsistent with mmap based allocations.  Hence it is being
 	deprecated.
 Who:	Ravikiran Thirumalai <kiran@scalex86.org>
 ---------------------------
 What:	CONFIG_THERMAL_HWMON
 When:	January 2009
 Why:	This option was introduced just to allow older lm-sensors userspace
 	to keep working over the upgrade to 2.6.26. At the scheduled time of
 	removal fixed lm-sensors (2.x or 3.x) should be readily available.
 Who:	Rene Herman <rene.herman@gmail.com>
 ---------------------------
 What:	Code that is now under CONFIG_WIRELESS_EXT_SYSFS
 	(in net/core/net-sysfs.c)
 When:	After the only user (hal) has seen a release with the patches
 	for enough time, probably some time in 2010.
 Why:	Over 1K .text/.data size reduction, data is available in other
 	ways (ioctls)
 Who:	Johannes Berg <johannes@sipsolutions.net>
 ---------------------------
 What: CONFIG_NF_CT_ACCT
 When: 2.6.29
 Why:  Accounting can now be enabled/disabled without kernel recompilation.
       Currently used only to set a default value for a feature that is also
       controlled by a kernel/module/sysfs/sysctl parameter.
 Who:  Krzysztof Piotr Oledzki <ole@ans.pl>
 ---------------------------
 What:	sysfs ui for changing p4-clockmod parameters
 When:	September 2009
 Why:	See commits 129f8ae9b1b5be94517da76009ea956e89104ce8 and
 	e088e4c9cdb618675874becb91b2fd581ee707e6.
 	Removal is subject to fixing any remaining bugs in ACPI which may
 	cause the thermal throttling not to happen at the right time.
 Who:	Dave Jones <davej@redhat.com>, Matthew Garrett <mjg@redhat.com>
 -----------------------------
 What:	__do_IRQ all in one fits nothing interrupt handler
 When:	2.6.32
 Why:	__do_IRQ was kept for easy migration to the type flow handlers.
 	More than two years of migration time is enough.
 Who:	Thomas Gleixner <tglx@linutronix.de>
 -----------------------------
 What:	fakephp and associated sysfs files in /sys/bus/pci/slots/
 When:	2011
 Why:	In 2.6.27, the semantics of /sys/bus/pci/slots was redefined to
 	represent a machine's physical PCI slots. The change in semantics
 	had userspace implications, as the hotplug core no longer allowed
 	drivers to create multiple sysfs files per physical slot (required
 	for multi-function devices, e.g.). fakephp was seen as a developer's
 	tool only, and its interface changed. Too late, we learned that
 	there were some users of the fakephp interface.
 	In 2.6.30, the original fakephp interface was restored. At the same
 	time, the PCI core gained the ability that fakephp provided, namely
 	function-level hot-remove and hot-add.
 	Since the PCI core now provides the same functionality, exposed in:
 		/sys/bus/pci/rescan
 		/sys/bus/pci/devices/.../remove
 		/sys/bus/pci/devices/.../rescan
 	there is no functional reason to maintain fakephp as well.
 	We will keep the existing module so that 'modprobe fakephp' will
 	present the old /sys/bus/pci/slots/... interface for compatibility,
 	but users are urged to migrate their applications to the API above.
 	After a reasonable transition period, we will remove the legacy
 	fakephp interface.
 Who:	Alex Chiang <achiang@hp.com>
 ---------------------------
 What:	CONFIG_RFKILL_INPUT
 When:	2.6.33
 Why:	Should be implemented in userspace, policy daemon.
 Who:	Johannes Berg <johannes@sipsolutions.net>
 ---------------------------
 What:	CONFIG_INOTIFY
 When:	2.6.33
 Why:	last user (audit) will be converted to the newer more generic
 	and more easily maintained fsnotify subsystem
 Who:	Eric Paris <eparis@redhat.com>
 ----------------------------
 What:	lock_policy_rwsem_* and unlock_policy_rwsem_* will not be
 	exported interface anymore.
 When:	2.6.33
 Why:	cpu_policy_rwsem has a new cleaner definition making it local to
 	cpufreq core and contained inside cpufreq.c. Other dependent
 	drivers should not use it in order to safely avoid lockdep issues.
 Who:	Venkatesh Pallipadi <venkatesh.pallipadi@intel.com>
 ----------------------------
 What:	sound-slot/service-* module aliases and related clutters in
 	sound/sound_core.c
 When:	August 2010
 Why:	OSS sound_core grabs all legacy minors (0-255) of SOUND_MAJOR
 	(14) and requests modules using custom sound-slot/service-*
 	module aliases.  The only benefit of doing this is allowing
 	use of custom module aliases which might as well be considered
 	a bug at this point.  This preemptive claiming prevents
 	alternative OSS implementations.
 	Till the feature is removed, the kernel will be requesting
 	both sound-slot/service-* and the standard char-major-* module
 	aliases and allow turning off the pre-claiming selectively via
 	CONFIG_SOUND_OSS_CORE_PRECLAIM and soundcore.preclaim_oss
 	kernel parameter.
 	After the transition phase is complete, both the custom module
 	aliases and switches to disable it will go away.  This removal
 	will also allow making ALSA OSS emulation independent of
 	sound_core.  The dependency will be broken then too.
 Who:	Tejun Heo <tj@kernel.org>
 ----------------------------
 What:	Support for VMware's guest paravirtuliazation technique [VMI] will be
 	dropped.
 When:	2.6.37 or earlier.
 Why:	With the recent innovations in CPU hardware acceleration technologies
 	from Intel and AMD, VMware ran a few experiments to compare these
 	techniques to guest paravirtualization technique on VMware's platform.
 	These hardware assisted virtualization techniques have outperformed the
 	performance benefits provided by VMI in most of the workloads. VMware
 	expects that these hardware features will be ubiquitous in a couple of
 	years, as a result, VMware has started a phased retirement of this
 	feature from the hypervisor. We will be removing this feature from the
 	Kernel too. Right now we are targeting 2.6.37 but can retire earlier if
 	technical reasons (read opportunity to remove major chunk of pvops)
 	arise.
 	Please note that VMI has always been an optimization and non-VMI kernels
 	still work fine on VMware's platform.
 	Latest versions of VMware's product which support VMI are,
 	Workstation 7.0 and VSphere 4.0 on ESX side, future maintainence
 	releases for these products will continue supporting VMI.
 	For more details about VMI retirement take a look at this,
 	http://blogs.vmware.com/guestosguide/2009/09/vmi-retirement.html
 Who:	Alok N Kataria <akataria@vmware.com>
 ----------------------------
 What:	Support for lcd_switch and display_get in asus-laptop driver
 When:	March 2010
 Why:	These two features use non-standard interfaces. There are the
 	only features that really need multiple path to guess what's
 	the right method name on a specific laptop.
 	Removing them will allow to remove a lot of code an significantly
 	clean the drivers.
 	This will affect the backlight code which won't be able to know
 	if the backlight is on or off. The platform display file will also be
 	write only (like the one in eeepc-laptop).
 	This should'nt affect a lot of user because they usually know
 	when their display is on or off.
 Who:	Corentin Chary <corentin.chary@gmail.com>
 ----------------------------
 What:	usbvideo quickcam_messenger driver
 When:	2.6.35
 Files:	drivers/media/video/usbvideo/quickcam_messenger.[ch]
 Why:	obsolete v4l1 driver replaced by gspca_stv06xx
 Who:	Hans de Goede <hdegoede@redhat.com>
 ----------------------------
 What:	ov511 v4l1 driver
 When:	2.6.35
 Files:	drivers/media/video/ov511.[ch]
 Why:	obsolete v4l1 driver replaced by gspca_ov519
 Who:	Hans de Goede <hdegoede@redhat.com>
 ----------------------------
 What:	w9968cf v4l1 driver
 When:	2.6.35
 Files:	drivers/media/video/w9968cf*.[ch]
 Why:	obsolete v4l1 driver replaced by gspca_ov519
 Who:	Hans de Goede <hdegoede@redhat.com>
 ----------------------------
 What:	ovcamchip sensor framework
 When:	2.6.35
 Files:	drivers/media/video/ovcamchip/*
 Why:	Only used by obsoleted v4l1 drivers
 Who:	Hans de Goede <hdegoede@redhat.com>
 ----------------------------
 What:	stv680 v4l1 driver
 When:	2.6.35
 Files:	drivers/media/video/stv680.[ch]
 Why:	obsolete v4l1 driver replaced by gspca_stv0680
 Who:	Hans de Goede <hdegoede@redhat.com>
 ----------------------------
 What:	zc0301 v4l driver
 When:	2.6.35
 Files:	drivers/media/video/zc0301/*
 Why:	Duplicate functionality with the gspca_zc3xx driver, zc0301 only
 	supports 2 USB-ID's (because it only supports a limited set of
 	sensors) wich are also supported by the gspca_zc3xx driver
 	(which supports 53 USB-ID's in total)
 Who:	Hans de Goede <hdegoede@redhat.com>
 ----------------------------
 What:	sysfs-class-rfkill state file
 When:	Feb 2014
 Files:	net/rfkill/core.c
 Why: 	Documented as obsolete since Feb 2010. This file is limited to 3
 	states while the rfkill drivers can have 4 states.
 Who: 	anybody or Florian Mickler <florian@mickler.org>
 ----------------------------
 What: 	sysfs-class-rfkill claim file
 When:	Feb 2012
 Files:	net/rfkill/core.c
 Why:	It is not possible to claim an rfkill driver since 2007. This is
 	Documented as obsolete since Feb 2010.
 Who: 	anybody or Florian Mickler <florian@mickler.org>
 ----------------------------
 What:	capifs
 When:	February 2011
 Files:	drivers/isdn/capi/capifs.*
 Why:	udev fully replaces this special file system that only contains CAPI
 	NCCI TTY device nodes. User space (pppdcapiplugin) works without
 	noticing the difference.
 Who:	Jan Kiszka <jan.kiszka@web.de>
 ----------------------------
-What:	KVM memory aliases support
-When:	July 2010
-Why:	Memory aliasing support is used for speeding up guest vga access
-	through the vga windows.
-	Modern userspace no longer uses this feature, so it's just bitrotted
-	code and can be removed with no impact.
-Who:	Avi Kivity <avi@redhat.com>
-----------------------------
 What:	xtime, wall_to_monotonic
 When:	2.6.36+
 Files:	kernel/time/timekeeping.c include/linux/time.h
 Why:	Cleaning up timekeeping internal values. Please use
 	existing timekeeping accessor functions to access
 	the equivalent functionality.
 Who:	John Stultz <johnstul@us.ibm.com>
 ----------------------------
 What:	KVM kernel-allocated memory slots
 When:	July 2010
 Why:	Since 2.6.25, kvm supports user-allocated memory slots, which are
 	much more flexible than kernel-allocated slots.  All current userspace
 	supports the newer interface and this code can be removed with no
 	impact.
 Who:	Avi Kivity <avi@redhat.com>
 ----------------------------
 What:	KVM paravirt mmu host support
 When:	January 2011
 Why:	The paravirt mmu host support is slower than non-paravirt mmu, both
 	on newer and older hardware.  It is already not exposed to the guest,
 	and kept only for live migration purposes.
 Who:	Avi Kivity <avi@redhat.com>
 ----------------------------
 What:	iwlwifi 50XX module parameters
 When:	2.6.40
 Why:	The "..50" modules parameters were used to configure 5000 series and
 	up devices; different set of module parameters also available for 4965
 	with same functionalities. Consolidate both set into single place
 	in drivers/net/wireless/iwlwifi/iwl-agn.c
 Who:	Wey-Yi Guy <wey-yi.w.guy@intel.com>
 ----------------------------
 What:	iwl4965 alias support
 When:	2.6.40
 Why:	Internal alias support has been present in module-init-tools for some
 	time, the MODULE_ALIAS("iwl4965") boilerplate aliases can be removed
 	with no impact.
 Who:	Wey-Yi Guy <wey-yi.w.guy@intel.com>
 ---------------------------
 What:	xt_NOTRACK
 Files:	net/netfilter/xt_NOTRACK.c
 When:	April 2011
 Why:	Superseded by xt_CT
 Who:	Netfilter developer team <netfilter-devel@vger.kernel.org>
 ---------------------------
 What:	video4linux /dev/vtx teletext API support
 When:	2.6.35
 Files:	drivers/media/video/saa5246a.c drivers/media/video/saa5249.c
 	include/linux/videotext.h
 Why:	The vtx device nodes have been superseded by vbi device nodes
 	for many years. No applications exist that use the vtx support.
 	Of the two i2c drivers that actually support this API the saa5249
 	has been impossible to use for a year now and no known hardware
 	that supports this device exists. The saa5246a is theoretically
 	supported by the old mxb boards, but it never actually worked.
 	In summary: there is no hardware that can use this API and there
 	are no applications actually implementing this API.
 	The vtx support still reserves minors 192-223 and we would really
 	like to reuse those for upcoming new functionality. In the unlikely
 	event that new hardware appears that wants to use the functionality
 	provided by the vtx API, then that functionality should be build
 	around the sliced VBI API instead.
 Who:	Hans Verkuil <hverkuil@xs4all.nl>
 ----------------------------
 What:	IRQF_DISABLED
 When:	2.6.36
 Why:	The flag is a NOOP as we run interrupt handlers with interrupts disabled
 Who:	Thomas Gleixner <tglx@linutronix.de>
 ----------------------------
 What:	old ieee1394 subsystem (CONFIG_IEEE1394)
 When:	2.6.37
 Files:	drivers/ieee1394/ except init_ohci1394_dma.c
 Why:	superseded by drivers/firewire/ (CONFIG_FIREWIRE) which offers more
 	features, better performance, and better security, all with smaller
 	and more modern code base
 Who:	Stefan Richter <stefanr@s5r6.in-berlin.de>
 ----------------------------
 What:	The acpi_sleep=s4_nonvs command line option
 When:	2.6.37
 Files:	arch/x86/kernel/acpi/sleep.c
 Why:	superseded by acpi_sleep=nonvs
 Who:	Rafael J. Wysocki <rjw@sisk.pl>
 ----------------------------

Documentation/kvm/api.txt

Diff comments View file @ a1f4d39

 The Definitive KVM (Kernel-based Virtual Machine) API Documentation
 ===================================================================
 1. General description
 The kvm API is a set of ioctls that are issued to control various aspects
 of a virtual machine.  The ioctls belong to three classes
  - System ioctls: These query and set global attributes which affect the
    whole kvm subsystem.  In addition a system ioctl is used to create
    virtual machines
  - VM ioctls: These query and set attributes that affect an entire virtual
    machine, for example memory layout.  In addition a VM ioctl is used to
    create virtual cpus (vcpus).
    Only run VM ioctls from the same process (address space) that was used
    to create the VM.
  - vcpu ioctls: These query and set attributes that control the operation
    of a single virtual cpu.
    Only run vcpu ioctls from the same thread that was used to create the
    vcpu.
 2. File descriptors
 The kvm API is centered around file descriptors.  An initial
 open("/dev/kvm") obtains a handle to the kvm subsystem; this handle
 can be used to issue system ioctls.  A KVM_CREATE_VM ioctl on this
 handle will create a VM file descriptor which can be used to issue VM
 ioctls.  A KVM_CREATE_VCPU ioctl on a VM fd will create a virtual cpu
 and return a file descriptor pointing to it.  Finally, ioctls on a vcpu
 fd can be used to control the vcpu, including the important task of
 actually running guest code.
 In general file descriptors can be migrated among processes by means
 of fork() and the SCM_RIGHTS facility of unix domain socket.  These
 kinds of tricks are explicitly not supported by kvm.  While they will
 not cause harm to the host, their actual behavior is not guaranteed by
 the API.  The only supported use is one virtual machine per process,
 and one vcpu per thread.
 3. Extensions
 As of Linux 2.6.22, the KVM ABI has been stabilized: no backward
 incompatible change are allowed.  However, there is an extension
 facility that allows backward-compatible extensions to the API to be
 queried and used.
 The extension mechanism is not based on on the Linux version number.
 Instead, kvm defines extension identifiers and a facility to query
 whether a particular extension identifier is available.  If it is, a
 set of ioctls is available for application use.
 4. API description
 This section describes ioctls that can be used to control kvm guests.
 For each ioctl, the following information is provided along with a
 description:
   Capability: which KVM extension provides this ioctl.  Can be 'basic',
       which means that is will be provided by any kernel that supports
       API version 12 (see section 4.1), or a KVM_CAP_xyz constant, which
       means availability needs to be checked with KVM_CHECK_EXTENSION
       (see section 4.4).
   Architectures: which instruction set architectures provide this ioctl.
       x86 includes both i386 and x86_64.
   Type: system, vm, or vcpu.
   Parameters: what parameters are accepted by the ioctl.
   Returns: the return value.  General error numbers (EBADF, ENOMEM, EINVAL)
       are not detailed, but errors with specific meanings are.
 4.1 KVM_GET_API_VERSION
 Capability: basic
 Architectures: all
 Type: system ioctl
 Parameters: none
 Returns: the constant KVM_API_VERSION (=12)
 This identifies the API version as the stable kvm API. It is not
 expected that this number will change.  However, Linux 2.6.20 and
 2.6.21 report earlier versions; these are not documented and not
 supported.  Applications should refuse to run if KVM_GET_API_VERSION
 returns a value other than 12.  If this check passes, all ioctls
 described as 'basic' will be available.
 4.2 KVM_CREATE_VM
 Capability: basic
 Architectures: all
 Type: system ioctl
 Parameters: none
 Returns: a VM fd that can be used to control the new virtual machine.
 The new VM has no virtual cpus and no memory.  An mmap() of a VM fd
 will access the virtual machine's physical address space; offset zero
 corresponds to guest physical address zero.  Use of mmap() on a VM fd
 is discouraged if userspace memory allocation (KVM_CAP_USER_MEMORY) is
 available.
 4.3 KVM_GET_MSR_INDEX_LIST
 Capability: basic
 Architectures: x86
 Type: system
 Parameters: struct kvm_msr_list (in/out)
 Returns: 0 on success; -1 on error
 Errors:
   E2BIG:     the msr index list is to be to fit in the array specified by
              the user.
 struct kvm_msr_list {
 	__u32 nmsrs; /* number of msrs in entries */
 	__u32 indices[0];
 };
 This ioctl returns the guest msrs that are supported.  The list varies
 by kvm version and host processor, but does not change otherwise.  The
 user fills in the size of the indices array in nmsrs, and in return
 kvm adjusts nmsrs to reflect the actual number of msrs and fills in
 the indices array with their numbers.
 4.4 KVM_CHECK_EXTENSION
 Capability: basic
 Architectures: all
 Type: system ioctl
 Parameters: extension identifier (KVM_CAP_*)
 Returns: 0 if unsupported; 1 (or some other positive integer) if supported
 The API allows the application to query about extensions to the core
 kvm API.  Userspace passes an extension identifier (an integer) and
 receives an integer that describes the extension availability.
 Generally 0 means no and 1 means yes, but some extensions may report
 additional information in the integer return value.
 4.5 KVM_GET_VCPU_MMAP_SIZE
 Capability: basic
 Architectures: all
 Type: system ioctl
 Parameters: none
 Returns: size of vcpu mmap area, in bytes
 The KVM_RUN ioctl (cf.) communicates with userspace via a shared
 memory region.  This ioctl returns the size of that region.  See the
 KVM_RUN documentation for details.
 4.6 KVM_SET_MEMORY_REGION
 Capability: basic
 Architectures: all
 Type: vm ioctl
 Parameters: struct kvm_memory_region (in)
 Returns: 0 on success, -1 on error
 struct kvm_memory_region {
 	__u32 slot;
 	__u32 flags;
 	__u64 guest_phys_addr;
 	__u64 memory_size; /* bytes */
 };
 /* for kvm_memory_region::flags */
 #define KVM_MEM_LOG_DIRTY_PAGES  1UL
 This ioctl allows the user to create or modify a guest physical memory
 slot.  When changing an existing slot, it may be moved in the guest
 physical memory space, or its flags may be modified.  It may not be
 resized.  Slots may not overlap.
 The flags field supports just one flag, KVM_MEM_LOG_DIRTY_PAGES, which
 instructs kvm to keep track of writes to memory within the slot.  See
 the KVM_GET_DIRTY_LOG ioctl.
 It is recommended to use the KVM_SET_USER_MEMORY_REGION ioctl instead
 of this API, if available.  This newer API allows placing guest memory
 at specified locations in the host address space, yielding better
 control and easy access.
 4.6 KVM_CREATE_VCPU
 Capability: basic
 Architectures: all
 Type: vm ioctl
 Parameters: vcpu id (apic id on x86)
 Returns: vcpu fd on success, -1 on error
 This API adds a vcpu to a virtual machine.  The vcpu id is a small integer
 in the range [0, max_vcpus).
 4.7 KVM_GET_DIRTY_LOG (vm ioctl)
 Capability: basic
 Architectures: x86
 Type: vm ioctl
 Parameters: struct kvm_dirty_log (in/out)
 Returns: 0 on success, -1 on error
 /* for KVM_GET_DIRTY_LOG */
 struct kvm_dirty_log {
 	__u32 slot;
 	__u32 padding;
 	union {
 		void __user *dirty_bitmap; /* one bit per page */
 		__u64 padding;
 	};
 };
 Given a memory slot, return a bitmap containing any pages dirtied
 since the last call to this ioctl.  Bit 0 is the first page in the
 memory slot.  Ensure the entire structure is cleared to avoid padding
 issues.
 4.8 KVM_SET_MEMORY_ALIAS
 Capability: basic
 Architectures: x86
 Type: vm ioctl
 Parameters: struct kvm_memory_alias (in)
 Returns: 0 (success), -1 (error)
-struct kvm_memory_alias {
+This ioctl is obsolete and has been removed.
-	__u32 slot;  /* this has a different namespace than memory slots */
-	__u32 flags;
-	__u64 guest_phys_addr;
-	__u64 memory_size;
-	__u64 target_phys_addr;
-};
-Defines a guest physical address space region as an alias to another
-region.  Useful for aliased address, for example the VGA low memory
-window. Should not be used with userspace memory.
 4.9 KVM_RUN
 Capability: basic
 Architectures: all
 Type: vcpu ioctl
 Parameters: none
 Returns: 0 on success, -1 on error
 Errors:
   EINTR:     an unmasked signal is pending
 This ioctl is used to run a guest virtual cpu.  While there are no
 explicit parameters, there is an implicit parameter block that can be
 obtained by mmap()ing the vcpu fd at offset 0, with the size given by
 KVM_GET_VCPU_MMAP_SIZE.  The parameter block is formatted as a 'struct
 kvm_run' (see below).
 4.10 KVM_GET_REGS
 Capability: basic
 Architectures: all
 Type: vcpu ioctl
 Parameters: struct kvm_regs (out)
 Returns: 0 on success, -1 on error
 Reads the general purpose registers from the vcpu.
 /* x86 */
 struct kvm_regs {
 	/* out (KVM_GET_REGS) / in (KVM_SET_REGS) */
 	__u64 rax, rbx, rcx, rdx;
 	__u64 rsi, rdi, rsp, rbp;
 	__u64 r8,  r9,  r10, r11;
 	__u64 r12, r13, r14, r15;
 	__u64 rip, rflags;
 };
 4.11 KVM_SET_REGS
 Capability: basic
 Architectures: all
 Type: vcpu ioctl
 Parameters: struct kvm_regs (in)
 Returns: 0 on success, -1 on error
 Writes the general purpose registers into the vcpu.
 See KVM_GET_REGS for the data structure.
 4.12 KVM_GET_SREGS
 Capability: basic
 Architectures: x86
 Type: vcpu ioctl
 Parameters: struct kvm_sregs (out)
 Returns: 0 on success, -1 on error
 Reads special registers from the vcpu.
 /* x86 */
 struct kvm_sregs {
 	struct kvm_segment cs, ds, es, fs, gs, ss;
 	struct kvm_segment tr, ldt;
 	struct kvm_dtable gdt, idt;
 	__u64 cr0, cr2, cr3, cr4, cr8;
 	__u64 efer;
 	__u64 apic_base;
 	__u64 interrupt_bitmap[(KVM_NR_INTERRUPTS + 63) / 64];
 };
 interrupt_bitmap is a bitmap of pending external interrupts.  At most
 one bit may be set.  This interrupt has been acknowledged by the APIC
 but not yet injected into the cpu core.
 4.13 KVM_SET_SREGS
 Capability: basic
 Architectures: x86
 Type: vcpu ioctl
 Parameters: struct kvm_sregs (in)
 Returns: 0 on success, -1 on error
 Writes special registers into the vcpu.  See KVM_GET_SREGS for the
 data structures.
 4.14 KVM_TRANSLATE
 Capability: basic
 Architectures: x86
 Type: vcpu ioctl
 Parameters: struct kvm_translation (in/out)
 Returns: 0 on success, -1 on error
 Translates a virtual address according to the vcpu's current address
 translation mode.
 struct kvm_translation {
 	/* in */
 	__u64 linear_address;
 	/* out */
 	__u64 physical_address;
 	__u8  valid;
 	__u8  writeable;
 	__u8  usermode;
 	__u8  pad[5];
 };
 4.15 KVM_INTERRUPT
 Capability: basic
 Architectures: x86
 Type: vcpu ioctl
 Parameters: struct kvm_interrupt (in)
 Returns: 0 on success, -1 on error
 Queues a hardware interrupt vector to be injected.  This is only
 useful if in-kernel local APIC is not used.
 /* for KVM_INTERRUPT */
 struct kvm_interrupt {
 	/* in */
 	__u32 irq;
 };
 Note 'irq' is an interrupt vector, not an interrupt pin or line.
 4.16 KVM_DEBUG_GUEST
 Capability: basic
 Architectures: none
 Type: vcpu ioctl
 Parameters: none)
 Returns: -1 on error
 Support for this has been removed.  Use KVM_SET_GUEST_DEBUG instead.
 4.17 KVM_GET_MSRS
 Capability: basic
 Architectures: x86
 Type: vcpu ioctl
 Parameters: struct kvm_msrs (in/out)
 Returns: 0 on success, -1 on error
 Reads model-specific registers from the vcpu.  Supported msr indices can
 be obtained using KVM_GET_MSR_INDEX_LIST.
 struct kvm_msrs {
 	__u32 nmsrs; /* number of msrs in entries */
 	__u32 pad;
 	struct kvm_msr_entry entries[0];
 };
 struct kvm_msr_entry {
 	__u32 index;
 	__u32 reserved;
 	__u64 data;
 };
 Application code should set the 'nmsrs' member (which indicates the
 size of the entries array) and the 'index' member of each array entry.
 kvm will fill in the 'data' member.
 4.18 KVM_SET_MSRS
 Capability: basic
 Architectures: x86
 Type: vcpu ioctl
 Parameters: struct kvm_msrs (in)
 Returns: 0 on success, -1 on error
 Writes model-specific registers to the vcpu.  See KVM_GET_MSRS for the
 data structures.
 Application code should set the 'nmsrs' member (which indicates the
 size of the entries array), and the 'index' and 'data' members of each
 array entry.
 4.19 KVM_SET_CPUID
 Capability: basic
 Architectures: x86
 Type: vcpu ioctl
 Parameters: struct kvm_cpuid (in)
 Returns: 0 on success, -1 on error
 Defines the vcpu responses to the cpuid instruction.  Applications
 should use the KVM_SET_CPUID2 ioctl if available.
 struct kvm_cpuid_entry {
 	__u32 function;
 	__u32 eax;
 	__u32 ebx;
 	__u32 ecx;
 	__u32 edx;
 	__u32 padding;
 };
 /* for KVM_SET_CPUID */
 struct kvm_cpuid {
 	__u32 nent;
 	__u32 padding;
 	struct kvm_cpuid_entry entries[0];
 };
 4.20 KVM_SET_SIGNAL_MASK
 Capability: basic
 Architectures: x86
 Type: vcpu ioctl
 Parameters: struct kvm_signal_mask (in)
 Returns: 0 on success, -1 on error
 Defines which signals are blocked during execution of KVM_RUN.  This
 signal mask temporarily overrides the threads signal mask.  Any
 unblocked signal received (except SIGKILL and SIGSTOP, which retain
 their traditional behaviour) will cause KVM_RUN to return with -EINTR.
 Note the signal will only be delivered if not blocked by the original
 signal mask.
 /* for KVM_SET_SIGNAL_MASK */
 struct kvm_signal_mask {
 	__u32 len;
 	__u8  sigset[0];
 };
 4.21 KVM_GET_FPU
 Capability: basic
 Architectures: x86
 Type: vcpu ioctl
 Parameters: struct kvm_fpu (out)
 Returns: 0 on success, -1 on error
 Reads the floating point state from the vcpu.
 /* for KVM_GET_FPU and KVM_SET_FPU */
 struct kvm_fpu {
 	__u8  fpr[8][16];
 	__u16 fcw;
 	__u16 fsw;
 	__u8  ftwx;  /* in fxsave format */
 	__u8  pad1;
 	__u16 last_opcode;
 	__u64 last_ip;
 	__u64 last_dp;
 	__u8  xmm[16][16];
 	__u32 mxcsr;
 	__u32 pad2;
 };
 4.22 KVM_SET_FPU
 Capability: basic
 Architectures: x86
 Type: vcpu ioctl
 Parameters: struct kvm_fpu (in)
 Returns: 0 on success, -1 on error
 Writes the floating point state to the vcpu.
 /* for KVM_GET_FPU and KVM_SET_FPU */
 struct kvm_fpu {
 	__u8  fpr[8][16];
 	__u16 fcw;
 	__u16 fsw;
 	__u8  ftwx;  /* in fxsave format */
 	__u8  pad1;
 	__u16 last_opcode;
 	__u64 last_ip;
 	__u64 last_dp;
 	__u8  xmm[16][16];
 	__u32 mxcsr;
 	__u32 pad2;
 };
 4.23 KVM_CREATE_IRQCHIP
 Capability: KVM_CAP_IRQCHIP
 Architectures: x86, ia64
 Type: vm ioctl
 Parameters: none
 Returns: 0 on success, -1 on error
 Creates an interrupt controller model in the kernel.  On x86, creates a virtual
 ioapic, a virtual PIC (two PICs, nested), and sets up future vcpus to have a
 local APIC.  IRQ routing for GSIs 0-15 is set to both PIC and IOAPIC; GSI 16-23
 only go to the IOAPIC.  On ia64, a IOSAPIC is created.
 4.24 KVM_IRQ_LINE
 Capability: KVM_CAP_IRQCHIP
 Architectures: x86, ia64
 Type: vm ioctl
 Parameters: struct kvm_irq_level
 Returns: 0 on success, -1 on error
 Sets the level of a GSI input to the interrupt controller model in the kernel.
 Requires that an interrupt controller model has been previously created with
 KVM_CREATE_IRQCHIP.  Note that edge-triggered interrupts require the level
 to be set to 1 and then back to 0.
 struct kvm_irq_level {
 	union {
 		__u32 irq;     /* GSI */
 		__s32 status;  /* not used for KVM_IRQ_LEVEL */
 	};
 	__u32 level;           /* 0 or 1 */
 };
 4.25 KVM_GET_IRQCHIP
 Capability: KVM_CAP_IRQCHIP
 Architectures: x86, ia64
 Type: vm ioctl
 Parameters: struct kvm_irqchip (in/out)
 Returns: 0 on success, -1 on error
 Reads the state of a kernel interrupt controller created with
 KVM_CREATE_IRQCHIP into a buffer provided by the caller.
 struct kvm_irqchip {
 	__u32 chip_id;  /* 0 = PIC1, 1 = PIC2, 2 = IOAPIC */
 	__u32 pad;
         union {
 		char dummy[512];  /* reserving space */
 		struct kvm_pic_state pic;
 		struct kvm_ioapic_state ioapic;
 	} chip;
 };
 4.26 KVM_SET_IRQCHIP
 Capability: KVM_CAP_IRQCHIP
 Architectures: x86, ia64
 Type: vm ioctl
 Parameters: struct kvm_irqchip (in)
 Returns: 0 on success, -1 on error
 Sets the state of a kernel interrupt controller created with
 KVM_CREATE_IRQCHIP from a buffer provided by the caller.
 struct kvm_irqchip {
 	__u32 chip_id;  /* 0 = PIC1, 1 = PIC2, 2 = IOAPIC */
 	__u32 pad;
         union {
 		char dummy[512];  /* reserving space */
 		struct kvm_pic_state pic;
 		struct kvm_ioapic_state ioapic;
 	} chip;
 };
 4.27 KVM_XEN_HVM_CONFIG
 Capability: KVM_CAP_XEN_HVM
 Architectures: x86
 Type: vm ioctl
 Parameters: struct kvm_xen_hvm_config (in)
 Returns: 0 on success, -1 on error
 Sets the MSR that the Xen HVM guest uses to initialize its hypercall
 page, and provides the starting address and size of the hypercall
 blobs in userspace.  When the guest writes the MSR, kvm copies one
 page of a blob (32- or 64-bit, depending on the vcpu mode) to guest
 memory.
 struct kvm_xen_hvm_config {
 	__u32 flags;
 	__u32 msr;
 	__u64 blob_addr_32;
 	__u64 blob_addr_64;
 	__u8 blob_size_32;
 	__u8 blob_size_64;
 	__u8 pad2[30];
 };
 4.27 KVM_GET_CLOCK
 Capability: KVM_CAP_ADJUST_CLOCK
 Architectures: x86
 Type: vm ioctl
 Parameters: struct kvm_clock_data (out)
 Returns: 0 on success, -1 on error
 Gets the current timestamp of kvmclock as seen by the current guest. In
 conjunction with KVM_SET_CLOCK, it is used to ensure monotonicity on scenarios
 such as migration.
 struct kvm_clock_data {
 	__u64 clock;  /* kvmclock current value */
 	__u32 flags;
 	__u32 pad[9];
 };
 4.28 KVM_SET_CLOCK
 Capability: KVM_CAP_ADJUST_CLOCK
 Architectures: x86
 Type: vm ioctl
 Parameters: struct kvm_clock_data (in)
 Returns: 0 on success, -1 on error
 Sets the current timestamp of kvmclock to the value specified in its parameter.
 In conjunction with KVM_GET_CLOCK, it is used to ensure monotonicity on scenarios
 such as migration.
 struct kvm_clock_data {
 	__u64 clock;  /* kvmclock current value */
 	__u32 flags;
 	__u32 pad[9];
 };
 4.29 KVM_GET_VCPU_EVENTS
 Capability: KVM_CAP_VCPU_EVENTS
 Extended by: KVM_CAP_INTR_SHADOW
 Architectures: x86
 Type: vm ioctl
 Parameters: struct kvm_vcpu_event (out)
 Returns: 0 on success, -1 on error
 Gets currently pending exceptions, interrupts, and NMIs as well as related
 states of the vcpu.
 struct kvm_vcpu_events {
 	struct {
 		__u8 injected;
 		__u8 nr;
 		__u8 has_error_code;
 		__u8 pad;
 		__u32 error_code;
 	} exception;
 	struct {
 		__u8 injected;
 		__u8 nr;
 		__u8 soft;
 		__u8 shadow;
 	} interrupt;
 	struct {
 		__u8 injected;
 		__u8 pending;
 		__u8 masked;
 		__u8 pad;
 	} nmi;
 	__u32 sipi_vector;
 	__u32 flags;
 };
 KVM_VCPUEVENT_VALID_SHADOW may be set in the flags field to signal that
 interrupt.shadow contains a valid state. Otherwise, this field is undefined.
 4.30 KVM_SET_VCPU_EVENTS
 Capability: KVM_CAP_VCPU_EVENTS
 Extended by: KVM_CAP_INTR_SHADOW
 Architectures: x86
 Type: vm ioctl
 Parameters: struct kvm_vcpu_event (in)
 Returns: 0 on success, -1 on error
 Set pending exceptions, interrupts, and NMIs as well as related states of the
 vcpu.
 See KVM_GET_VCPU_EVENTS for the data structure.
 Fields that may be modified asynchronously by running VCPUs can be excluded
 from the update. These fields are nmi.pending and sipi_vector. Keep the
 corresponding bits in the flags field cleared to suppress overwriting the
 current in-kernel state. The bits are:
 KVM_VCPUEVENT_VALID_NMI_PENDING - transfer nmi.pending to the kernel
 KVM_VCPUEVENT_VALID_SIPI_VECTOR - transfer sipi_vector
 If KVM_CAP_INTR_SHADOW is available, KVM_VCPUEVENT_VALID_SHADOW can be set in
 the flags field to signal that interrupt.shadow contains a valid state and
 shall be written into the VCPU.
 4.32 KVM_GET_DEBUGREGS
 Capability: KVM_CAP_DEBUGREGS
 Architectures: x86
 Type: vm ioctl
 Parameters: struct kvm_debugregs (out)
 Returns: 0 on success, -1 on error
 Reads debug registers from the vcpu.
 struct kvm_debugregs {
 	__u64 db[4];
 	__u64 dr6;
 	__u64 dr7;
 	__u64 flags;
 	__u64 reserved[9];
 };
 4.33 KVM_SET_DEBUGREGS
 Capability: KVM_CAP_DEBUGREGS
 Architectures: x86
 Type: vm ioctl
 Parameters: struct kvm_debugregs (in)
 Returns: 0 on success, -1 on error
 Writes debug registers into the vcpu.
 See KVM_GET_DEBUGREGS for the data structure. The flags field is unused
 yet and must be cleared on entry.
 4.34 KVM_SET_USER_MEMORY_REGION
 Capability: KVM_CAP_USER_MEM
 Architectures: all
 Type: vm ioctl
 Parameters: struct kvm_userspace_memory_region (in)
 Returns: 0 on success, -1 on error
 struct kvm_userspace_memory_region {
 	__u32 slot;
 	__u32 flags;
 	__u64 guest_phys_addr;
 	__u64 memory_size; /* bytes */
 	__u64 userspace_addr; /* start of the userspace allocated memory */
 };
 /* for kvm_memory_region::flags */
 #define KVM_MEM_LOG_DIRTY_PAGES  1UL
 This ioctl allows the user to create or modify a guest physical memory
 slot.  When changing an existing slot, it may be moved in the guest
 physical memory space, or its flags may be modified.  It may not be
 resized.  Slots may not overlap in guest physical address space.
 Memory for the region is taken starting at the address denoted by the
 field userspace_addr, which must point at user addressable memory for
 the entire memory slot size.  Any object may back this memory, including
 anonymous memory, ordinary files, and hugetlbfs.
 It is recommended that the lower 21 bits of guest_phys_addr and userspace_addr
 be identical.  This allows large pages in the guest to be backed by large
 pages in the host.
 The flags field supports just one flag, KVM_MEM_LOG_DIRTY_PAGES, which
 instructs kvm to keep track of writes to memory within the slot.  See
 the KVM_GET_DIRTY_LOG ioctl.
 When the KVM_CAP_SYNC_MMU capability, changes in the backing of the memory
 region are automatically reflected into the guest.  For example, an mmap()
 that affects the region will be made visible immediately.  Another example
 is madvise(MADV_DROP).
 It is recommended to use this API instead of the KVM_SET_MEMORY_REGION ioctl.
 The KVM_SET_MEMORY_REGION does not allow fine grained control over memory
 allocation and is deprecated.
 4.35 KVM_SET_TSS_ADDR
 Capability: KVM_CAP_SET_TSS_ADDR
 Architectures: x86
 Type: vm ioctl
 Parameters: unsigned long tss_address (in)
 Returns: 0 on success, -1 on error
 This ioctl defines the physical address of a three-page region in the guest
 physical address space.  The region must be within the first 4GB of the
 guest physical address space and must not conflict with any memory slot
 or any mmio address.  The guest may malfunction if it accesses this memory
 region.
 This ioctl is required on Intel-based hosts.  This is needed on Intel hardware
 because of a quirk in the virtualization implementation (see the internals
 documentation when it pops into existence).
 4.36 KVM_ENABLE_CAP
 Capability: KVM_CAP_ENABLE_CAP
 Architectures: ppc
 Type: vcpu ioctl
 Parameters: struct kvm_enable_cap (in)
 Returns: 0 on success; -1 on error
 +Not all extensions are enabled by default. Using this ioctl the application
 can enable an extension, making it available to the guest.
 On systems that do not support this ioctl, it always fails. On systems that
 do support it, it only works for extensions that are supported for enablement.
 To check if a capability can be enabled, the KVM_CHECK_EXTENSION ioctl should
 be used.
 struct kvm_enable_cap {
        /* in */
        __u32 cap;
 The capability that is supposed to get enabled.
        __u32 flags;
 A bitfield indicating future enhancements. Has to be 0 for now.
        __u64 args[4];
 Arguments for enabling a feature. If a feature needs initial values to
 function properly, this is the place to put them.
        __u8  pad[64];
 };
 4.37 KVM_GET_MP_STATE
 Capability: KVM_CAP_MP_STATE
 Architectures: x86, ia64
 Type: vcpu ioctl
 Parameters: struct kvm_mp_state (out)
 Returns: 0 on success; -1 on error
 struct kvm_mp_state {
 	__u32 mp_state;
 };
 Returns the vcpu's current "multiprocessing state" (though also valid on
 uniprocessor guests).
 Possible values are:
  - KVM_MP_STATE_RUNNABLE:        the vcpu is currently running
  - KVM_MP_STATE_UNINITIALIZED:   the vcpu is an application processor (AP)
                                  which has not yet received an INIT signal
  - KVM_MP_STATE_INIT_RECEIVED:   the vcpu has received an INIT signal, and is
                                  now ready for a SIPI
  - KVM_MP_STATE_HALTED:          the vcpu has executed a HLT instruction and
                                  is waiting for an interrupt
  - KVM_MP_STATE_SIPI_RECEIVED:   the vcpu has just received a SIPI (vector
                                  accesible via KVM_GET_VCPU_EVENTS)
 This ioctl is only useful after KVM_CREATE_IRQCHIP.  Without an in-kernel
 irqchip, the multiprocessing state must be maintained by userspace.
 4.38 KVM_SET_MP_STATE
 Capability: KVM_CAP_MP_STATE
 Architectures: x86, ia64
 Type: vcpu ioctl
 Parameters: struct kvm_mp_state (in)
 Returns: 0 on success; -1 on error
 Sets the vcpu's current "multiprocessing state"; see KVM_GET_MP_STATE for
 arguments.
 This ioctl is only useful after KVM_CREATE_IRQCHIP.  Without an in-kernel
 irqchip, the multiprocessing state must be maintained by userspace.
 4.39 KVM_SET_IDENTITY_MAP_ADDR
 Capability: KVM_CAP_SET_IDENTITY_MAP_ADDR
 Architectures: x86
 Type: vm ioctl
 Parameters: unsigned long identity (in)
 Returns: 0 on success, -1 on error
 This ioctl defines the physical address of a one-page region in the guest
 physical address space.  The region must be within the first 4GB of the
 guest physical address space and must not conflict with any memory slot
 or any mmio address.  The guest may malfunction if it accesses this memory
 region.
 This ioctl is required on Intel-based hosts.  This is needed on Intel hardware
 because of a quirk in the virtualization implementation (see the internals
 documentation when it pops into existence).
 4.40 KVM_SET_BOOT_CPU_ID
 Capability: KVM_CAP_SET_BOOT_CPU_ID
 Architectures: x86, ia64
 Type: vm ioctl
 Parameters: unsigned long vcpu_id
 Returns: 0 on success, -1 on error
 Define which vcpu is the Bootstrap Processor (BSP).  Values are the same
 as the vcpu id in KVM_CREATE_VCPU.  If this ioctl is not called, the default
 is vcpu 0.
 4.41 KVM_GET_XSAVE
 Capability: KVM_CAP_XSAVE
 Architectures: x86
 Type: vcpu ioctl
 Parameters: struct kvm_xsave (out)
 Returns: 0 on success, -1 on error
 struct kvm_xsave {
 	__u32 region[1024];
 };
 This ioctl would copy current vcpu's xsave struct to the userspace.
 4.42 KVM_SET_XSAVE
 Capability: KVM_CAP_XSAVE
 Architectures: x86
 Type: vcpu ioctl
 Parameters: struct kvm_xsave (in)
 Returns: 0 on success, -1 on error
 struct kvm_xsave {
 	__u32 region[1024];
 };
 This ioctl would copy userspace's xsave struct to the kernel.
 4.43 KVM_GET_XCRS
 Capability: KVM_CAP_XCRS
 Architectures: x86
 Type: vcpu ioctl
 Parameters: struct kvm_xcrs (out)
 Returns: 0 on success, -1 on error
 struct kvm_xcr {
 	__u32 xcr;
 	__u32 reserved;
 	__u64 value;
 };
 struct kvm_xcrs {
 	__u32 nr_xcrs;
 	__u32 flags;
 	struct kvm_xcr xcrs[KVM_MAX_XCRS];
 	__u64 padding[16];
 };
 This ioctl would copy current vcpu's xcrs to the userspace.
 4.44 KVM_SET_XCRS
 Capability: KVM_CAP_XCRS
 Architectures: x86
 Type: vcpu ioctl
 Parameters: struct kvm_xcrs (in)
 Returns: 0 on success, -1 on error
 struct kvm_xcr {
 	__u32 xcr;
 	__u32 reserved;
 	__u64 value;
 };
 struct kvm_xcrs {
 	__u32 nr_xcrs;
 	__u32 flags;
 	struct kvm_xcr xcrs[KVM_MAX_XCRS];
 	__u64 padding[16];
 };
 This ioctl would set vcpu's xcr to the value userspace specified.
 5. The kvm_run structure
 Application code obtains a pointer to the kvm_run structure by
 mmap()ing a vcpu fd.  From that point, application code can control
 execution by changing fields in kvm_run prior to calling the KVM_RUN
 ioctl, and obtain information about the reason KVM_RUN returned by
 looking up structure members.
 struct kvm_run {
 	/* in */
 	__u8 request_interrupt_window;
 Request that KVM_RUN return when it becomes possible to inject external
 interrupts into the guest.  Useful in conjunction with KVM_INTERRUPT.
 	__u8 padding1[7];
 	/* out */
 	__u32 exit_reason;
 When KVM_RUN has returned successfully (return value 0), this informs
 application code why KVM_RUN has returned.  Allowable values for this
 field are detailed below.
 	__u8 ready_for_interrupt_injection;
 If request_interrupt_window has been specified, this field indicates
 an interrupt can be injected now with KVM_INTERRUPT.
 	__u8 if_flag;
 The value of the current interrupt flag.  Only valid if in-kernel
 local APIC is not used.
 	__u8 padding2[2];
 	/* in (pre_kvm_run), out (post_kvm_run) */
 	__u64 cr8;
 The value of the cr8 register.  Only valid if in-kernel local APIC is
 not used.  Both input and output.
 	__u64 apic_base;
 The value of the APIC BASE msr.  Only valid if in-kernel local
 APIC is not used.  Both input and output.
 	union {
 		/* KVM_EXIT_UNKNOWN */
 		struct {
 			__u64 hardware_exit_reason;
 		} hw;
 If exit_reason is KVM_EXIT_UNKNOWN, the vcpu has exited due to unknown
 reasons.  Further architecture-specific information is available in
 hardware_exit_reason.
 		/* KVM_EXIT_FAIL_ENTRY */
 		struct {
 			__u64 hardware_entry_failure_reason;
 		} fail_entry;
 If exit_reason is KVM_EXIT_FAIL_ENTRY, the vcpu could not be run due
 to unknown reasons.  Further architecture-specific information is
 available in hardware_entry_failure_reason.
 		/* KVM_EXIT_EXCEPTION */
 		struct {
 			__u32 exception;
 			__u32 error_code;
 		} ex;
 Unused.
 		/* KVM_EXIT_IO */
 		struct {
 #define KVM_EXIT_IO_IN  0
 #define KVM_EXIT_IO_OUT 1
 			__u8 direction;
 			__u8 size; /* bytes */
 			__u16 port;
 			__u32 count;
 			__u64 data_offset; /* relative to kvm_run start */
 		} io;
 If exit_reason is KVM_EXIT_IO, then the vcpu has
 executed a port I/O instruction which could not be satisfied by kvm.
 data_offset describes where the data is located (KVM_EXIT_IO_OUT) or
 where kvm expects application code to place the data for the next
 KVM_RUN invocation (KVM_EXIT_IO_IN).  Data format is a packed array.
 		struct {
 			struct kvm_debug_exit_arch arch;
 		} debug;
 Unused.
 		/* KVM_EXIT_MMIO */
 		struct {
 			__u64 phys_addr;
 			__u8  data[8];
 			__u32 len;
 			__u8  is_write;
 		} mmio;
 If exit_reason is KVM_EXIT_MMIO, then the vcpu has
 executed a memory-mapped I/O instruction which could not be satisfied
 by kvm.  The 'data' member contains the written data if 'is_write' is
 true, and should be filled by application code otherwise.
 NOTE: For KVM_EXIT_IO, KVM_EXIT_MMIO and KVM_EXIT_OSI, the corresponding
 operations are complete (and guest state is consistent) only after userspace
 has re-entered the kernel with KVM_RUN.  The kernel side will first finish
 incomplete operations and then check for pending signals.  Userspace
 can re-enter the guest with an unmasked signal pending to complete
 pending operations.
 		/* KVM_EXIT_HYPERCALL */
 		struct {
 			__u64 nr;
 			__u64 args[6];
 			__u64 ret;
 			__u32 longmode;
 			__u32 pad;
 		} hypercall;
 Unused.  This was once used for 'hypercall to userspace'.  To implement
 such functionality, use KVM_EXIT_IO (x86) or KVM_EXIT_MMIO (all except s390).
 Note KVM_EXIT_IO is significantly faster than KVM_EXIT_MMIO.
 		/* KVM_EXIT_TPR_ACCESS */
 		struct {
 			__u64 rip;
 			__u32 is_write;
 			__u32 pad;
 		} tpr_access;
 To be documented (KVM_TPR_ACCESS_REPORTING).
 		/* KVM_EXIT_S390_SIEIC */
 		struct {
 			__u8 icptcode;
 			__u64 mask; /* psw upper half */
 			__u64 addr; /* psw lower half */
 			__u16 ipa;
 			__u32 ipb;
 		} s390_sieic;
 s390 specific.
 		/* KVM_EXIT_S390_RESET */
 #define KVM_S390_RESET_POR       1
 #define KVM_S390_RESET_CLEAR     2
 #define KVM_S390_RESET_SUBSYSTEM 4
 #define KVM_S390_RESET_CPU_INIT  8
 #define KVM_S390_RESET_IPL       16
 		__u64 s390_reset_flags;
 s390 specific.
 		/* KVM_EXIT_DCR */
 		struct {
 			__u32 dcrn;
 			__u32 data;
 			__u8  is_write;
 		} dcr;
 powerpc specific.
 		/* KVM_EXIT_OSI */
 		struct {
 			__u64 gprs[32];
 		} osi;
 MOL uses a special hypercall interface it calls 'OSI'. To enable it, we catch
 hypercalls and exit with this exit struct that contains all the guest gprs.
 If exit_reason is KVM_EXIT_OSI, then the vcpu has triggered such a hypercall.
 Userspace can now handle the hypercall and when it's done modify the gprs as
 necessary. Upon guest entry all guest GPRs will then be replaced by the values
 in this struct.
 		/* Fix the size of the union. */
 		char padding[256];
 	};
 };

arch/ia64/kvm/kvm-ia64.c

Diff comments View file @ a1f4d39

 /*
  * kvm_ia64.c: Basic KVM suppport On Itanium series processors
  *
  *
  * 	Copyright (C) 2007, Intel Corporation.
  *  	Xiantao Zhang  (xiantao.zhang@intel.com)
  *
  * This program is free software; you can redistribute it and/or modify it
  * under the terms and conditions of the GNU General Public License,
  * version 2, as published by the Free Software Foundation.
  *
  * This program is distributed in the hope it will be useful, but WITHOUT
  * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
  * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
  * more details.
  *
  * You should have received a copy of the GNU General Public License along with
  * this program; if not, write to the Free Software Foundation, Inc., 59 Temple
  * Place - Suite 330, Boston, MA 02111-1307 USA.
  *
  */
 #include <linux/module.h>
 #include <linux/errno.h>
 #include <linux/percpu.h>
 #include <linux/fs.h>
 #include <linux/slab.h>
 #include <linux/smp.h>
 #include <linux/kvm_host.h>
 #include <linux/kvm.h>
 #include <linux/bitops.h>
 #include <linux/hrtimer.h>
 #include <linux/uaccess.h>
 #include <linux/iommu.h>
 #include <linux/intel-iommu.h>
 #include <asm/pgtable.h>
 #include <asm/gcc_intrin.h>
 #include <asm/pal.h>
 #include <asm/cacheflush.h>
 #include <asm/div64.h>
 #include <asm/tlb.h>
 #include <asm/elf.h>
 #include <asm/sn/addrs.h>
 #include <asm/sn/clksupport.h>
 #include <asm/sn/shub_mmr.h>
 #include "misc.h"
 #include "vti.h"
 #include "iodev.h"
 #include "ioapic.h"
 #include "lapic.h"
 #include "irq.h"
 static unsigned long kvm_vmm_base;
 static unsigned long kvm_vsa_base;
 static unsigned long kvm_vm_buffer;
 static unsigned long kvm_vm_buffer_size;
 unsigned long kvm_vmm_gp;
 static long vp_env_info;
 static struct kvm_vmm_info *kvm_vmm_info;
 static DEFINE_PER_CPU(struct kvm_vcpu *, last_vcpu);
 struct kvm_stats_debugfs_item debugfs_entries[] = {
 	{ NULL }
 };
 static unsigned long kvm_get_itc(struct kvm_vcpu *vcpu)
 {
 #if defined(CONFIG_IA64_SGI_SN2) || defined(CONFIG_IA64_GENERIC)
 	if (vcpu->kvm->arch.is_sn2)
 		return rtc_time();
 	else
 #endif
 		return ia64_getreg(_IA64_REG_AR_ITC);
 }
 static void kvm_flush_icache(unsigned long start, unsigned long len)
 {
 	int l;
 	for (l = 0; l < (len + 32); l += 32)
 		ia64_fc((void *)(start + l));
 	ia64_sync_i();
 	ia64_srlz_i();
 }
 static void kvm_flush_tlb_all(void)
 {
 	unsigned long i, j, count0, count1, stride0, stride1, addr;
 	long flags;
 	addr    = local_cpu_data->ptce_base;
 	count0  = local_cpu_data->ptce_count[0];
 	count1  = local_cpu_data->ptce_count[1];
 	stride0 = local_cpu_data->ptce_stride[0];
 	stride1 = local_cpu_data->ptce_stride[1];
 	local_irq_save(flags);
 	for (i = 0; i < count0; ++i) {
 		for (j = 0; j < count1; ++j) {
 			ia64_ptce(addr);
 			addr += stride1;
 		}
 		addr += stride0;
 	}
 	local_irq_restore(flags);
 	ia64_srlz_i();			/* srlz.i implies srlz.d */
 }
 long ia64_pal_vp_create(u64 *vpd, u64 *host_iva, u64 *opt_handler)
 {
 	struct ia64_pal_retval iprv;
 	PAL_CALL_STK(iprv, PAL_VP_CREATE, (u64)vpd, (u64)host_iva,
 			(u64)opt_handler);
 	return iprv.status;
 }
 static  DEFINE_SPINLOCK(vp_lock);
 int kvm_arch_hardware_enable(void *garbage)
 {
 	long  status;
 	long  tmp_base;
 	unsigned long pte;
 	unsigned long saved_psr;
 	int slot;
 	pte = pte_val(mk_pte_phys(__pa(kvm_vmm_base), PAGE_KERNEL));
 	local_irq_save(saved_psr);
 	slot = ia64_itr_entry(0x3, KVM_VMM_BASE, pte, KVM_VMM_SHIFT);
 	local_irq_restore(saved_psr);
 	if (slot < 0)
 		return -EINVAL;
 	spin_lock(&vp_lock);
 	status = ia64_pal_vp_init_env(kvm_vsa_base ?
 				VP_INIT_ENV : VP_INIT_ENV_INITALIZE,
 			__pa(kvm_vm_buffer), KVM_VM_BUFFER_BASE, &tmp_base);
 	if (status != 0) {
 		spin_unlock(&vp_lock);
 		printk(KERN_WARNING"kvm: Failed to Enable VT Support!!!!\n");
 		return -EINVAL;
 	}
 	if (!kvm_vsa_base) {
 		kvm_vsa_base = tmp_base;
 		printk(KERN_INFO"kvm: kvm_vsa_base:0x%lx\n", kvm_vsa_base);
 	}
 	spin_unlock(&vp_lock);
 	ia64_ptr_entry(0x3, slot);
 	return 0;
 }
 void kvm_arch_hardware_disable(void *garbage)
 {
 	long status;
 	int slot;
 	unsigned long pte;
 	unsigned long saved_psr;
 	unsigned long host_iva = ia64_getreg(_IA64_REG_CR_IVA);
 	pte = pte_val(mk_pte_phys(__pa(kvm_vmm_base),
 				PAGE_KERNEL));
 	local_irq_save(saved_psr);
 	slot = ia64_itr_entry(0x3, KVM_VMM_BASE, pte, KVM_VMM_SHIFT);
 	local_irq_restore(saved_psr);
 	if (slot < 0)
 		return;
 	status = ia64_pal_vp_exit_env(host_iva);
 	if (status)
 		printk(KERN_DEBUG"kvm: Failed to disable VT support! :%ld\n",
 				status);
 	ia64_ptr_entry(0x3, slot);
 }
 void kvm_arch_check_processor_compat(void *rtn)
 {
 	*(int *)rtn = 0;
 }
 int kvm_dev_ioctl_check_extension(long ext)
 {
 	int r;
 	switch (ext) {
 	case KVM_CAP_IRQCHIP:
 	case KVM_CAP_MP_STATE:
 	case KVM_CAP_IRQ_INJECT_STATUS:
 		r = 1;
 		break;
 	case KVM_CAP_COALESCED_MMIO:
 		r = KVM_COALESCED_MMIO_PAGE_OFFSET;
 		break;
 	case KVM_CAP_IOMMU:
 		r = iommu_found();
 		break;
 	default:
 		r = 0;
 	}
 	return r;
 }
 static int handle_vm_error(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
 {
 	kvm_run->exit_reason = KVM_EXIT_UNKNOWN;
 	kvm_run->hw.hardware_exit_reason = 1;
 	return 0;
 }
 static int handle_mmio(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
 {
 	struct kvm_mmio_req *p;
 	struct kvm_io_device *mmio_dev;
 	int r;
 	p = kvm_get_vcpu_ioreq(vcpu);
 	if ((p->addr & PAGE_MASK) == IOAPIC_DEFAULT_BASE_ADDRESS)
 		goto mmio;
 	vcpu->mmio_needed = 1;
 	vcpu->mmio_phys_addr = kvm_run->mmio.phys_addr = p->addr;
 	vcpu->mmio_size = kvm_run->mmio.len = p->size;
 	vcpu->mmio_is_write = kvm_run->mmio.is_write = !p->dir;
 	if (vcpu->mmio_is_write)
 		memcpy(vcpu->mmio_data, &p->data, p->size);
 	memcpy(kvm_run->mmio.data, &p->data, p->size);
 	kvm_run->exit_reason = KVM_EXIT_MMIO;
 	return 0;
 mmio:
 	if (p->dir)
 		r = kvm_io_bus_read(vcpu->kvm, KVM_MMIO_BUS, p->addr,
 				    p->size, &p->data);
 	else
 		r = kvm_io_bus_write(vcpu->kvm, KVM_MMIO_BUS, p->addr,
 				     p->size, &p->data);
 	if (r)
 		printk(KERN_ERR"kvm: No iodevice found! addr:%lx\n", p->addr);
 	p->state = STATE_IORESP_READY;
 	return 1;
 }
 static int handle_pal_call(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
 {
 	struct exit_ctl_data *p;
 	p = kvm_get_exit_data(vcpu);
 	if (p->exit_reason == EXIT_REASON_PAL_CALL)
 		return kvm_pal_emul(vcpu, kvm_run);
 	else {
 		kvm_run->exit_reason = KVM_EXIT_UNKNOWN;
 		kvm_run->hw.hardware_exit_reason = 2;
 		return 0;
 	}
 }
 static int handle_sal_call(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
 {
 	struct exit_ctl_data *p;
 	p = kvm_get_exit_data(vcpu);
 	if (p->exit_reason == EXIT_REASON_SAL_CALL) {
 		kvm_sal_emul(vcpu);
 		return 1;
 	} else {
 		kvm_run->exit_reason = KVM_EXIT_UNKNOWN;
 		kvm_run->hw.hardware_exit_reason = 3;
 		return 0;
 	}
 }
 static int __apic_accept_irq(struct kvm_vcpu *vcpu, uint64_t vector)
 {
 	struct vpd *vpd = to_host(vcpu->kvm, vcpu->arch.vpd);
 	if (!test_and_set_bit(vector, &vpd->irr[0])) {
 		vcpu->arch.irq_new_pending = 1;
 		kvm_vcpu_kick(vcpu);
 		return 1;
 	}
 	return 0;
 }
 /*
  *  offset: address offset to IPI space.
  *  value:  deliver value.
  */
 static void vcpu_deliver_ipi(struct kvm_vcpu *vcpu, uint64_t dm,
 				uint64_t vector)
 {
 	switch (dm) {
 	case SAPIC_FIXED:
 		break;
 	case SAPIC_NMI:
 		vector = 2;
 		break;
 	case SAPIC_EXTINT:
 		vector = 0;
 		break;
 	case SAPIC_INIT:
 	case SAPIC_PMI:
 	default:
 		printk(KERN_ERR"kvm: Unimplemented Deliver reserved IPI!\n");
 		return;
 	}
 	__apic_accept_irq(vcpu, vector);
 }
 static struct kvm_vcpu *lid_to_vcpu(struct kvm *kvm, unsigned long id,
 			unsigned long eid)
 {
 	union ia64_lid lid;
 	int i;
 	struct kvm_vcpu *vcpu;
 	kvm_for_each_vcpu(i, vcpu, kvm) {
 		lid.val = VCPU_LID(vcpu);
 		if (lid.id == id && lid.eid == eid)
 			return vcpu;
 	}
 	return NULL;
 }
 static int handle_ipi(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
 {
 	struct exit_ctl_data *p = kvm_get_exit_data(vcpu);
 	struct kvm_vcpu *target_vcpu;
 	struct kvm_pt_regs *regs;
 	union ia64_ipi_a addr = p->u.ipi_data.addr;
 	union ia64_ipi_d data = p->u.ipi_data.data;
 	target_vcpu = lid_to_vcpu(vcpu->kvm, addr.id, addr.eid);
 	if (!target_vcpu)
 		return handle_vm_error(vcpu, kvm_run);
 	if (!target_vcpu->arch.launched) {
 		regs = vcpu_regs(target_vcpu);
 		regs->cr_iip = vcpu->kvm->arch.rdv_sal_data.boot_ip;
 		regs->r1 = vcpu->kvm->arch.rdv_sal_data.boot_gp;
 		target_vcpu->arch.mp_state = KVM_MP_STATE_RUNNABLE;
 		if (waitqueue_active(&target_vcpu->wq))
 			wake_up_interruptible(&target_vcpu->wq);
 	} else {
 		vcpu_deliver_ipi(target_vcpu, data.dm, data.vector);
 		if (target_vcpu != vcpu)
 			kvm_vcpu_kick(target_vcpu);
 	}
 	return 1;
 }
 struct call_data {
 	struct kvm_ptc_g ptc_g_data;
 	struct kvm_vcpu *vcpu;
 };
 static void vcpu_global_purge(void *info)
 {
 	struct call_data *p = (struct call_data *)info;
 	struct kvm_vcpu *vcpu = p->vcpu;
 	if (test_bit(KVM_REQ_TLB_FLUSH, &vcpu->requests))
 		return;
 	set_bit(KVM_REQ_PTC_G, &vcpu->requests);
 	if (vcpu->arch.ptc_g_count < MAX_PTC_G_NUM) {
 		vcpu->arch.ptc_g_data[vcpu->arch.ptc_g_count++] =
 							p->ptc_g_data;
 	} else {
 		clear_bit(KVM_REQ_PTC_G, &vcpu->requests);
 		vcpu->arch.ptc_g_count = 0;
 		set_bit(KVM_REQ_TLB_FLUSH, &vcpu->requests);
 	}
 }
 static int handle_global_purge(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
 {
 	struct exit_ctl_data *p = kvm_get_exit_data(vcpu);
 	struct kvm *kvm = vcpu->kvm;
 	struct call_data call_data;
 	int i;
 	struct kvm_vcpu *vcpui;
 	call_data.ptc_g_data = p->u.ptc_g_data;
 	kvm_for_each_vcpu(i, vcpui, kvm) {
 		if (vcpui->arch.mp_state == KVM_MP_STATE_UNINITIALIZED ||
 				vcpu == vcpui)
 			continue;
 		if (waitqueue_active(&vcpui->wq))
 			wake_up_interruptible(&vcpui->wq);
 		if (vcpui->cpu != -1) {
 			call_data.vcpu = vcpui;
 			smp_call_function_single(vcpui->cpu,
 					vcpu_global_purge, &call_data, 1);
 		} else
 			printk(KERN_WARNING"kvm: Uninit vcpu received ipi!\n");
 	}
 	return 1;
 }
 static int handle_switch_rr6(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
 {
 	return 1;
 }
 static int kvm_sn2_setup_mappings(struct kvm_vcpu *vcpu)
 {
 	unsigned long pte, rtc_phys_addr, map_addr;
 	int slot;
 	map_addr = KVM_VMM_BASE + (1UL << KVM_VMM_SHIFT);
 	rtc_phys_addr = LOCAL_MMR_OFFSET | SH_RTC;
 	pte = pte_val(mk_pte_phys(rtc_phys_addr, PAGE_KERNEL_UC));
 	slot = ia64_itr_entry(0x3, map_addr, pte, PAGE_SHIFT);
 	vcpu->arch.sn_rtc_tr_slot = slot;
 	if (slot < 0) {
 		printk(KERN_ERR "Mayday mayday! RTC mapping failed!\n");
 		slot = 0;
 	}
 	return slot;
 }
 int kvm_emulate_halt(struct kvm_vcpu *vcpu)
 {
 	ktime_t kt;
 	long itc_diff;
 	unsigned long vcpu_now_itc;
 	unsigned long expires;
 	struct hrtimer *p_ht = &vcpu->arch.hlt_timer;
 	unsigned long cyc_per_usec = local_cpu_data->cyc_per_usec;
 	struct vpd *vpd = to_host(vcpu->kvm, vcpu->arch.vpd);
 	if (irqchip_in_kernel(vcpu->kvm)) {
 		vcpu_now_itc = kvm_get_itc(vcpu) + vcpu->arch.itc_offset;
 		if (time_after(vcpu_now_itc, vpd->itm)) {
 			vcpu->arch.timer_check = 1;
 			return 1;
 		}
 		itc_diff = vpd->itm - vcpu_now_itc;
 		if (itc_diff < 0)
 			itc_diff = -itc_diff;
 		expires = div64_u64(itc_diff, cyc_per_usec);
 		kt = ktime_set(0, 1000 * expires);
 		vcpu->arch.ht_active = 1;
 		hrtimer_start(p_ht, kt, HRTIMER_MODE_ABS);
 		vcpu->arch.mp_state = KVM_MP_STATE_HALTED;
 		kvm_vcpu_block(vcpu);
 		hrtimer_cancel(p_ht);
 		vcpu->arch.ht_active = 0;
 		if (test_and_clear_bit(KVM_REQ_UNHALT, &vcpu->requests) ||
 				kvm_cpu_has_pending_timer(vcpu))
 			if (vcpu->arch.mp_state == KVM_MP_STATE_HALTED)
 				vcpu->arch.mp_state = KVM_MP_STATE_RUNNABLE;
 		if (vcpu->arch.mp_state != KVM_MP_STATE_RUNNABLE)
 			return -EINTR;
 		return 1;
 	} else {
 		printk(KERN_ERR"kvm: Unsupported userspace halt!");
 		return 0;
 	}
 }
 static int handle_vm_shutdown(struct kvm_vcpu *vcpu,
 		struct kvm_run *kvm_run)
 {
 	kvm_run->exit_reason = KVM_EXIT_SHUTDOWN;
 	return 0;
 }
 static int handle_external_interrupt(struct kvm_vcpu *vcpu,
 		struct kvm_run *kvm_run)
 {
 	return 1;
 }
 static int handle_vcpu_debug(struct kvm_vcpu *vcpu,
 				struct kvm_run *kvm_run)
 {
 	printk("VMM: %s", vcpu->arch.log_buf);
 	return 1;
 }
 static int (*kvm_vti_exit_handlers[])(struct kvm_vcpu *vcpu,
 		struct kvm_run *kvm_run) = {
 	[EXIT_REASON_VM_PANIC]              = handle_vm_error,
 	[EXIT_REASON_MMIO_INSTRUCTION]      = handle_mmio,
 	[EXIT_REASON_PAL_CALL]              = handle_pal_call,
 	[EXIT_REASON_SAL_CALL]              = handle_sal_call,
 	[EXIT_REASON_SWITCH_RR6]            = handle_switch_rr6,
 	[EXIT_REASON_VM_DESTROY]            = handle_vm_shutdown,
 	[EXIT_REASON_EXTERNAL_INTERRUPT]    = handle_external_interrupt,
 	[EXIT_REASON_IPI]		    = handle_ipi,
 	[EXIT_REASON_PTC_G]		    = handle_global_purge,
 	[EXIT_REASON_DEBUG]		    = handle_vcpu_debug,
 };
 static const int kvm_vti_max_exit_handlers =
 		sizeof(kvm_vti_exit_handlers)/sizeof(*kvm_vti_exit_handlers);
 static uint32_t kvm_get_exit_reason(struct kvm_vcpu *vcpu)
 {
 	struct exit_ctl_data *p_exit_data;
 	p_exit_data = kvm_get_exit_data(vcpu);
 	return p_exit_data->exit_reason;
 }
 /*
  * The guest has exited.  See if we can fix it or if we need userspace
  * assistance.
  */
 static int kvm_handle_exit(struct kvm_run *kvm_run, struct kvm_vcpu *vcpu)
 {
 	u32 exit_reason = kvm_get_exit_reason(vcpu);
 	vcpu->arch.last_exit = exit_reason;
 	if (exit_reason < kvm_vti_max_exit_handlers
 			&& kvm_vti_exit_handlers[exit_reason])
 		return kvm_vti_exit_handlers[exit_reason](vcpu, kvm_run);
 	else {
 		kvm_run->exit_reason = KVM_EXIT_UNKNOWN;
 		kvm_run->hw.hardware_exit_reason = exit_reason;
 	}
 	return 0;
 }
 static inline void vti_set_rr6(unsigned long rr6)
 {
 	ia64_set_rr(RR6, rr6);
 	ia64_srlz_i();
 }
 static int kvm_insert_vmm_mapping(struct kvm_vcpu *vcpu)
 {
 	unsigned long pte;
 	struct kvm *kvm = vcpu->kvm;
 	int r;
 	/*Insert a pair of tr to map vmm*/
 	pte = pte_val(mk_pte_phys(__pa(kvm_vmm_base), PAGE_KERNEL));
 	r = ia64_itr_entry(0x3, KVM_VMM_BASE, pte, KVM_VMM_SHIFT);
 	if (r < 0)
 		goto out;
 	vcpu->arch.vmm_tr_slot = r;
 	/*Insert a pairt of tr to map data of vm*/
 	pte = pte_val(mk_pte_phys(__pa(kvm->arch.vm_base), PAGE_KERNEL));
 	r = ia64_itr_entry(0x3, KVM_VM_DATA_BASE,
 					pte, KVM_VM_DATA_SHIFT);
 	if (r < 0)
 		goto out;
 	vcpu->arch.vm_tr_slot = r;
 #if defined(CONFIG_IA64_SGI_SN2) || defined(CONFIG_IA64_GENERIC)
 	if (kvm->arch.is_sn2) {
 		r = kvm_sn2_setup_mappings(vcpu);
 		if (r < 0)
 			goto out;
 	}
 #endif
 	r = 0;
 out:
 	return r;
 }
 static void kvm_purge_vmm_mapping(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
 	ia64_ptr_entry(0x3, vcpu->arch.vmm_tr_slot);
 	ia64_ptr_entry(0x3, vcpu->arch.vm_tr_slot);
 #if defined(CONFIG_IA64_SGI_SN2) || defined(CONFIG_IA64_GENERIC)
 	if (kvm->arch.is_sn2)
 		ia64_ptr_entry(0x3, vcpu->arch.sn_rtc_tr_slot);
 #endif
 }
 static int kvm_vcpu_pre_transition(struct kvm_vcpu *vcpu)
 {
 	unsigned long psr;
 	int r;
 	int cpu = smp_processor_id();
 	if (vcpu->arch.last_run_cpu != cpu ||
 			per_cpu(last_vcpu, cpu) != vcpu) {
 		per_cpu(last_vcpu, cpu) = vcpu;
 		vcpu->arch.last_run_cpu = cpu;
 		kvm_flush_tlb_all();
 	}
 	vcpu->arch.host_rr6 = ia64_get_rr(RR6);
 	vti_set_rr6(vcpu->arch.vmm_rr);
 	local_irq_save(psr);
 	r = kvm_insert_vmm_mapping(vcpu);
 	local_irq_restore(psr);
 	return r;
 }
 static void kvm_vcpu_post_transition(struct kvm_vcpu *vcpu)
 {
 	kvm_purge_vmm_mapping(vcpu);
 	vti_set_rr6(vcpu->arch.host_rr6);
 }
 static int __vcpu_run(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
 {
 	union context *host_ctx, *guest_ctx;
 	int r, idx;
 	idx = srcu_read_lock(&vcpu->kvm->srcu);
 again:
 	if (signal_pending(current)) {
 		r = -EINTR;
 		kvm_run->exit_reason = KVM_EXIT_INTR;
 		goto out;
 	}
 	preempt_disable();
 	local_irq_disable();
 	/*Get host and guest context with guest address space.*/
 	host_ctx = kvm_get_host_context(vcpu);
 	guest_ctx = kvm_get_guest_context(vcpu);
 	clear_bit(KVM_REQ_KICK, &vcpu->requests);
 	r = kvm_vcpu_pre_transition(vcpu);
 	if (r < 0)
 		goto vcpu_run_fail;
 	srcu_read_unlock(&vcpu->kvm->srcu, idx);
 	kvm_guest_enter();
 	/*
 	 * Transition to the guest
 	 */
 	kvm_vmm_info->tramp_entry(host_ctx, guest_ctx);
 	kvm_vcpu_post_transition(vcpu);
 	vcpu->arch.launched = 1;
 	set_bit(KVM_REQ_KICK, &vcpu->requests);
 	local_irq_enable();
 	/*
 	 * We must have an instruction between local_irq_enable() and
 	 * kvm_guest_exit(), so the timer interrupt isn't delayed by
 	 * the interrupt shadow.  The stat.exits increment will do nicely.
 	 * But we need to prevent reordering, hence this barrier():
 	 */
 	barrier();
 	kvm_guest_exit();
 	preempt_enable();
 	idx = srcu_read_lock(&vcpu->kvm->srcu);
 	r = kvm_handle_exit(kvm_run, vcpu);
 	if (r > 0) {
 		if (!need_resched())
 			goto again;
 	}
 out:
 	srcu_read_unlock(&vcpu->kvm->srcu, idx);
 	if (r > 0) {
 		kvm_resched(vcpu);
 		idx = srcu_read_lock(&vcpu->kvm->srcu);
 		goto again;
 	}
 	return r;
 vcpu_run_fail:
 	local_irq_enable();
 	preempt_enable();
 	kvm_run->exit_reason = KVM_EXIT_FAIL_ENTRY;
 	goto out;
 }
 static void kvm_set_mmio_data(struct kvm_vcpu *vcpu)
 {
 	struct kvm_mmio_req *p = kvm_get_vcpu_ioreq(vcpu);
 	if (!vcpu->mmio_is_write)
 		memcpy(&p->data, vcpu->mmio_data, 8);
 	p->state = STATE_IORESP_READY;
 }
 int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
 {
 	int r;
 	sigset_t sigsaved;
 	if (vcpu->sigset_active)
 		sigprocmask(SIG_SETMASK, &vcpu->sigset, &sigsaved);
 	if (unlikely(vcpu->arch.mp_state == KVM_MP_STATE_UNINITIALIZED)) {
 		kvm_vcpu_block(vcpu);
 		clear_bit(KVM_REQ_UNHALT, &vcpu->requests);
 		r = -EAGAIN;
 		goto out;
 	}
 	if (vcpu->mmio_needed) {
 		memcpy(vcpu->mmio_data, kvm_run->mmio.data, 8);
 		kvm_set_mmio_data(vcpu);
 		vcpu->mmio_read_completed = 1;
 		vcpu->mmio_needed = 0;
 	}
 	r = __vcpu_run(vcpu, kvm_run);
 out:
 	if (vcpu->sigset_active)
 		sigprocmask(SIG_SETMASK, &sigsaved, NULL);
 	return r;
 }
 static struct kvm *kvm_alloc_kvm(void)
 {
 	struct kvm *kvm;
 	uint64_t  vm_base;
 	BUG_ON(sizeof(struct kvm) > KVM_VM_STRUCT_SIZE);
 	vm_base = __get_free_pages(GFP_KERNEL, get_order(KVM_VM_DATA_SIZE));
 	if (!vm_base)
 		return ERR_PTR(-ENOMEM);
 	memset((void *)vm_base, 0, KVM_VM_DATA_SIZE);
 	kvm = (struct kvm *)(vm_base +
 			offsetof(struct kvm_vm_data, kvm_vm_struct));
 	kvm->arch.vm_base = vm_base;
 	printk(KERN_DEBUG"kvm: vm's data area:0x%lx\n", vm_base);
 	return kvm;
 }
 struct kvm_io_range {
 	unsigned long start;
 	unsigned long size;
 	unsigned long type;
 };
 static const struct kvm_io_range io_ranges[] = {
 	{VGA_IO_START, VGA_IO_SIZE, GPFN_FRAME_BUFFER},
 	{MMIO_START, MMIO_SIZE, GPFN_LOW_MMIO},
 	{LEGACY_IO_START, LEGACY_IO_SIZE, GPFN_LEGACY_IO},
 	{IO_SAPIC_START, IO_SAPIC_SIZE, GPFN_IOSAPIC},
 	{PIB_START, PIB_SIZE, GPFN_PIB},
 };
 static void kvm_build_io_pmt(struct kvm *kvm)
 {
 	unsigned long i, j;
 	/* Mark I/O ranges */
 	for (i = 0; i < (sizeof(io_ranges) / sizeof(struct kvm_io_range));
 							i++) {
 		for (j = io_ranges[i].start;
 				j < io_ranges[i].start + io_ranges[i].size;
 				j += PAGE_SIZE)
 			kvm_set_pmt_entry(kvm, j >> PAGE_SHIFT,
 					io_ranges[i].type, 0);
 	}
 }
 /*Use unused rids to virtualize guest rid.*/
 #define GUEST_PHYSICAL_RR0	0x1739
 #define GUEST_PHYSICAL_RR4	0x2739
 #define VMM_INIT_RR		0x1660
 static void kvm_init_vm(struct kvm *kvm)
 {
 	BUG_ON(!kvm);
 	kvm->arch.metaphysical_rr0 = GUEST_PHYSICAL_RR0;
 	kvm->arch.metaphysical_rr4 = GUEST_PHYSICAL_RR4;
 	kvm->arch.vmm_init_rr = VMM_INIT_RR;
 	/*
 	 *Fill P2M entries for MMIO/IO ranges
 	 */
 	kvm_build_io_pmt(kvm);
 	INIT_LIST_HEAD(&kvm->arch.assigned_dev_head);
 	/* Reserve bit 0 of irq_sources_bitmap for userspace irq source */
 	set_bit(KVM_USERSPACE_IRQ_SOURCE_ID, &kvm->arch.irq_sources_bitmap);
 }
 struct  kvm *kvm_arch_create_vm(void)
 {
 	struct kvm *kvm = kvm_alloc_kvm();
 	if (IS_ERR(kvm))
 		return ERR_PTR(-ENOMEM);
 	kvm->arch.is_sn2 = ia64_platform_is("sn2");
 	kvm_init_vm(kvm);
 	return kvm;
 }
 static int kvm_vm_ioctl_get_irqchip(struct kvm *kvm,
 					struct kvm_irqchip *chip)
 {
 	int r;
 	r = 0;
 	switch (chip->chip_id) {
 	case KVM_IRQCHIP_IOAPIC:
 		r = kvm_get_ioapic(kvm, &chip->chip.ioapic);
 		break;
 	default:
 		r = -EINVAL;
 		break;
 	}
 	return r;
 }
 static int kvm_vm_ioctl_set_irqchip(struct kvm *kvm, struct kvm_irqchip *chip)
 {
 	int r;
 	r = 0;
 	switch (chip->chip_id) {
 	case KVM_IRQCHIP_IOAPIC:
 		r = kvm_set_ioapic(kvm, &chip->chip.ioapic);
 		break;
 	default:
 		r = -EINVAL;
 		break;
 	}
 	return r;
 }
 #define RESTORE_REGS(_x) vcpu->arch._x = regs->_x
 int kvm_arch_vcpu_ioctl_set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs)
 {
 	struct vpd *vpd = to_host(vcpu->kvm, vcpu->arch.vpd);
 	int i;
 	for (i = 0; i < 16; i++) {
 		vpd->vgr[i] = regs->vpd.vgr[i];
 		vpd->vbgr[i] = regs->vpd.vbgr[i];
 	}
 	for (i = 0; i < 128; i++)
 		vpd->vcr[i] = regs->vpd.vcr[i];
 	vpd->vhpi = regs->vpd.vhpi;
 	vpd->vnat = regs->vpd.vnat;
 	vpd->vbnat = regs->vpd.vbnat;
 	vpd->vpsr = regs->vpd.vpsr;
 	vpd->vpr = regs->vpd.vpr;
 	memcpy(&vcpu->arch.guest, &regs->saved_guest, sizeof(union context));
 	RESTORE_REGS(mp_state);
 	RESTORE_REGS(vmm_rr);
 	memcpy(vcpu->arch.itrs, regs->itrs, sizeof(struct thash_data) * NITRS);
 	memcpy(vcpu->arch.dtrs, regs->dtrs, sizeof(struct thash_data) * NDTRS);
 	RESTORE_REGS(itr_regions);
 	RESTORE_REGS(dtr_regions);
 	RESTORE_REGS(tc_regions);
 	RESTORE_REGS(irq_check);
 	RESTORE_REGS(itc_check);
 	RESTORE_REGS(timer_check);
 	RESTORE_REGS(timer_pending);
 	RESTORE_REGS(last_itc);
 	for (i = 0; i < 8; i++) {
 		vcpu->arch.vrr[i] = regs->vrr[i];
 		vcpu->arch.ibr[i] = regs->ibr[i];
 		vcpu->arch.dbr[i] = regs->dbr[i];
 	}
 	for (i = 0; i < 4; i++)
 		vcpu->arch.insvc[i] = regs->insvc[i];
 	RESTORE_REGS(xtp);
 	RESTORE_REGS(metaphysical_rr0);
 	RESTORE_REGS(metaphysical_rr4);
 	RESTORE_REGS(metaphysical_saved_rr0);
 	RESTORE_REGS(metaphysical_saved_rr4);
 	RESTORE_REGS(fp_psr);
 	RESTORE_REGS(saved_gp);
 	vcpu->arch.irq_new_pending = 1;
 	vcpu->arch.itc_offset = regs->saved_itc - kvm_get_itc(vcpu);
 	set_bit(KVM_REQ_RESUME, &vcpu->requests);
 	return 0;
 }
 long kvm_arch_vm_ioctl(struct file *filp,
 		unsigned int ioctl, unsigned long arg)
 {
 	struct kvm *kvm = filp->private_data;
 	void __user *argp = (void __user *)arg;
 	int r = -ENOTTY;
 	switch (ioctl) {
 	case KVM_SET_MEMORY_REGION: {
 		struct kvm_memory_region kvm_mem;
 		struct kvm_userspace_memory_region kvm_userspace_mem;
 		r = -EFAULT;
 		if (copy_from_user(&kvm_mem, argp, sizeof kvm_mem))
 			goto out;
 		kvm_userspace_mem.slot = kvm_mem.slot;
 		kvm_userspace_mem.flags = kvm_mem.flags;
 		kvm_userspace_mem.guest_phys_addr =
 					kvm_mem.guest_phys_addr;
 		kvm_userspace_mem.memory_size = kvm_mem.memory_size;
 		r = kvm_vm_ioctl_set_memory_region(kvm,
 					&kvm_userspace_mem, 0);
 		if (r)
 			goto out;
 		break;
 		}
 	case KVM_CREATE_IRQCHIP:
 		r = -EFAULT;
 		r = kvm_ioapic_init(kvm);
 		if (r)
 			goto out;
 		r = kvm_setup_default_irq_routing(kvm);
 		if (r) {
 			kvm_ioapic_destroy(kvm);
 			goto out;
 		}
 		break;
 	case KVM_IRQ_LINE_STATUS:
 	case KVM_IRQ_LINE: {
 		struct kvm_irq_level irq_event;
 		r = -EFAULT;
 		if (copy_from_user(&irq_event, argp, sizeof irq_event))
 			goto out;
 		r = -ENXIO;
 		if (irqchip_in_kernel(kvm)) {
 			__s32 status;
 			status = kvm_set_irq(kvm, KVM_USERSPACE_IRQ_SOURCE_ID,
 				    irq_event.irq, irq_event.level);
 			if (ioctl == KVM_IRQ_LINE_STATUS) {
 				r = -EFAULT;
 				irq_event.status = status;
 				if (copy_to_user(argp, &irq_event,
 							sizeof irq_event))
 					goto out;
 			}
 			r = 0;
 		}
 		break;
 		}
 	case KVM_GET_IRQCHIP: {
 		/* 0: PIC master, 1: PIC slave, 2: IOAPIC */
 		struct kvm_irqchip chip;
 		r = -EFAULT;
 		if (copy_from_user(&chip, argp, sizeof chip))
 				goto out;
 		r = -ENXIO;
 		if (!irqchip_in_kernel(kvm))
 			goto out;
 		r = kvm_vm_ioctl_get_irqchip(kvm, &chip);
 		if (r)
 			goto out;
 		r = -EFAULT;
 		if (copy_to_user(argp, &chip, sizeof chip))
 				goto out;
 		r = 0;
 		break;
 		}
 	case KVM_SET_IRQCHIP: {
 		/* 0: PIC master, 1: PIC slave, 2: IOAPIC */
 		struct kvm_irqchip chip;
 		r = -EFAULT;
 		if (copy_from_user(&chip, argp, sizeof chip))
 				goto out;
 		r = -ENXIO;
 		if (!irqchip_in_kernel(kvm))
 			goto out;
 		r = kvm_vm_ioctl_set_irqchip(kvm, &chip);
 		if (r)
 			goto out;
 		r = 0;
 		break;
 		}
 	default:
 		;
 	}
 out:
 	return r;
 }
 int kvm_arch_vcpu_ioctl_set_sregs(struct kvm_vcpu *vcpu,
 		struct kvm_sregs *sregs)
 {
 	return -EINVAL;
 }
 int kvm_arch_vcpu_ioctl_get_sregs(struct kvm_vcpu *vcpu,
 		struct kvm_sregs *sregs)
 {
 	return -EINVAL;
 }
 int kvm_arch_vcpu_ioctl_translate(struct kvm_vcpu *vcpu,
 		struct kvm_translation *tr)
 {
 	return -EINVAL;
 }
 static int kvm_alloc_vmm_area(void)
 {
 	if (!kvm_vmm_base && (kvm_vm_buffer_size < KVM_VM_BUFFER_SIZE)) {
 		kvm_vmm_base = __get_free_pages(GFP_KERNEL,
 				get_order(KVM_VMM_SIZE));
 		if (!kvm_vmm_base)
 			return -ENOMEM;
 		memset((void *)kvm_vmm_base, 0, KVM_VMM_SIZE);
 		kvm_vm_buffer = kvm_vmm_base + VMM_SIZE;
 		printk(KERN_DEBUG"kvm:VMM's Base Addr:0x%lx, vm_buffer:0x%lx\n",
 				kvm_vmm_base, kvm_vm_buffer);
 	}
 	return 0;
 }
 static void kvm_free_vmm_area(void)
 {
 	if (kvm_vmm_base) {
 		/*Zero this area before free to avoid bits leak!!*/
 		memset((void *)kvm_vmm_base, 0, KVM_VMM_SIZE);
 		free_pages(kvm_vmm_base, get_order(KVM_VMM_SIZE));
 		kvm_vmm_base  = 0;
 		kvm_vm_buffer = 0;
 		kvm_vsa_base = 0;
 	}
 }
 static int vti_init_vpd(struct kvm_vcpu *vcpu)
 {
 	int i;
 	union cpuid3_t cpuid3;
 	struct vpd *vpd = to_host(vcpu->kvm, vcpu->arch.vpd);
 	if (IS_ERR(vpd))
 		return PTR_ERR(vpd);
 	/* CPUID init */
 	for (i = 0; i < 5; i++)
 		vpd->vcpuid[i] = ia64_get_cpuid(i);
 	/* Limit the CPUID number to 5 */
 	cpuid3.value = vpd->vcpuid[3];
 	cpuid3.number = 4;	/* 5 - 1 */
 	vpd->vcpuid[3] = cpuid3.value;
 	/*Set vac and vdc fields*/
 	vpd->vac.a_from_int_cr = 1;
 	vpd->vac.a_to_int_cr = 1;
 	vpd->vac.a_from_psr = 1;
 	vpd->vac.a_from_cpuid = 1;
 	vpd->vac.a_cover = 1;
 	vpd->vac.a_bsw = 1;
 	vpd->vac.a_int = 1;
 	vpd->vdc.d_vmsw = 1;
 	/*Set virtual buffer*/
 	vpd->virt_env_vaddr = KVM_VM_BUFFER_BASE;
 	return 0;
 }
 static int vti_create_vp(struct kvm_vcpu *vcpu)
 {
 	long ret;
 	struct vpd *vpd = vcpu->arch.vpd;
 	unsigned long  vmm_ivt;
 	vmm_ivt = kvm_vmm_info->vmm_ivt;
 	printk(KERN_DEBUG "kvm: vcpu:%p,ivt: 0x%lx\n", vcpu, vmm_ivt);
 	ret = ia64_pal_vp_create((u64 *)vpd, (u64 *)vmm_ivt, 0);
 	if (ret) {
 		printk(KERN_ERR"kvm: ia64_pal_vp_create failed!\n");
 		return -EINVAL;
 	}
 	return 0;
 }
 static void init_ptce_info(struct kvm_vcpu *vcpu)
 {
 	ia64_ptce_info_t ptce = {0};
 	ia64_get_ptce(&ptce);
 	vcpu->arch.ptce_base = ptce.base;
 	vcpu->arch.ptce_count[0] = ptce.count[0];
 	vcpu->arch.ptce_count[1] = ptce.count[1];
 	vcpu->arch.ptce_stride[0] = ptce.stride[0];
 	vcpu->arch.ptce_stride[1] = ptce.stride[1];
 }
 static void kvm_migrate_hlt_timer(struct kvm_vcpu *vcpu)
 {
 	struct hrtimer *p_ht = &vcpu->arch.hlt_timer;
 	if (hrtimer_cancel(p_ht))
 		hrtimer_start_expires(p_ht, HRTIMER_MODE_ABS);
 }
 static enum hrtimer_restart hlt_timer_fn(struct hrtimer *data)
 {
 	struct kvm_vcpu *vcpu;
 	wait_queue_head_t *q;
 	vcpu  = container_of(data, struct kvm_vcpu, arch.hlt_timer);
 	q = &vcpu->wq;
 	if (vcpu->arch.mp_state != KVM_MP_STATE_HALTED)
 		goto out;
 	if (waitqueue_active(q))
 		wake_up_interruptible(q);
 out:
 	vcpu->arch.timer_fired = 1;
 	vcpu->arch.timer_check = 1;
 	return HRTIMER_NORESTART;
 }
 #define PALE_RESET_ENTRY    0x80000000ffffffb0UL
 int kvm_arch_vcpu_init(struct kvm_vcpu *vcpu)
 {
 	struct kvm_vcpu *v;
 	int r;
 	int i;
 	long itc_offset;
 	struct kvm *kvm = vcpu->kvm;
 	struct kvm_pt_regs *regs = vcpu_regs(vcpu);
 	union context *p_ctx = &vcpu->arch.guest;
 	struct kvm_vcpu *vmm_vcpu = to_guest(vcpu->kvm, vcpu);
 	/*Init vcpu context for first run.*/
 	if (IS_ERR(vmm_vcpu))
 		return PTR_ERR(vmm_vcpu);
 	if (kvm_vcpu_is_bsp(vcpu)) {
 		vcpu->arch.mp_state = KVM_MP_STATE_RUNNABLE;
 		/*Set entry address for first run.*/
 		regs->cr_iip = PALE_RESET_ENTRY;
 		/*Initialize itc offset for vcpus*/
 		itc_offset = 0UL - kvm_get_itc(vcpu);
 		for (i = 0; i < KVM_MAX_VCPUS; i++) {
 			v = (struct kvm_vcpu *)((char *)vcpu +
 					sizeof(struct kvm_vcpu_data) * i);
 			v->arch.itc_offset = itc_offset;
 			v->arch.last_itc = 0;
 		}
 	} else
 		vcpu->arch.mp_state = KVM_MP_STATE_UNINITIALIZED;
 	r = -ENOMEM;
 	vcpu->arch.apic = kzalloc(sizeof(struct kvm_lapic), GFP_KERNEL);
 	if (!vcpu->arch.apic)
 		goto out;
 	vcpu->arch.apic->vcpu = vcpu;
 	p_ctx->gr[1] = 0;
 	p_ctx->gr[12] = (unsigned long)((char *)vmm_vcpu + KVM_STK_OFFSET);
 	p_ctx->gr[13] = (unsigned long)vmm_vcpu;
 	p_ctx->psr = 0x1008522000UL;
 	p_ctx->ar[40] = FPSR_DEFAULT; /*fpsr*/
 	p_ctx->caller_unat = 0;
 	p_ctx->pr = 0x0;
 	p_ctx->ar[36] = 0x0; /*unat*/
 	p_ctx->ar[19] = 0x0; /*rnat*/
 	p_ctx->ar[18] = (unsigned long)vmm_vcpu +
 				((sizeof(struct kvm_vcpu)+15) & ~15);
 	p_ctx->ar[64] = 0x0; /*pfs*/
 	p_ctx->cr[0] = 0x7e04UL;
 	p_ctx->cr[2] = (unsigned long)kvm_vmm_info->vmm_ivt;
 	p_ctx->cr[8] = 0x3c;
 	/*Initilize region register*/
 	p_ctx->rr[0] = 0x30;
 	p_ctx->rr[1] = 0x30;
 	p_ctx->rr[2] = 0x30;
 	p_ctx->rr[3] = 0x30;
 	p_ctx->rr[4] = 0x30;
 	p_ctx->rr[5] = 0x30;
 	p_ctx->rr[7] = 0x30;
 	/*Initilize branch register 0*/
 	p_ctx->br[0] = *(unsigned long *)kvm_vmm_info->vmm_entry;
 	vcpu->arch.vmm_rr = kvm->arch.vmm_init_rr;
 	vcpu->arch.metaphysical_rr0 = kvm->arch.metaphysical_rr0;
 	vcpu->arch.metaphysical_rr4 = kvm->arch.metaphysical_rr4;
 	hrtimer_init(&vcpu->arch.hlt_timer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS);
 	vcpu->arch.hlt_timer.function = hlt_timer_fn;
 	vcpu->arch.last_run_cpu = -1;
 	vcpu->arch.vpd = (struct vpd *)VPD_BASE(vcpu->vcpu_id);
 	vcpu->arch.vsa_base = kvm_vsa_base;
 	vcpu->arch.__gp = kvm_vmm_gp;
 	vcpu->arch.dirty_log_lock_pa = __pa(&kvm->arch.dirty_log_lock);
 	vcpu->arch.vhpt.hash = (struct thash_data *)VHPT_BASE(vcpu->vcpu_id);
 	vcpu->arch.vtlb.hash = (struct thash_data *)VTLB_BASE(vcpu->vcpu_id);
 	init_ptce_info(vcpu);
 	r = 0;
 out:
 	return r;
 }
 static int vti_vcpu_setup(struct kvm_vcpu *vcpu, int id)
 {
 	unsigned long psr;
 	int r;
 	local_irq_save(psr);
 	r = kvm_insert_vmm_mapping(vcpu);
 	local_irq_restore(psr);
 	if (r)
 		goto fail;
 	r = kvm_vcpu_init(vcpu, vcpu->kvm, id);
 	if (r)
 		goto fail;
 	r = vti_init_vpd(vcpu);
 	if (r) {
 		printk(KERN_DEBUG"kvm: vpd init error!!\n");
 		goto uninit;
 	}
 	r = vti_create_vp(vcpu);
 	if (r)
 		goto uninit;
 	kvm_purge_vmm_mapping(vcpu);
 	return 0;
 uninit:
 	kvm_vcpu_uninit(vcpu);
 fail:
 	return r;
 }
 struct kvm_vcpu *kvm_arch_vcpu_create(struct kvm *kvm,
 		unsigned int id)
 {
 	struct kvm_vcpu *vcpu;
 	unsigned long vm_base = kvm->arch.vm_base;
 	int r;
 	int cpu;
 	BUG_ON(sizeof(struct kvm_vcpu) > VCPU_STRUCT_SIZE/2);
 	r = -EINVAL;
 	if (id >= KVM_MAX_VCPUS) {
 		printk(KERN_ERR"kvm: Can't configure vcpus > %ld",
 				KVM_MAX_VCPUS);
 		goto fail;
 	}
 	r = -ENOMEM;
 	if (!vm_base) {
 		printk(KERN_ERR"kvm: Create vcpu[%d] error!\n", id);
 		goto fail;
 	}
 	vcpu = (struct kvm_vcpu *)(vm_base + offsetof(struct kvm_vm_data,
 					vcpu_data[id].vcpu_struct));
 	vcpu->kvm = kvm;
 	cpu = get_cpu();
 	r = vti_vcpu_setup(vcpu, id);
 	put_cpu();
 	if (r) {
 		printk(KERN_DEBUG"kvm: vcpu_setup error!!\n");
 		goto fail;
 	}
 	return vcpu;
 fail:
 	return ERR_PTR(r);
 }
 int kvm_arch_vcpu_setup(struct kvm_vcpu *vcpu)
 {
 	return 0;
 }
 int kvm_arch_vcpu_ioctl_get_fpu(struct kvm_vcpu *vcpu, struct kvm_fpu *fpu)
 {
 	return -EINVAL;
 }
 int kvm_arch_vcpu_ioctl_set_fpu(struct kvm_vcpu *vcpu, struct kvm_fpu *fpu)
 {
 	return -EINVAL;
 }
 int kvm_arch_vcpu_ioctl_set_guest_debug(struct kvm_vcpu *vcpu,
 					struct kvm_guest_debug *dbg)
 {
 	return -EINVAL;
 }
 static void free_kvm(struct kvm *kvm)
 {
 	unsigned long vm_base = kvm->arch.vm_base;
 	if (vm_base) {
 		memset((void *)vm_base, 0, KVM_VM_DATA_SIZE);
 		free_pages(vm_base, get_order(KVM_VM_DATA_SIZE));
 	}
 }
 static void kvm_release_vm_pages(struct kvm *kvm)
 {
 	struct kvm_memslots *slots;
 	struct kvm_memory_slot *memslot;
 	int i, j;
 	unsigned long base_gfn;
 	slots = kvm_memslots(kvm);
 	for (i = 0; i < slots->nmemslots; i++) {
 		memslot = &slots->memslots[i];
 		base_gfn = memslot->base_gfn;
 		for (j = 0; j < memslot->npages; j++) {
 			if (memslot->rmap[j])
 				put_page((struct page *)memslot->rmap[j]);
 		}
 	}
 }
 void kvm_arch_sync_events(struct kvm *kvm)
 {
 }
 void kvm_arch_destroy_vm(struct kvm *kvm)
 {
 	kvm_iommu_unmap_guest(kvm);
 #ifdef  KVM_CAP_DEVICE_ASSIGNMENT
 	kvm_free_all_assigned_devices(kvm);
 #endif
 	kfree(kvm->arch.vioapic);
 	kvm_release_vm_pages(kvm);
 	kvm_free_physmem(kvm);
 	cleanup_srcu_struct(&kvm->srcu);
 	free_kvm(kvm);
 }
 void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
 {
 }
 void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
 {
 	if (cpu != vcpu->cpu) {
 		vcpu->cpu = cpu;
 		if (vcpu->arch.ht_active)
 			kvm_migrate_hlt_timer(vcpu);
 	}
 }
 #define SAVE_REGS(_x) 	regs->_x = vcpu->arch._x
 int kvm_arch_vcpu_ioctl_get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs)
 {
 	struct vpd *vpd = to_host(vcpu->kvm, vcpu->arch.vpd);
 	int i;
 	vcpu_load(vcpu);
 	for (i = 0; i < 16; i++) {
 		regs->vpd.vgr[i] = vpd->vgr[i];
 		regs->vpd.vbgr[i] = vpd->vbgr[i];
 	}
 	for (i = 0; i < 128; i++)
 		regs->vpd.vcr[i] = vpd->vcr[i];
 	regs->vpd.vhpi = vpd->vhpi;
 	regs->vpd.vnat = vpd->vnat;
 	regs->vpd.vbnat = vpd->vbnat;
 	regs->vpd.vpsr = vpd->vpsr;
 	regs->vpd.vpr = vpd->vpr;
 	memcpy(&regs->saved_guest, &vcpu->arch.guest, sizeof(union context));
 	SAVE_REGS(mp_state);
 	SAVE_REGS(vmm_rr);
 	memcpy(regs->itrs, vcpu->arch.itrs, sizeof(struct thash_data) * NITRS);
 	memcpy(regs->dtrs, vcpu->arch.dtrs, sizeof(struct thash_data) * NDTRS);
 	SAVE_REGS(itr_regions);
 	SAVE_REGS(dtr_regions);
 	SAVE_REGS(tc_regions);
 	SAVE_REGS(irq_check);
 	SAVE_REGS(itc_check);
 	SAVE_REGS(timer_check);
 	SAVE_REGS(timer_pending);
 	SAVE_REGS(last_itc);
 	for (i = 0; i < 8; i++) {
 		regs->vrr[i] = vcpu->arch.vrr[i];
 		regs->ibr[i] = vcpu->arch.ibr[i];
 		regs->dbr[i] = vcpu->arch.dbr[i];
 	}
 	for (i = 0; i < 4; i++)
 		regs->insvc[i] = vcpu->arch.insvc[i];
 	regs->saved_itc = vcpu->arch.itc_offset + kvm_get_itc(vcpu);
 	SAVE_REGS(xtp);
 	SAVE_REGS(metaphysical_rr0);
 	SAVE_REGS(metaphysical_rr4);
 	SAVE_REGS(metaphysical_saved_rr0);
 	SAVE_REGS(metaphysical_saved_rr4);
 	SAVE_REGS(fp_psr);
 	SAVE_REGS(saved_gp);
 	vcpu_put(vcpu);
 	return 0;
 }
 int kvm_arch_vcpu_ioctl_get_stack(struct kvm_vcpu *vcpu,
 				  struct kvm_ia64_vcpu_stack *stack)
 {
 	memcpy(stack, vcpu, sizeof(struct kvm_ia64_vcpu_stack));
 	return 0;
 }
 int kvm_arch_vcpu_ioctl_set_stack(struct kvm_vcpu *vcpu,
 				  struct kvm_ia64_vcpu_stack *stack)
 {
 	memcpy(vcpu + 1, &stack->stack[0] + sizeof(struct kvm_vcpu),
 	       sizeof(struct kvm_ia64_vcpu_stack) - sizeof(struct kvm_vcpu));
 	vcpu->arch.exit_data = ((struct kvm_vcpu *)stack)->arch.exit_data;
 	return 0;
 }
 void kvm_arch_vcpu_uninit(struct kvm_vcpu *vcpu)
 {
 	hrtimer_cancel(&vcpu->arch.hlt_timer);
 	kfree(vcpu->arch.apic);
 }
 long kvm_arch_vcpu_ioctl(struct file *filp,
 			 unsigned int ioctl, unsigned long arg)
 {
 	struct kvm_vcpu *vcpu = filp->private_data;
 	void __user *argp = (void __user *)arg;
 	struct kvm_ia64_vcpu_stack *stack = NULL;
 	long r;
 	switch (ioctl) {
 	case KVM_IA64_VCPU_GET_STACK: {
 		struct kvm_ia64_vcpu_stack __user *user_stack;
 	        void __user *first_p = argp;
 		r = -EFAULT;
 		if (copy_from_user(&user_stack, first_p, sizeof(void *)))
 			goto out;
 		if (!access_ok(VERIFY_WRITE, user_stack,
 			       sizeof(struct kvm_ia64_vcpu_stack))) {
 			printk(KERN_INFO "KVM_IA64_VCPU_GET_STACK: "
 			       "Illegal user destination address for stack\n");
 			goto out;
 		}
 		stack = kzalloc(sizeof(struct kvm_ia64_vcpu_stack), GFP_KERNEL);
 		if (!stack) {
 			r = -ENOMEM;
 			goto out;
 		}
 		r = kvm_arch_vcpu_ioctl_get_stack(vcpu, stack);
 		if (r)
 			goto out;
 		if (copy_to_user(user_stack, stack,
 				 sizeof(struct kvm_ia64_vcpu_stack))) {
 			r = -EFAULT;
 			goto out;
 		}
 		break;
 	}
 	case KVM_IA64_VCPU_SET_STACK: {
 		struct kvm_ia64_vcpu_stack __user *user_stack;
 	        void __user *first_p = argp;
 		r = -EFAULT;
 		if (copy_from_user(&user_stack, first_p, sizeof(void *)))
 			goto out;
 		if (!access_ok(VERIFY_READ, user_stack,
 			    sizeof(struct kvm_ia64_vcpu_stack))) {
 			printk(KERN_INFO "KVM_IA64_VCPU_SET_STACK: "
 			       "Illegal user address for stack\n");
 			goto out;
 		}
 		stack = kmalloc(sizeof(struct kvm_ia64_vcpu_stack), GFP_KERNEL);
 		if (!stack) {
 			r = -ENOMEM;
 			goto out;
 		}
 		if (copy_from_user(stack, user_stack,
 				   sizeof(struct kvm_ia64_vcpu_stack)))
 			goto out;
 		r = kvm_arch_vcpu_ioctl_set_stack(vcpu, stack);
 		break;
 	}
 	default:
 		r = -EINVAL;
 	}
 out:
 	kfree(stack);
 	return r;
 }
 int kvm_arch_prepare_memory_region(struct kvm *kvm,
 		struct kvm_memory_slot *memslot,
 		struct kvm_memory_slot old,
 		struct kvm_userspace_memory_region *mem,
 		int user_alloc)
 {
 	unsigned long i;
 	unsigned long pfn;
 	int npages = memslot->npages;
 	unsigned long base_gfn = memslot->base_gfn;
 	if (base_gfn + npages > (KVM_MAX_MEM_SIZE >> PAGE_SHIFT))
 		return -ENOMEM;
 	for (i = 0; i < npages; i++) {
 		pfn = gfn_to_pfn(kvm, base_gfn + i);
 		if (!kvm_is_mmio_pfn(pfn)) {
 			kvm_set_pmt_entry(kvm, base_gfn + i,
 					pfn << PAGE_SHIFT,
 				_PAGE_AR_RWX | _PAGE_MA_WB);
 			memslot->rmap[i] = (unsigned long)pfn_to_page(pfn);
 		} else {
 			kvm_set_pmt_entry(kvm, base_gfn + i,
 					GPFN_PHYS_MMIO | (pfn << PAGE_SHIFT),
 					_PAGE_MA_UC);
 			memslot->rmap[i] = 0;
 			}
 	}
 	return 0;
 }
 void kvm_arch_commit_memory_region(struct kvm *kvm,
 		struct kvm_userspace_memory_region *mem,
 		struct kvm_memory_slot old,
 		int user_alloc)
 {
 	return;
 }
 void kvm_arch_flush_shadow(struct kvm *kvm)
 {
 	kvm_flush_remote_tlbs(kvm);
 }
 long kvm_arch_dev_ioctl(struct file *filp,
 			unsigned int ioctl, unsigned long arg)
 {
 	return -EINVAL;
 }
 void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu)
 {
 	kvm_vcpu_uninit(vcpu);
 }
 static int vti_cpu_has_kvm_support(void)
 {
 	long  avail = 1, status = 1, control = 1;
 	long ret;
 	ret = ia64_pal_proc_get_features(&avail, &status, &control, 0);
 	if (ret)
 		goto out;
 	if (!(avail & PAL_PROC_VM_BIT))
 		goto out;
 	printk(KERN_DEBUG"kvm: Hardware Supports VT\n");
 	ret = ia64_pal_vp_env_info(&kvm_vm_buffer_size, &vp_env_info);
 	if (ret)
 		goto out;
 	printk(KERN_DEBUG"kvm: VM Buffer Size:0x%lx\n", kvm_vm_buffer_size);
 	if (!(vp_env_info & VP_OPCODE)) {
 		printk(KERN_WARNING"kvm: No opcode ability on hardware, "
 				"vm_env_info:0x%lx\n", vp_env_info);
 	}
 	return 1;
 out:
 	return 0;
 }
 /*
  * On SN2, the ITC isn't stable, so copy in fast path code to use the
  * SN2 RTC, replacing the ITC based default verion.
  */
 static void kvm_patch_vmm(struct kvm_vmm_info *vmm_info,
 			  struct module *module)
 {
 	unsigned long new_ar, new_ar_sn2;
 	unsigned long module_base;
 	if (!ia64_platform_is("sn2"))
 		return;
 	module_base = (unsigned long)module->module_core;
 	new_ar = kvm_vmm_base + vmm_info->patch_mov_ar - module_base;
 	new_ar_sn2 = kvm_vmm_base + vmm_info->patch_mov_ar_sn2 - module_base;
 	printk(KERN_INFO "kvm: Patching ITC emulation to use SGI SN2 RTC "
 	       "as source\n");
 	/*
 	 * Copy the SN2 version of mov_ar into place. They are both
 	 * the same size, so 6 bundles is sufficient (6 * 0x10).
 	 */
 	memcpy((void *)new_ar, (void *)new_ar_sn2, 0x60);
 }
 static int kvm_relocate_vmm(struct kvm_vmm_info *vmm_info,
 			    struct module *module)
 {
 	unsigned long module_base;
 	unsigned long vmm_size;
 	unsigned long vmm_offset, func_offset, fdesc_offset;
 	struct fdesc *p_fdesc;
 	BUG_ON(!module);
 	if (!kvm_vmm_base) {
 		printk("kvm: kvm area hasn't been initilized yet!!\n");
 		return -EFAULT;
 	}
 	/*Calculate new position of relocated vmm module.*/
 	module_base = (unsigned long)module->module_core;
 	vmm_size = module->core_size;
 	if (unlikely(vmm_size > KVM_VMM_SIZE))
 		return -EFAULT;
 	memcpy((void *)kvm_vmm_base, (void *)module_base, vmm_size);
 	kvm_patch_vmm(vmm_info, module);
 	kvm_flush_icache(kvm_vmm_base, vmm_size);
 	/*Recalculate kvm_vmm_info based on new VMM*/
 	vmm_offset = vmm_info->vmm_ivt - module_base;
 	kvm_vmm_info->vmm_ivt = KVM_VMM_BASE + vmm_offset;
 	printk(KERN_DEBUG"kvm: Relocated VMM's IVT Base Addr:%lx\n",
 			kvm_vmm_info->vmm_ivt);
 	fdesc_offset = (unsigned long)vmm_info->vmm_entry - module_base;
 	kvm_vmm_info->vmm_entry = (kvm_vmm_entry *)(KVM_VMM_BASE +
 							fdesc_offset);
 	func_offset = *(unsigned long *)vmm_info->vmm_entry - module_base;
 	p_fdesc = (struct fdesc *)(kvm_vmm_base + fdesc_offset);
 	p_fdesc->ip = KVM_VMM_BASE + func_offset;
 	p_fdesc->gp = KVM_VMM_BASE+(p_fdesc->gp - module_base);
 	printk(KERN_DEBUG"kvm: Relocated VMM's Init Entry Addr:%lx\n",
 			KVM_VMM_BASE+func_offset);
 	fdesc_offset = (unsigned long)vmm_info->tramp_entry - module_base;
 	kvm_vmm_info->tramp_entry = (kvm_tramp_entry *)(KVM_VMM_BASE +
 			fdesc_offset);
 	func_offset = *(unsigned long *)vmm_info->tramp_entry - module_base;
 	p_fdesc = (struct fdesc *)(kvm_vmm_base + fdesc_offset);
 	p_fdesc->ip = KVM_VMM_BASE + func_offset;
 	p_fdesc->gp = KVM_VMM_BASE + (p_fdesc->gp - module_base);
 	kvm_vmm_gp = p_fdesc->gp;
 	printk(KERN_DEBUG"kvm: Relocated VMM's Entry IP:%p\n",
 						kvm_vmm_info->vmm_entry);
 	printk(KERN_DEBUG"kvm: Relocated VMM's Trampoline Entry IP:0x%lx\n",
 						KVM_VMM_BASE + func_offset);
 	return 0;
 }
 int kvm_arch_init(void *opaque)
 {
 	int r;
 	struct kvm_vmm_info *vmm_info = (struct kvm_vmm_info *)opaque;
 	if (!vti_cpu_has_kvm_support()) {
 		printk(KERN_ERR "kvm: No Hardware Virtualization Support!\n");
 		r = -EOPNOTSUPP;
 		goto out;
 	}
 	if (kvm_vmm_info) {
 		printk(KERN_ERR "kvm: Already loaded VMM module!\n");
 		r = -EEXIST;
 		goto out;
 	}
 	r = -ENOMEM;
 	kvm_vmm_info = kzalloc(sizeof(struct kvm_vmm_info), GFP_KERNEL);
 	if (!kvm_vmm_info)
 		goto out;
 	if (kvm_alloc_vmm_area())
 		goto out_free0;
 	r = kvm_relocate_vmm(vmm_info, vmm_info->module);
 	if (r)
 		goto out_free1;
 	return 0;
 out_free1:
 	kvm_free_vmm_area();
 out_free0:
 	kfree(kvm_vmm_info);
 out:
 	return r;
 }
 void kvm_arch_exit(void)
 {
 	kvm_free_vmm_area();
 	kfree(kvm_vmm_info);
 	kvm_vmm_info = NULL;
 }
 static int kvm_ia64_sync_dirty_log(struct kvm *kvm,
 		struct kvm_dirty_log *log)
 {
 	struct kvm_memory_slot *memslot;
 	int r, i;
 	long base;
 	unsigned long n;
 	unsigned long *dirty_bitmap = (unsigned long *)(kvm->arch.vm_base +
 			offsetof(struct kvm_vm_data, kvm_mem_dirty_log));
 	r = -EINVAL;
 	if (log->slot >= KVM_MEMORY_SLOTS)
 		goto out;
 	memslot = &kvm->memslots->memslots[log->slot];
 	r = -ENOENT;
 	if (!memslot->dirty_bitmap)
 		goto out;
 	n = kvm_dirty_bitmap_bytes(memslot);
 	base = memslot->base_gfn / BITS_PER_LONG;
 	for (i = 0; i < n/sizeof(long); ++i) {
 		memslot->dirty_bitmap[i] = dirty_bitmap[base + i];
 		dirty_bitmap[base + i] = 0;
 	}
 	r = 0;
 out:
 	return r;
 }
 int kvm_vm_ioctl_get_dirty_log(struct kvm *kvm,
 		struct kvm_dirty_log *log)
 {
 	int r;
 	unsigned long n;
 	struct kvm_memory_slot *memslot;
 	int is_dirty = 0;
 	mutex_lock(&kvm->slots_lock);
 	spin_lock(&kvm->arch.dirty_log_lock);
 	r = kvm_ia64_sync_dirty_log(kvm, log);
 	if (r)
 		goto out;
 	r = kvm_get_dirty_log(kvm, log, &is_dirty);
 	if (r)
 		goto out;
 	/* If nothing is dirty, don't bother messing with page tables. */
 	if (is_dirty) {
 		kvm_flush_remote_tlbs(kvm);
 		memslot = &kvm->memslots->memslots[log->slot];
 		n = kvm_dirty_bitmap_bytes(memslot);
 		memset(memslot->dirty_bitmap, 0, n);
 	}
 	r = 0;
 out:
 	mutex_unlock(&kvm->slots_lock);
 	spin_unlock(&kvm->arch.dirty_log_lock);
 	return r;
 }
 int kvm_arch_hardware_setup(void)
 {
 	return 0;
 }
 void kvm_arch_hardware_unsetup(void)
 {
 }
 void kvm_vcpu_kick(struct kvm_vcpu *vcpu)
 {
 	int me;
 	int cpu = vcpu->cpu;
 	if (waitqueue_active(&vcpu->wq))
 		wake_up_interruptible(&vcpu->wq);
 	me = get_cpu();
 	if (cpu != me && (unsigned) cpu < nr_cpu_ids && cpu_online(cpu))
 		if (!test_and_set_bit(KVM_REQ_KICK, &vcpu->requests))
 			smp_send_reschedule(cpu);
 	put_cpu();
 }
 int kvm_apic_set_irq(struct kvm_vcpu *vcpu, struct kvm_lapic_irq *irq)
 {
 	return __apic_accept_irq(vcpu, irq->vector);
 }
 int kvm_apic_match_physical_addr(struct kvm_lapic *apic, u16 dest)
 {
 	return apic->vcpu->vcpu_id == dest;
 }
 int kvm_apic_match_logical_addr(struct kvm_lapic *apic, u8 mda)
 {
 	return 0;
 }
 int kvm_apic_compare_prio(struct kvm_vcpu *vcpu1, struct kvm_vcpu *vcpu2)
 {
 	return vcpu1->arch.xtp - vcpu2->arch.xtp;
 }
 int kvm_apic_match_dest(struct kvm_vcpu *vcpu, struct kvm_lapic *source,
 		int short_hand, int dest, int dest_mode)
 {
 	struct kvm_lapic *target = vcpu->arch.apic;
 	return (dest_mode == 0) ?
 		kvm_apic_match_physical_addr(target, dest) :
 		kvm_apic_match_logical_addr(target, dest);
 }
 static int find_highest_bits(int *dat)
 {
 	u32  bits, bitnum;
 	int i;
 	/* loop for all 256 bits */
 	for (i = 7; i >= 0 ; i--) {
 		bits = dat[i];
 		if (bits) {
 			bitnum = fls(bits);
 			return i * 32 + bitnum - 1;
 		}
 	}
 	return -1;
 }
 int kvm_highest_pending_irq(struct kvm_vcpu *vcpu)
 {
     struct vpd *vpd = to_host(vcpu->kvm, vcpu->arch.vpd);
     if (vpd->irr[0] & (1UL << NMI_VECTOR))
 		return NMI_VECTOR;
     if (vpd->irr[0] & (1UL << ExtINT_VECTOR))
 		return ExtINT_VECTOR;
     return find_highest_bits((int *)&vpd->irr[0]);
 }
 int kvm_cpu_has_pending_timer(struct kvm_vcpu *vcpu)
 {
 	return vcpu->arch.timer_fired;
 }
-gfn_t unalias_gfn(struct kvm *kvm, gfn_t gfn)
-{
-	return gfn;
-}
 int kvm_arch_vcpu_runnable(struct kvm_vcpu *vcpu)
 {
 	return (vcpu->arch.mp_state == KVM_MP_STATE_RUNNABLE) ||
 		(kvm_highest_pending_irq(vcpu) != -1);
 }
 int kvm_arch_vcpu_ioctl_get_mpstate(struct kvm_vcpu *vcpu,
 				    struct kvm_mp_state *mp_state)
 {
 	mp_state->mp_state = vcpu->arch.mp_state;
 	return 0;
 }
 static int vcpu_reset(struct kvm_vcpu *vcpu)
 {
 	int r;
 	long psr;
 	local_irq_save(psr);
 	r = kvm_insert_vmm_mapping(vcpu);
 	local_irq_restore(psr);
 	if (r)
 		goto fail;
 	vcpu->arch.launched = 0;
 	kvm_arch_vcpu_uninit(vcpu);
 	r = kvm_arch_vcpu_init(vcpu);
 	if (r)
 		goto fail;
 	kvm_purge_vmm_mapping(vcpu);
 	r = 0;
 fail:
 	return r;
 }
 int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu,
 				    struct kvm_mp_state *mp_state)
 {
 	int r = 0;
 	vcpu->arch.mp_state = mp_state->mp_state;
 	if (vcpu->arch.mp_state == KVM_MP_STATE_UNINITIALIZED)
 		r = vcpu_reset(vcpu);
 	return r;
 }

arch/powerpc/kvm/powerpc.c

Diff comments View file @ a1f4d39

 /*
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU General Public License, version 2, as
  * published by the Free Software Foundation.
  *
  * This program is distributed in the hope that it will be useful,
  * but WITHOUT ANY WARRANTY; without even the implied warranty of
  * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  * GNU General Public License for more details.
  *
  * You should have received a copy of the GNU General Public License
  * along with this program; if not, write to the Free Software
  * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
  *
  * Copyright IBM Corp. 2007
  *
  * Authors: Hollis Blanchard <hollisb@us.ibm.com>
  *          Christian Ehrhardt <ehrhardt@linux.vnet.ibm.com>
  */
 #include <linux/errno.h>
 #include <linux/err.h>
 #include <linux/kvm_host.h>
 #include <linux/module.h>
 #include <linux/vmalloc.h>
 #include <linux/hrtimer.h>
 #include <linux/fs.h>
 #include <linux/slab.h>
 #include <asm/cputable.h>
 #include <asm/uaccess.h>
 #include <asm/kvm_ppc.h>
 #include <asm/tlbflush.h>
 #include "timing.h"
 #include "../mm/mmu_decl.h"
 #define CREATE_TRACE_POINTS
 #include "trace.h"
-gfn_t unalias_gfn(struct kvm *kvm, gfn_t gfn)
-{
-	return gfn;
-}
 int kvm_arch_vcpu_runnable(struct kvm_vcpu *v)
 {
 	return !(v->arch.msr & MSR_WE) || !!(v->arch.pending_exceptions);
 }
 int kvmppc_emulate_mmio(struct kvm_run *run, struct kvm_vcpu *vcpu)
 {
 	enum emulation_result er;
 	int r;
 	er = kvmppc_emulate_instruction(run, vcpu);
 	switch (er) {
 	case EMULATE_DONE:
 		/* Future optimization: only reload non-volatiles if they were
 		 * actually modified. */
 		r = RESUME_GUEST_NV;
 		break;
 	case EMULATE_DO_MMIO:
 		run->exit_reason = KVM_EXIT_MMIO;
 		/* We must reload nonvolatiles because "update" load/store
 		 * instructions modify register state. */
 		/* Future optimization: only reload non-volatiles if they were
 		 * actually modified. */
 		r = RESUME_HOST_NV;
 		break;
 	case EMULATE_FAIL:
 		/* XXX Deliver Program interrupt to guest. */
 		printk(KERN_EMERG "%s: emulation failed (%08x)\n", __func__,
 		       kvmppc_get_last_inst(vcpu));
 		r = RESUME_HOST;
 		break;
 	default:
 		BUG();
 	}
 	return r;
 }
 int kvm_arch_hardware_enable(void *garbage)
 {
 	return 0;
 }
 void kvm_arch_hardware_disable(void *garbage)
 {
 }
 int kvm_arch_hardware_setup(void)
 {
 	return 0;
 }
 void kvm_arch_hardware_unsetup(void)
 {
 }
 void kvm_arch_check_processor_compat(void *rtn)
 {
 	*(int *)rtn = kvmppc_core_check_processor_compat();
 }
 struct kvm *kvm_arch_create_vm(void)
 {
 	struct kvm *kvm;
 	kvm = kzalloc(sizeof(struct kvm), GFP_KERNEL);
 	if (!kvm)
 		return ERR_PTR(-ENOMEM);
 	return kvm;
 }
 static void kvmppc_free_vcpus(struct kvm *kvm)
 {
 	unsigned int i;
 	struct kvm_vcpu *vcpu;
 	kvm_for_each_vcpu(i, vcpu, kvm)
 		kvm_arch_vcpu_free(vcpu);
 	mutex_lock(&kvm->lock);
 	for (i = 0; i < atomic_read(&kvm->online_vcpus); i++)
 		kvm->vcpus[i] = NULL;
 	atomic_set(&kvm->online_vcpus, 0);
 	mutex_unlock(&kvm->lock);
 }
 void kvm_arch_sync_events(struct kvm *kvm)
 {
 }
 void kvm_arch_destroy_vm(struct kvm *kvm)
 {
 	kvmppc_free_vcpus(kvm);
 	kvm_free_physmem(kvm);
 	cleanup_srcu_struct(&kvm->srcu);
 	kfree(kvm);
 }
 int kvm_dev_ioctl_check_extension(long ext)
 {
 	int r;
 	switch (ext) {
 	case KVM_CAP_PPC_SEGSTATE:
 	case KVM_CAP_PPC_PAIRED_SINGLES:
 	case KVM_CAP_PPC_UNSET_IRQ:
 	case KVM_CAP_ENABLE_CAP:
 	case KVM_CAP_PPC_OSI:
 		r = 1;
 		break;
 	case KVM_CAP_COALESCED_MMIO:
 		r = KVM_COALESCED_MMIO_PAGE_OFFSET;
 		break;
 	default:
 		r = 0;
 		break;
 	}
 	return r;
 }
 long kvm_arch_dev_ioctl(struct file *filp,
                         unsigned int ioctl, unsigned long arg)
 {
 	return -EINVAL;
 }
 int kvm_arch_prepare_memory_region(struct kvm *kvm,
                                    struct kvm_memory_slot *memslot,
                                    struct kvm_memory_slot old,
                                    struct kvm_userspace_memory_region *mem,
                                    int user_alloc)
 {
 	return 0;
 }
 void kvm_arch_commit_memory_region(struct kvm *kvm,
                struct kvm_userspace_memory_region *mem,
                struct kvm_memory_slot old,
                int user_alloc)
 {
        return;
 }
 void kvm_arch_flush_shadow(struct kvm *kvm)
 {
 }
 struct kvm_vcpu *kvm_arch_vcpu_create(struct kvm *kvm, unsigned int id)
 {
 	struct kvm_vcpu *vcpu;
 	vcpu = kvmppc_core_vcpu_create(kvm, id);
 	if (!IS_ERR(vcpu))
 		kvmppc_create_vcpu_debugfs(vcpu, id);
 	return vcpu;
 }
 void kvm_arch_vcpu_free(struct kvm_vcpu *vcpu)
 {
 	/* Make sure we're not using the vcpu anymore */
 	hrtimer_cancel(&vcpu->arch.dec_timer);
 	tasklet_kill(&vcpu->arch.tasklet);
 	kvmppc_remove_vcpu_debugfs(vcpu);
 	kvmppc_core_vcpu_free(vcpu);
 }
 void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu)
 {
 	kvm_arch_vcpu_free(vcpu);
 }
 int kvm_cpu_has_pending_timer(struct kvm_vcpu *vcpu)
 {
 	return kvmppc_core_pending_dec(vcpu);
 }
 static void kvmppc_decrementer_func(unsigned long data)
 {
 	struct kvm_vcpu *vcpu = (struct kvm_vcpu *)data;
 	kvmppc_core_queue_dec(vcpu);
 	if (waitqueue_active(&vcpu->wq)) {
 		wake_up_interruptible(&vcpu->wq);
 		vcpu->stat.halt_wakeup++;
 	}
 }
 /*
  * low level hrtimer wake routine. Because this runs in hardirq context
  * we schedule a tasklet to do the real work.
  */
 enum hrtimer_restart kvmppc_decrementer_wakeup(struct hrtimer *timer)
 {
 	struct kvm_vcpu *vcpu;
 	vcpu = container_of(timer, struct kvm_vcpu, arch.dec_timer);
 	tasklet_schedule(&vcpu->arch.tasklet);
 	return HRTIMER_NORESTART;
 }
 int kvm_arch_vcpu_init(struct kvm_vcpu *vcpu)
 {
 	hrtimer_init(&vcpu->arch.dec_timer, CLOCK_REALTIME, HRTIMER_MODE_ABS);
 	tasklet_init(&vcpu->arch.tasklet, kvmppc_decrementer_func, (ulong)vcpu);
 	vcpu->arch.dec_timer.function = kvmppc_decrementer_wakeup;
 	return 0;
 }
 void kvm_arch_vcpu_uninit(struct kvm_vcpu *vcpu)
 {
 	kvmppc_mmu_destroy(vcpu);
 }
 void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
 {
 	kvmppc_core_vcpu_load(vcpu, cpu);
 }
 void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
 {
 	kvmppc_core_vcpu_put(vcpu);
 }
 int kvm_arch_vcpu_ioctl_set_guest_debug(struct kvm_vcpu *vcpu,
                                         struct kvm_guest_debug *dbg)
 {
 	return -EINVAL;
 }
 static void kvmppc_complete_dcr_load(struct kvm_vcpu *vcpu,
                                      struct kvm_run *run)
 {
 	kvmppc_set_gpr(vcpu, vcpu->arch.io_gpr, run->dcr.data);
 }
 static void kvmppc_complete_mmio_load(struct kvm_vcpu *vcpu,
                                       struct kvm_run *run)
 {
 	u64 uninitialized_var(gpr);
 	if (run->mmio.len > sizeof(gpr)) {
 		printk(KERN_ERR "bad MMIO length: %d\n", run->mmio.len);
 		return;
 	}
 	if (vcpu->arch.mmio_is_bigendian) {
 		switch (run->mmio.len) {
 		case 8: gpr = *(u64 *)run->mmio.data; break;
 		case 4: gpr = *(u32 *)run->mmio.data; break;
 		case 2: gpr = *(u16 *)run->mmio.data; break;
 		case 1: gpr = *(u8 *)run->mmio.data; break;
 		}
 	} else {
 		/* Convert BE data from userland back to LE. */
 		switch (run->mmio.len) {
 		case 4: gpr = ld_le32((u32 *)run->mmio.data); break;
 		case 2: gpr = ld_le16((u16 *)run->mmio.data); break;
 		case 1: gpr = *(u8 *)run->mmio.data; break;
 		}
 	}
 	if (vcpu->arch.mmio_sign_extend) {
 		switch (run->mmio.len) {
 #ifdef CONFIG_PPC64
 		case 4:
 			gpr = (s64)(s32)gpr;
 			break;
 #endif
 		case 2:
 			gpr = (s64)(s16)gpr;
 			break;
 		case 1:
 			gpr = (s64)(s8)gpr;
 			break;
 		}
 	}
 	kvmppc_set_gpr(vcpu, vcpu->arch.io_gpr, gpr);
 	switch (vcpu->arch.io_gpr & KVM_REG_EXT_MASK) {
 	case KVM_REG_GPR:
 		kvmppc_set_gpr(vcpu, vcpu->arch.io_gpr, gpr);
 		break;
 	case KVM_REG_FPR:
 		vcpu->arch.fpr[vcpu->arch.io_gpr & KVM_REG_MASK] = gpr;
 		break;
 #ifdef CONFIG_PPC_BOOK3S
 	case KVM_REG_QPR:
 		vcpu->arch.qpr[vcpu->arch.io_gpr & KVM_REG_MASK] = gpr;
 		break;
 	case KVM_REG_FQPR:
 		vcpu->arch.fpr[vcpu->arch.io_gpr & KVM_REG_MASK] = gpr;
 		vcpu->arch.qpr[vcpu->arch.io_gpr & KVM_REG_MASK] = gpr;
 		break;
 #endif
 	default:
 		BUG();
 	}
 }
 int kvmppc_handle_load(struct kvm_run *run, struct kvm_vcpu *vcpu,
                        unsigned int rt, unsigned int bytes, int is_bigendian)
 {
 	if (bytes > sizeof(run->mmio.data)) {
 		printk(KERN_ERR "%s: bad MMIO length: %d\n", __func__,
 		       run->mmio.len);
 	}
 	run->mmio.phys_addr = vcpu->arch.paddr_accessed;
 	run->mmio.len = bytes;
 	run->mmio.is_write = 0;
 	vcpu->arch.io_gpr = rt;
 	vcpu->arch.mmio_is_bigendian = is_bigendian;
 	vcpu->mmio_needed = 1;
 	vcpu->mmio_is_write = 0;
 	vcpu->arch.mmio_sign_extend = 0;
 	return EMULATE_DO_MMIO;
 }
 /* Same as above, but sign extends */
 int kvmppc_handle_loads(struct kvm_run *run, struct kvm_vcpu *vcpu,
                         unsigned int rt, unsigned int bytes, int is_bigendian)
 {
 	int r;
 	r = kvmppc_handle_load(run, vcpu, rt, bytes, is_bigendian);
 	vcpu->arch.mmio_sign_extend = 1;
 	return r;
 }
 int kvmppc_handle_store(struct kvm_run *run, struct kvm_vcpu *vcpu,
                         u64 val, unsigned int bytes, int is_bigendian)
 {
 	void *data = run->mmio.data;
 	if (bytes > sizeof(run->mmio.data)) {
 		printk(KERN_ERR "%s: bad MMIO length: %d\n", __func__,
 		       run->mmio.len);
 	}
 	run->mmio.phys_addr = vcpu->arch.paddr_accessed;
 	run->mmio.len = bytes;
 	run->mmio.is_write = 1;
 	vcpu->mmio_needed = 1;
 	vcpu->mmio_is_write = 1;
 	/* Store the value at the lowest bytes in 'data'. */
 	if (is_bigendian) {
 		switch (bytes) {
 		case 8: *(u64 *)data = val; break;
 		case 4: *(u32 *)data = val; break;
 		case 2: *(u16 *)data = val; break;
 		case 1: *(u8  *)data = val; break;
 		}
 	} else {
 		/* Store LE value into 'data'. */
 		switch (bytes) {
 		case 4: st_le32(data, val); break;
 		case 2: st_le16(data, val); break;
 		case 1: *(u8 *)data = val; break;
 		}
 	}
 	return EMULATE_DO_MMIO;
 }
 int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run)
 {
 	int r;
 	sigset_t sigsaved;
 	if (vcpu->sigset_active)
 		sigprocmask(SIG_SETMASK, &vcpu->sigset, &sigsaved);
 	if (vcpu->mmio_needed) {
 		if (!vcpu->mmio_is_write)
 			kvmppc_complete_mmio_load(vcpu, run);
 		vcpu->mmio_needed = 0;
 	} else if (vcpu->arch.dcr_needed) {
 		if (!vcpu->arch.dcr_is_write)
 			kvmppc_complete_dcr_load(vcpu, run);
 		vcpu->arch.dcr_needed = 0;
 	} else if (vcpu->arch.osi_needed) {
 		u64 *gprs = run->osi.gprs;
 		int i;
 		for (i = 0; i < 32; i++)
 			kvmppc_set_gpr(vcpu, i, gprs[i]);
 		vcpu->arch.osi_needed = 0;
 	}
 	kvmppc_core_deliver_interrupts(vcpu);
 	local_irq_disable();
 	kvm_guest_enter();
 	r = __kvmppc_vcpu_run(run, vcpu);
 	kvm_guest_exit();
 	local_irq_enable();
 	if (vcpu->sigset_active)
 		sigprocmask(SIG_SETMASK, &sigsaved, NULL);
 	return r;
 }
 int kvm_vcpu_ioctl_interrupt(struct kvm_vcpu *vcpu, struct kvm_interrupt *irq)
 {
 	if (irq->irq == KVM_INTERRUPT_UNSET)
 		kvmppc_core_dequeue_external(vcpu, irq);
 	else
 		kvmppc_core_queue_external(vcpu, irq);
 	if (waitqueue_active(&vcpu->wq)) {
 		wake_up_interruptible(&vcpu->wq);
 		vcpu->stat.halt_wakeup++;
 	}
 	return 0;
 }
 static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
 				     struct kvm_enable_cap *cap)
 {
 	int r;
 	if (cap->flags)
 		return -EINVAL;
 	switch (cap->cap) {
 	case KVM_CAP_PPC_OSI:
 		r = 0;
 		vcpu->arch.osi_enabled = true;
 		break;
 	default:
 		r = -EINVAL;
 		break;
 	}
 	return r;
 }
 int kvm_arch_vcpu_ioctl_get_mpstate(struct kvm_vcpu *vcpu,
                                     struct kvm_mp_state *mp_state)
 {
 	return -EINVAL;
 }
 int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu,
                                     struct kvm_mp_state *mp_state)
 {
 	return -EINVAL;
 }
 long kvm_arch_vcpu_ioctl(struct file *filp,
                          unsigned int ioctl, unsigned long arg)
 {
 	struct kvm_vcpu *vcpu = filp->private_data;
 	void __user *argp = (void __user *)arg;
 	long r;
 	switch (ioctl) {
 	case KVM_INTERRUPT: {
 		struct kvm_interrupt irq;
 		r = -EFAULT;
 		if (copy_from_user(&irq, argp, sizeof(irq)))
 			goto out;
 		r = kvm_vcpu_ioctl_interrupt(vcpu, &irq);
 		goto out;
 	}
 	case KVM_ENABLE_CAP:
 	{
 		struct kvm_enable_cap cap;
 		r = -EFAULT;
 		if (copy_from_user(&cap, argp, sizeof(cap)))
 			goto out;
 		r = kvm_vcpu_ioctl_enable_cap(vcpu, &cap);
 		break;
 	}
 	default:
 		r = -EINVAL;
 	}
 out:
 	return r;
 }
 long kvm_arch_vm_ioctl(struct file *filp,
                        unsigned int ioctl, unsigned long arg)
 {
 	long r;
 	switch (ioctl) {
 	default:
 		r = -ENOTTY;
 	}
 	return r;
 }
 int kvm_arch_init(void *opaque)
 {
 	return 0;
 }
 void kvm_arch_exit(void)
 {
 }

arch/s390/kvm/kvm-s390.c

Diff comments View file @ a1f4d39

 /*
  * s390host.c --  hosting zSeries kernel virtual machines
  *
  * Copyright IBM Corp. 2008,2009
  *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU General Public License (version 2 only)
  * as published by the Free Software Foundation.
  *
  *    Author(s): Carsten Otte <cotte@de.ibm.com>
  *               Christian Borntraeger <borntraeger@de.ibm.com>
  *               Heiko Carstens <heiko.carstens@de.ibm.com>
  *               Christian Ehrhardt <ehrhardt@de.ibm.com>
  */
 #include <linux/compiler.h>
 #include <linux/err.h>
 #include <linux/fs.h>
 #include <linux/hrtimer.h>
 #include <linux/init.h>
 #include <linux/kvm.h>
 #include <linux/kvm_host.h>
 #include <linux/module.h>
 #include <linux/slab.h>
 #include <linux/timer.h>
 #include <asm/asm-offsets.h>
 #include <asm/lowcore.h>
 #include <asm/pgtable.h>
 #include <asm/nmi.h>
 #include <asm/system.h>
 #include "kvm-s390.h"
 #include "gaccess.h"
 #define VCPU_STAT(x) offsetof(struct kvm_vcpu, stat.x), KVM_STAT_VCPU
 struct kvm_stats_debugfs_item debugfs_entries[] = {
 	{ "userspace_handled", VCPU_STAT(exit_userspace) },
 	{ "exit_null", VCPU_STAT(exit_null) },
 	{ "exit_validity", VCPU_STAT(exit_validity) },
 	{ "exit_stop_request", VCPU_STAT(exit_stop_request) },
 	{ "exit_external_request", VCPU_STAT(exit_external_request) },
 	{ "exit_external_interrupt", VCPU_STAT(exit_external_interrupt) },
 	{ "exit_instruction", VCPU_STAT(exit_instruction) },
 	{ "exit_program_interruption", VCPU_STAT(exit_program_interruption) },
 	{ "exit_instr_and_program_int", VCPU_STAT(exit_instr_and_program) },
 	{ "instruction_lctlg", VCPU_STAT(instruction_lctlg) },
 	{ "instruction_lctl", VCPU_STAT(instruction_lctl) },
 	{ "deliver_emergency_signal", VCPU_STAT(deliver_emergency_signal) },
 	{ "deliver_service_signal", VCPU_STAT(deliver_service_signal) },
 	{ "deliver_virtio_interrupt", VCPU_STAT(deliver_virtio_interrupt) },
 	{ "deliver_stop_signal", VCPU_STAT(deliver_stop_signal) },
 	{ "deliver_prefix_signal", VCPU_STAT(deliver_prefix_signal) },
 	{ "deliver_restart_signal", VCPU_STAT(deliver_restart_signal) },
 	{ "deliver_program_interruption", VCPU_STAT(deliver_program_int) },
 	{ "exit_wait_state", VCPU_STAT(exit_wait_state) },
 	{ "instruction_stidp", VCPU_STAT(instruction_stidp) },
 	{ "instruction_spx", VCPU_STAT(instruction_spx) },
 	{ "instruction_stpx", VCPU_STAT(instruction_stpx) },
 	{ "instruction_stap", VCPU_STAT(instruction_stap) },
 	{ "instruction_storage_key", VCPU_STAT(instruction_storage_key) },
 	{ "instruction_stsch", VCPU_STAT(instruction_stsch) },
 	{ "instruction_chsc", VCPU_STAT(instruction_chsc) },
 	{ "instruction_stsi", VCPU_STAT(instruction_stsi) },
 	{ "instruction_stfl", VCPU_STAT(instruction_stfl) },
 	{ "instruction_sigp_sense", VCPU_STAT(instruction_sigp_sense) },
 	{ "instruction_sigp_emergency", VCPU_STAT(instruction_sigp_emergency) },
 	{ "instruction_sigp_stop", VCPU_STAT(instruction_sigp_stop) },
 	{ "instruction_sigp_set_arch", VCPU_STAT(instruction_sigp_arch) },
 	{ "instruction_sigp_set_prefix", VCPU_STAT(instruction_sigp_prefix) },
 	{ "instruction_sigp_restart", VCPU_STAT(instruction_sigp_restart) },
 	{ "diagnose_44", VCPU_STAT(diagnose_44) },
 	{ NULL }
 };
 static unsigned long long *facilities;
 /* Section: not file related */
 int kvm_arch_hardware_enable(void *garbage)
 {
 	/* every s390 is virtualization enabled ;-) */
 	return 0;
 }
 void kvm_arch_hardware_disable(void *garbage)
 {
 }
 int kvm_arch_hardware_setup(void)
 {
 	return 0;
 }
 void kvm_arch_hardware_unsetup(void)
 {
 }
 void kvm_arch_check_processor_compat(void *rtn)
 {
 }
 int kvm_arch_init(void *opaque)
 {
 	return 0;
 }
 void kvm_arch_exit(void)
 {
 }
 /* Section: device related */
 long kvm_arch_dev_ioctl(struct file *filp,
 			unsigned int ioctl, unsigned long arg)
 {
 	if (ioctl == KVM_S390_ENABLE_SIE)
 		return s390_enable_sie();
 	return -EINVAL;
 }
 int kvm_dev_ioctl_check_extension(long ext)
 {
 	int r;
 	switch (ext) {
 	case KVM_CAP_S390_PSW:
 		r = 1;
 		break;
 	default:
 		r = 0;
 	}
 	return r;
 }
 /* Section: vm related */
 /*
  * Get (and clear) the dirty memory log for a memory slot.
  */
 int kvm_vm_ioctl_get_dirty_log(struct kvm *kvm,
 			       struct kvm_dirty_log *log)
 {
 	return 0;
 }
 long kvm_arch_vm_ioctl(struct file *filp,
 		       unsigned int ioctl, unsigned long arg)
 {
 	struct kvm *kvm = filp->private_data;
 	void __user *argp = (void __user *)arg;
 	int r;
 	switch (ioctl) {
 	case KVM_S390_INTERRUPT: {
 		struct kvm_s390_interrupt s390int;
 		r = -EFAULT;
 		if (copy_from_user(&s390int, argp, sizeof(s390int)))
 			break;
 		r = kvm_s390_inject_vm(kvm, &s390int);
 		break;
 	}
 	default:
 		r = -ENOTTY;
 	}
 	return r;
 }
 struct kvm *kvm_arch_create_vm(void)
 {
 	struct kvm *kvm;
 	int rc;
 	char debug_name[16];
 	rc = s390_enable_sie();
 	if (rc)
 		goto out_nokvm;
 	rc = -ENOMEM;
 	kvm = kzalloc(sizeof(struct kvm), GFP_KERNEL);
 	if (!kvm)
 		goto out_nokvm;
 	kvm->arch.sca = (struct sca_block *) get_zeroed_page(GFP_KERNEL);
 	if (!kvm->arch.sca)
 		goto out_nosca;
 	sprintf(debug_name, "kvm-%u", current->pid);
 	kvm->arch.dbf = debug_register(debug_name, 8, 2, 8 * sizeof(long));
 	if (!kvm->arch.dbf)
 		goto out_nodbf;
 	spin_lock_init(&kvm->arch.float_int.lock);
 	INIT_LIST_HEAD(&kvm->arch.float_int.list);
 	debug_register_view(kvm->arch.dbf, &debug_sprintf_view);
 	VM_EVENT(kvm, 3, "%s", "vm created");
 	return kvm;
 out_nodbf:
 	free_page((unsigned long)(kvm->arch.sca));
 out_nosca:
 	kfree(kvm);
 out_nokvm:
 	return ERR_PTR(rc);
 }
 void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu)
 {
 	VCPU_EVENT(vcpu, 3, "%s", "free cpu");
 	clear_bit(63 - vcpu->vcpu_id, (unsigned long *) &vcpu->kvm->arch.sca->mcn);
 	if (vcpu->kvm->arch.sca->cpu[vcpu->vcpu_id].sda ==
 		(__u64) vcpu->arch.sie_block)
 		vcpu->kvm->arch.sca->cpu[vcpu->vcpu_id].sda = 0;
 	smp_mb();
 	free_page((unsigned long)(vcpu->arch.sie_block));
 	kvm_vcpu_uninit(vcpu);
 	kfree(vcpu);
 }
 static void kvm_free_vcpus(struct kvm *kvm)
 {
 	unsigned int i;
 	struct kvm_vcpu *vcpu;
 	kvm_for_each_vcpu(i, vcpu, kvm)
 		kvm_arch_vcpu_destroy(vcpu);
 	mutex_lock(&kvm->lock);
 	for (i = 0; i < atomic_read(&kvm->online_vcpus); i++)
 		kvm->vcpus[i] = NULL;
 	atomic_set(&kvm->online_vcpus, 0);
 	mutex_unlock(&kvm->lock);
 }
 void kvm_arch_sync_events(struct kvm *kvm)
 {
 }
 void kvm_arch_destroy_vm(struct kvm *kvm)
 {
 	kvm_free_vcpus(kvm);
 	kvm_free_physmem(kvm);
 	free_page((unsigned long)(kvm->arch.sca));
 	debug_unregister(kvm->arch.dbf);
 	cleanup_srcu_struct(&kvm->srcu);
 	kfree(kvm);
 }
 /* Section: vcpu related */
 int kvm_arch_vcpu_init(struct kvm_vcpu *vcpu)
 {
 	return 0;
 }
 void kvm_arch_vcpu_uninit(struct kvm_vcpu *vcpu)
 {
 	/* Nothing todo */
 }
 void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
 {
 	save_fp_regs(&vcpu->arch.host_fpregs);
 	save_access_regs(vcpu->arch.host_acrs);
 	vcpu->arch.guest_fpregs.fpc &= FPC_VALID_MASK;
 	restore_fp_regs(&vcpu->arch.guest_fpregs);
 	restore_access_regs(vcpu->arch.guest_acrs);
 }
 void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
 {
 	save_fp_regs(&vcpu->arch.guest_fpregs);
 	save_access_regs(vcpu->arch.guest_acrs);
 	restore_fp_regs(&vcpu->arch.host_fpregs);
 	restore_access_regs(vcpu->arch.host_acrs);
 }
 static void kvm_s390_vcpu_initial_reset(struct kvm_vcpu *vcpu)
 {
 	/* this equals initial cpu reset in pop, but we don't switch to ESA */
 	vcpu->arch.sie_block->gpsw.mask = 0UL;
 	vcpu->arch.sie_block->gpsw.addr = 0UL;
 	vcpu->arch.sie_block->prefix    = 0UL;
 	vcpu->arch.sie_block->ihcpu     = 0xffff;
 	vcpu->arch.sie_block->cputm     = 0UL;
 	vcpu->arch.sie_block->ckc       = 0UL;
 	vcpu->arch.sie_block->todpr     = 0;
 	memset(vcpu->arch.sie_block->gcr, 0, 16 * sizeof(__u64));
 	vcpu->arch.sie_block->gcr[0]  = 0xE0UL;
 	vcpu->arch.sie_block->gcr[14] = 0xC2000000UL;
 	vcpu->arch.guest_fpregs.fpc = 0;
 	asm volatile("lfpc %0" : : "Q" (vcpu->arch.guest_fpregs.fpc));
 	vcpu->arch.sie_block->gbea = 1;
 }
 int kvm_arch_vcpu_setup(struct kvm_vcpu *vcpu)
 {
 	atomic_set(&vcpu->arch.sie_block->cpuflags, CPUSTAT_ZARCH);
 	set_bit(KVM_REQ_MMU_RELOAD, &vcpu->requests);
 	vcpu->arch.sie_block->ecb   = 6;
 	vcpu->arch.sie_block->eca   = 0xC1002001U;
 	vcpu->arch.sie_block->fac   = (int) (long) facilities;
 	hrtimer_init(&vcpu->arch.ckc_timer, CLOCK_REALTIME, HRTIMER_MODE_ABS);
 	tasklet_init(&vcpu->arch.tasklet, kvm_s390_tasklet,
 		     (unsigned long) vcpu);
 	vcpu->arch.ckc_timer.function = kvm_s390_idle_wakeup;
 	get_cpu_id(&vcpu->arch.cpu_id);
 	vcpu->arch.cpu_id.version = 0xff;
 	return 0;
 }
 struct kvm_vcpu *kvm_arch_vcpu_create(struct kvm *kvm,
 				      unsigned int id)
 {
 	struct kvm_vcpu *vcpu = kzalloc(sizeof(struct kvm_vcpu), GFP_KERNEL);
 	int rc = -ENOMEM;
 	if (!vcpu)
 		goto out_nomem;
 	vcpu->arch.sie_block = (struct kvm_s390_sie_block *)
 					get_zeroed_page(GFP_KERNEL);
 	if (!vcpu->arch.sie_block)
 		goto out_free_cpu;
 	vcpu->arch.sie_block->icpua = id;
 	BUG_ON(!kvm->arch.sca);
 	if (!kvm->arch.sca->cpu[id].sda)
 		kvm->arch.sca->cpu[id].sda = (__u64) vcpu->arch.sie_block;
 	vcpu->arch.sie_block->scaoh = (__u32)(((__u64)kvm->arch.sca) >> 32);
 	vcpu->arch.sie_block->scaol = (__u32)(__u64)kvm->arch.sca;
 	set_bit(63 - id, (unsigned long *) &kvm->arch.sca->mcn);
 	spin_lock_init(&vcpu->arch.local_int.lock);
 	INIT_LIST_HEAD(&vcpu->arch.local_int.list);
 	vcpu->arch.local_int.float_int = &kvm->arch.float_int;
 	spin_lock(&kvm->arch.float_int.lock);
 	kvm->arch.float_int.local_int[id] = &vcpu->arch.local_int;
 	init_waitqueue_head(&vcpu->arch.local_int.wq);
 	vcpu->arch.local_int.cpuflags = &vcpu->arch.sie_block->cpuflags;
 	spin_unlock(&kvm->arch.float_int.lock);
 	rc = kvm_vcpu_init(vcpu, kvm, id);
 	if (rc)
 		goto out_free_sie_block;
 	VM_EVENT(kvm, 3, "create cpu %d at %p, sie block at %p", id, vcpu,
 		 vcpu->arch.sie_block);
 	return vcpu;
 out_free_sie_block:
 	free_page((unsigned long)(vcpu->arch.sie_block));
 out_free_cpu:
 	kfree(vcpu);
 out_nomem:
 	return ERR_PTR(rc);
 }
 int kvm_arch_vcpu_runnable(struct kvm_vcpu *vcpu)
 {
 	/* kvm common code refers to this, but never calls it */
 	BUG();
 	return 0;
 }
 static int kvm_arch_vcpu_ioctl_initial_reset(struct kvm_vcpu *vcpu)
 {
 	kvm_s390_vcpu_initial_reset(vcpu);
 	return 0;
 }
 int kvm_arch_vcpu_ioctl_set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs)
 {
 	memcpy(&vcpu->arch.guest_gprs, &regs->gprs, sizeof(regs->gprs));
 	return 0;
 }
 int kvm_arch_vcpu_ioctl_get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs)
 {
 	memcpy(&regs->gprs, &vcpu->arch.guest_gprs, sizeof(regs->gprs));
 	return 0;
 }
 int kvm_arch_vcpu_ioctl_set_sregs(struct kvm_vcpu *vcpu,
 				  struct kvm_sregs *sregs)
 {
 	memcpy(&vcpu->arch.guest_acrs, &sregs->acrs, sizeof(sregs->acrs));
 	memcpy(&vcpu->arch.sie_block->gcr, &sregs->crs, sizeof(sregs->crs));
 	return 0;
 }
 int kvm_arch_vcpu_ioctl_get_sregs(struct kvm_vcpu *vcpu,
 				  struct kvm_sregs *sregs)
 {
 	memcpy(&sregs->acrs, &vcpu->arch.guest_acrs, sizeof(sregs->acrs));
 	memcpy(&sregs->crs, &vcpu->arch.sie_block->gcr, sizeof(sregs->crs));
 	return 0;
 }
 int kvm_arch_vcpu_ioctl_set_fpu(struct kvm_vcpu *vcpu, struct kvm_fpu *fpu)
 {
 	memcpy(&vcpu->arch.guest_fpregs.fprs, &fpu->fprs, sizeof(fpu->fprs));
 	vcpu->arch.guest_fpregs.fpc = fpu->fpc;
 	return 0;
 }
 int kvm_arch_vcpu_ioctl_get_fpu(struct kvm_vcpu *vcpu, struct kvm_fpu *fpu)
 {
 	memcpy(&fpu->fprs, &vcpu->arch.guest_fpregs.fprs, sizeof(fpu->fprs));
 	fpu->fpc = vcpu->arch.guest_fpregs.fpc;
 	return 0;
 }
 static int kvm_arch_vcpu_ioctl_set_initial_psw(struct kvm_vcpu *vcpu, psw_t psw)
 {
 	int rc = 0;
 	if (atomic_read(&vcpu->arch.sie_block->cpuflags) & CPUSTAT_RUNNING)
 		rc = -EBUSY;
 	else {
 		vcpu->run->psw_mask = psw.mask;
 		vcpu->run->psw_addr = psw.addr;
 	}
 	return rc;
 }
 int kvm_arch_vcpu_ioctl_translate(struct kvm_vcpu *vcpu,
 				  struct kvm_translation *tr)
 {
 	return -EINVAL; /* not implemented yet */
 }
 int kvm_arch_vcpu_ioctl_set_guest_debug(struct kvm_vcpu *vcpu,
 					struct kvm_guest_debug *dbg)
 {
 	return -EINVAL; /* not implemented yet */
 }
 int kvm_arch_vcpu_ioctl_get_mpstate(struct kvm_vcpu *vcpu,
 				    struct kvm_mp_state *mp_state)
 {
 	return -EINVAL; /* not implemented yet */
 }
 int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu,
 				    struct kvm_mp_state *mp_state)
 {
 	return -EINVAL; /* not implemented yet */
 }
 static void __vcpu_run(struct kvm_vcpu *vcpu)
 {
 	memcpy(&vcpu->arch.sie_block->gg14, &vcpu->arch.guest_gprs[14], 16);
 	if (need_resched())
 		schedule();
 	if (test_thread_flag(TIF_MCCK_PENDING))
 		s390_handle_mcck();
 	kvm_s390_deliver_pending_interrupts(vcpu);
 	vcpu->arch.sie_block->icptcode = 0;
 	local_irq_disable();
 	kvm_guest_enter();
 	local_irq_enable();
 	VCPU_EVENT(vcpu, 6, "entering sie flags %x",
 		   atomic_read(&vcpu->arch.sie_block->cpuflags));
 	if (sie64a(vcpu->arch.sie_block, vcpu->arch.guest_gprs)) {
 		VCPU_EVENT(vcpu, 3, "%s", "fault in sie instruction");
 		kvm_s390_inject_program_int(vcpu, PGM_ADDRESSING);
 	}
 	VCPU_EVENT(vcpu, 6, "exit sie icptcode %d",
 		   vcpu->arch.sie_block->icptcode);
 	local_irq_disable();
 	kvm_guest_exit();
 	local_irq_enable();
 	memcpy(&vcpu->arch.guest_gprs[14], &vcpu->arch.sie_block->gg14, 16);
 }
 int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
 {
 	int rc;
 	sigset_t sigsaved;
 rerun_vcpu:
 	if (vcpu->requests)
 		if (test_and_clear_bit(KVM_REQ_MMU_RELOAD, &vcpu->requests))
 			kvm_s390_vcpu_set_mem(vcpu);
 	/* verify, that memory has been registered */
 	if (!vcpu->arch.sie_block->gmslm) {
 		vcpu_put(vcpu);
 		VCPU_EVENT(vcpu, 3, "%s", "no memory registered to run vcpu");
 		return -EINVAL;
 	}
 	if (vcpu->sigset_active)
 		sigprocmask(SIG_SETMASK, &vcpu->sigset, &sigsaved);
 	atomic_set_mask(CPUSTAT_RUNNING, &vcpu->arch.sie_block->cpuflags);
 	BUG_ON(vcpu->kvm->arch.float_int.local_int[vcpu->vcpu_id] == NULL);
 	switch (kvm_run->exit_reason) {
 	case KVM_EXIT_S390_SIEIC:
 	case KVM_EXIT_UNKNOWN:
 	case KVM_EXIT_INTR:
 	case KVM_EXIT_S390_RESET:
 		break;
 	default:
 		BUG();
 	}
 	vcpu->arch.sie_block->gpsw.mask = kvm_run->psw_mask;
 	vcpu->arch.sie_block->gpsw.addr = kvm_run->psw_addr;
 	might_fault();
 	do {
 		__vcpu_run(vcpu);
 		rc = kvm_handle_sie_intercept(vcpu);
 	} while (!signal_pending(current) && !rc);
 	if (rc == SIE_INTERCEPT_RERUNVCPU)
 		goto rerun_vcpu;
 	if (signal_pending(current) && !rc) {
 		kvm_run->exit_reason = KVM_EXIT_INTR;
 		rc = -EINTR;
 	}
 	if (rc == -EOPNOTSUPP) {
 		/* intercept cannot be handled in-kernel, prepare kvm-run */
 		kvm_run->exit_reason         = KVM_EXIT_S390_SIEIC;
 		kvm_run->s390_sieic.icptcode = vcpu->arch.sie_block->icptcode;
 		kvm_run->s390_sieic.ipa      = vcpu->arch.sie_block->ipa;
 		kvm_run->s390_sieic.ipb      = vcpu->arch.sie_block->ipb;
 		rc = 0;
 	}
 	if (rc == -EREMOTE) {
 		/* intercept was handled, but userspace support is needed
 		 * kvm_run has been prepared by the handler */
 		rc = 0;
 	}
 	kvm_run->psw_mask     = vcpu->arch.sie_block->gpsw.mask;
 	kvm_run->psw_addr     = vcpu->arch.sie_block->gpsw.addr;
 	if (vcpu->sigset_active)
 		sigprocmask(SIG_SETMASK, &sigsaved, NULL);
 	vcpu->stat.exit_userspace++;
 	return rc;
 }
 static int __guestcopy(struct kvm_vcpu *vcpu, u64 guestdest, const void *from,
 		       unsigned long n, int prefix)
 {
 	if (prefix)
 		return copy_to_guest(vcpu, guestdest, from, n);
 	else
 		return copy_to_guest_absolute(vcpu, guestdest, from, n);
 }
 /*
  * store status at address
  * we use have two special cases:
  * KVM_S390_STORE_STATUS_NOADDR: -> 0x1200 on 64 bit
  * KVM_S390_STORE_STATUS_PREFIXED: -> prefix
  */
 int kvm_s390_vcpu_store_status(struct kvm_vcpu *vcpu, unsigned long addr)
 {
 	const unsigned char archmode = 1;
 	int prefix;
 	if (addr == KVM_S390_STORE_STATUS_NOADDR) {
 		if (copy_to_guest_absolute(vcpu, 163ul, &archmode, 1))
 			return -EFAULT;
 		addr = SAVE_AREA_BASE;
 		prefix = 0;
 	} else if (addr == KVM_S390_STORE_STATUS_PREFIXED) {
 		if (copy_to_guest(vcpu, 163ul, &archmode, 1))
 			return -EFAULT;
 		addr = SAVE_AREA_BASE;
 		prefix = 1;
 	} else
 		prefix = 0;
 	if (__guestcopy(vcpu, addr + offsetof(struct save_area, fp_regs),
 			vcpu->arch.guest_fpregs.fprs, 128, prefix))
 		return -EFAULT;
 	if (__guestcopy(vcpu, addr + offsetof(struct save_area, gp_regs),
 			vcpu->arch.guest_gprs, 128, prefix))
 		return -EFAULT;
 	if (__guestcopy(vcpu, addr + offsetof(struct save_area, psw),
 			&vcpu->arch.sie_block->gpsw, 16, prefix))
 		return -EFAULT;
 	if (__guestcopy(vcpu, addr + offsetof(struct save_area, pref_reg),
 			&vcpu->arch.sie_block->prefix, 4, prefix))
 		return -EFAULT;
 	if (__guestcopy(vcpu,
 			addr + offsetof(struct save_area, fp_ctrl_reg),
 			&vcpu->arch.guest_fpregs.fpc, 4, prefix))
 		return -EFAULT;
 	if (__guestcopy(vcpu, addr + offsetof(struct save_area, tod_reg),
 			&vcpu->arch.sie_block->todpr, 4, prefix))
 		return -EFAULT;
 	if (__guestcopy(vcpu, addr + offsetof(struct save_area, timer),
 			&vcpu->arch.sie_block->cputm, 8, prefix))
 		return -EFAULT;
 	if (__guestcopy(vcpu, addr + offsetof(struct save_area, clk_cmp),
 			&vcpu->arch.sie_block->ckc, 8, prefix))
 		return -EFAULT;
 	if (__guestcopy(vcpu, addr + offsetof(struct save_area, acc_regs),
 			&vcpu->arch.guest_acrs, 64, prefix))
 		return -EFAULT;
 	if (__guestcopy(vcpu,
 			addr + offsetof(struct save_area, ctrl_regs),
 			&vcpu->arch.sie_block->gcr, 128, prefix))
 		return -EFAULT;
 	return 0;
 }
 long kvm_arch_vcpu_ioctl(struct file *filp,
 			 unsigned int ioctl, unsigned long arg)
 {
 	struct kvm_vcpu *vcpu = filp->private_data;
 	void __user *argp = (void __user *)arg;
 	long r;
 	switch (ioctl) {
 	case KVM_S390_INTERRUPT: {
 		struct kvm_s390_interrupt s390int;
 		r = -EFAULT;
 		if (copy_from_user(&s390int, argp, sizeof(s390int)))
 			break;
 		r = kvm_s390_inject_vcpu(vcpu, &s390int);
 		break;
 	}
 	case KVM_S390_STORE_STATUS:
 		r = kvm_s390_vcpu_store_status(vcpu, arg);
 		break;
 	case KVM_S390_SET_INITIAL_PSW: {
 		psw_t psw;
 		r = -EFAULT;
 		if (copy_from_user(&psw, argp, sizeof(psw)))
 			break;
 		r = kvm_arch_vcpu_ioctl_set_initial_psw(vcpu, psw);
 		break;
 	}
 	case KVM_S390_INITIAL_RESET:
 		r = kvm_arch_vcpu_ioctl_initial_reset(vcpu);
 		break;
 	default:
 		r = -EINVAL;
 	}
 	return r;
 }
 /* Section: memory related */
 int kvm_arch_prepare_memory_region(struct kvm *kvm,
 				   struct kvm_memory_slot *memslot,
 				   struct kvm_memory_slot old,
 				   struct kvm_userspace_memory_region *mem,
 				   int user_alloc)
 {
 	/* A few sanity checks. We can have exactly one memory slot which has
 	   to start at guest virtual zero and which has to be located at a
 	   page boundary in userland and which has to end at a page boundary.
 	   The memory in userland is ok to be fragmented into various different
 	   vmas. It is okay to mmap() and munmap() stuff in this slot after
 	   doing this call at any time */
 	if (mem->slot)
 		return -EINVAL;
 	if (mem->guest_phys_addr)
 		return -EINVAL;
 	if (mem->userspace_addr & (PAGE_SIZE - 1))
 		return -EINVAL;
 	if (mem->memory_size & (PAGE_SIZE - 1))
 		return -EINVAL;
 	if (!user_alloc)
 		return -EINVAL;
 	return 0;
 }
 void kvm_arch_commit_memory_region(struct kvm *kvm,
 				struct kvm_userspace_memory_region *mem,
 				struct kvm_memory_slot old,
 				int user_alloc)
 {
 	int i;
 	struct kvm_vcpu *vcpu;
 	/* request update of sie control block for all available vcpus */
 	kvm_for_each_vcpu(i, vcpu, kvm) {
 		if (test_and_set_bit(KVM_REQ_MMU_RELOAD, &vcpu->requests))
 			continue;
 		kvm_s390_inject_sigp_stop(vcpu, ACTION_RELOADVCPU_ON_STOP);
 	}
 }
 void kvm_arch_flush_shadow(struct kvm *kvm)
 {
 }
-gfn_t unalias_gfn(struct kvm *kvm, gfn_t gfn)
-{
-	return gfn;
-}
 static int __init kvm_s390_init(void)
 {
 	int ret;
 	ret = kvm_init(NULL, sizeof(struct kvm_vcpu), 0, THIS_MODULE);
 	if (ret)
 		return ret;
 	/*
 	 * guests can ask for up to 255+1 double words, we need a full page
 	 * to hold the maximum amount of facilites. On the other hand, we
 	 * only set facilities that are known to work in KVM.
 	 */
 	facilities = (unsigned long long *) get_zeroed_page(GFP_KERNEL|GFP_DMA);
 	if (!facilities) {
 		kvm_exit();
 		return -ENOMEM;
 	}
 	stfle(facilities, 1);
 	facilities[0] &= 0xff00fff3f0700000ULL;
 	return 0;
 }
 static void __exit kvm_s390_exit(void)
 {
 	free_page((unsigned long) facilities);
 	kvm_exit();
 }
 module_init(kvm_s390_init);
 module_exit(kvm_s390_exit);

arch/x86/include/asm/kvm_host.h

Diff comments View file @ a1f4d39

 /*
  * Kernel-based Virtual Machine driver for Linux
  *
  * This header defines architecture specific interfaces, x86 version
  *
  * This work is licensed under the terms of the GNU GPL, version 2.  See
  * the COPYING file in the top-level directory.
  *
  */
 #ifndef _ASM_X86_KVM_HOST_H
 #define _ASM_X86_KVM_HOST_H
 #include <linux/types.h>
 #include <linux/mm.h>
 #include <linux/mmu_notifier.h>
 #include <linux/tracepoint.h>
 #include <linux/kvm.h>
 #include <linux/kvm_para.h>
 #include <linux/kvm_types.h>
 #include <asm/pvclock-abi.h>
 #include <asm/desc.h>
 #include <asm/mtrr.h>
 #include <asm/msr-index.h>
 #define KVM_MAX_VCPUS 64
 #define KVM_MEMORY_SLOTS 32
 /* memory slots that does not exposed to userspace */
 #define KVM_PRIVATE_MEM_SLOTS 4
 #define KVM_PIO_PAGE_OFFSET 1
 #define KVM_COALESCED_MMIO_PAGE_OFFSET 2
 #define CR3_PAE_RESERVED_BITS ((X86_CR3_PWT | X86_CR3_PCD) - 1)
 #define CR3_NONPAE_RESERVED_BITS ((PAGE_SIZE-1) & ~(X86_CR3_PWT | X86_CR3_PCD))
 #define CR3_L_MODE_RESERVED_BITS (CR3_NONPAE_RESERVED_BITS |	\
 				  0xFFFFFF0000000000ULL)
 #define INVALID_PAGE (~(hpa_t)0)
 #define UNMAPPED_GVA (~(gpa_t)0)
 /* KVM Hugepage definitions for x86 */
 #define KVM_NR_PAGE_SIZES	3
 #define KVM_HPAGE_SHIFT(x)	(PAGE_SHIFT + (((x) - 1) * 9))
 #define KVM_HPAGE_SIZE(x)	(1UL << KVM_HPAGE_SHIFT(x))
 #define KVM_HPAGE_MASK(x)	(~(KVM_HPAGE_SIZE(x) - 1))
 #define KVM_PAGES_PER_HPAGE(x)	(KVM_HPAGE_SIZE(x) / PAGE_SIZE)
 #define DE_VECTOR 0
 #define DB_VECTOR 1
 #define BP_VECTOR 3
 #define OF_VECTOR 4
 #define BR_VECTOR 5
 #define UD_VECTOR 6
 #define NM_VECTOR 7
 #define DF_VECTOR 8
 #define TS_VECTOR 10
 #define NP_VECTOR 11
 #define SS_VECTOR 12
 #define GP_VECTOR 13
 #define PF_VECTOR 14
 #define MF_VECTOR 16
 #define MC_VECTOR 18
 #define SELECTOR_TI_MASK (1 << 2)
 #define SELECTOR_RPL_MASK 0x03
 #define IOPL_SHIFT 12
-#define KVM_ALIAS_SLOTS 4
 #define KVM_PERMILLE_MMU_PAGES 20
 #define KVM_MIN_ALLOC_MMU_PAGES 64
 #define KVM_MMU_HASH_SHIFT 10
 #define KVM_NUM_MMU_PAGES (1 << KVM_MMU_HASH_SHIFT)
 #define KVM_MIN_FREE_MMU_PAGES 5
 #define KVM_REFILL_PAGES 25
 #define KVM_MAX_CPUID_ENTRIES 40
 #define KVM_NR_FIXED_MTRR_REGION 88
 #define KVM_NR_VAR_MTRR 8
 extern spinlock_t kvm_lock;
 extern struct list_head vm_list;
 struct kvm_vcpu;
 struct kvm;
 enum kvm_reg {
 	VCPU_REGS_RAX = 0,
 	VCPU_REGS_RCX = 1,
 	VCPU_REGS_RDX = 2,
 	VCPU_REGS_RBX = 3,
 	VCPU_REGS_RSP = 4,
 	VCPU_REGS_RBP = 5,
 	VCPU_REGS_RSI = 6,
 	VCPU_REGS_RDI = 7,
 #ifdef CONFIG_X86_64
 	VCPU_REGS_R8 = 8,
 	VCPU_REGS_R9 = 9,
 	VCPU_REGS_R10 = 10,
 	VCPU_REGS_R11 = 11,
 	VCPU_REGS_R12 = 12,
 	VCPU_REGS_R13 = 13,
 	VCPU_REGS_R14 = 14,
 	VCPU_REGS_R15 = 15,
 #endif
 	VCPU_REGS_RIP,
 	NR_VCPU_REGS
 };
 enum kvm_reg_ex {
 	VCPU_EXREG_PDPTR = NR_VCPU_REGS,
 };
 enum {
 	VCPU_SREG_ES,
 	VCPU_SREG_CS,
 	VCPU_SREG_SS,
 	VCPU_SREG_DS,
 	VCPU_SREG_FS,
 	VCPU_SREG_GS,
 	VCPU_SREG_TR,
 	VCPU_SREG_LDTR,
 };
 #include <asm/kvm_emulate.h>
 #define KVM_NR_MEM_OBJS 40
 #define KVM_NR_DB_REGS	4
 #define DR6_BD		(1 << 13)
 #define DR6_BS		(1 << 14)
 #define DR6_FIXED_1	0xffff0ff0
 #define DR6_VOLATILE	0x0000e00f
 #define DR7_BP_EN_MASK	0x000000ff
 #define DR7_GE		(1 << 9)
 #define DR7_GD		(1 << 13)
 #define DR7_FIXED_1	0x00000400
 #define DR7_VOLATILE	0xffff23ff
 /*
  * We don't want allocation failures within the mmu code, so we preallocate
  * enough memory for a single page fault in a cache.
  */
 struct kvm_mmu_memory_cache {
 	int nobjs;
 	void *objects[KVM_NR_MEM_OBJS];
 };
 #define NR_PTE_CHAIN_ENTRIES 5
 struct kvm_pte_chain {
 	u64 *parent_ptes[NR_PTE_CHAIN_ENTRIES];
 	struct hlist_node link;
 };
 /*
  * kvm_mmu_page_role, below, is defined as:
  *
  *   bits 0:3 - total guest paging levels (2-4, or zero for real mode)
  *   bits 4:7 - page table level for this shadow (1-4)
  *   bits 8:9 - page table quadrant for 2-level guests
  *   bit   16 - direct mapping of virtual to physical mapping at gfn
  *              used for real mode and two-dimensional paging
  *   bits 17:19 - common access permissions for all ptes in this shadow page
  */
 union kvm_mmu_page_role {
 	unsigned word;
 	struct {
 		unsigned level:4;
 		unsigned cr4_pae:1;
 		unsigned quadrant:2;
 		unsigned pad_for_nice_hex_output:6;
 		unsigned direct:1;
 		unsigned access:3;
 		unsigned invalid:1;
 		unsigned nxe:1;
 		unsigned cr0_wp:1;
 	};
 };
 struct kvm_mmu_page {
 	struct list_head link;
 	struct hlist_node hash_link;
 	/*
 	 * The following two entries are used to key the shadow page in the
 	 * hash table.
 	 */
 	gfn_t gfn;
 	union kvm_mmu_page_role role;
 	u64 *spt;
 	/* hold the gfn of each spte inside spt */
 	gfn_t *gfns;
 	/*
 	 * One bit set per slot which has memory
 	 * in this shadow page.
 	 */
 	DECLARE_BITMAP(slot_bitmap, KVM_MEMORY_SLOTS + KVM_PRIVATE_MEM_SLOTS);
 	bool multimapped;         /* More than one parent_pte? */
 	bool unsync;
 	int root_count;          /* Currently serving as active root */
 	unsigned int unsync_children;
 	union {
 		u64 *parent_pte;               /* !multimapped */
 		struct hlist_head parent_ptes; /* multimapped, kvm_pte_chain */
 	};
 	DECLARE_BITMAP(unsync_child_bitmap, 512);
 };
 struct kvm_pv_mmu_op_buffer {
 	void *ptr;
 	unsigned len;
 	unsigned processed;
 	char buf[512] __aligned(sizeof(long));
 };
 struct kvm_pio_request {
 	unsigned long count;
 	int in;
 	int port;
 	int size;
 };
 /*
  * x86 supports 3 paging modes (4-level 64-bit, 3-level 64-bit, and 2-level
  * 32-bit).  The kvm_mmu structure abstracts the details of the current mmu
  * mode.
  */
 struct kvm_mmu {
 	void (*new_cr3)(struct kvm_vcpu *vcpu);
 	int (*page_fault)(struct kvm_vcpu *vcpu, gva_t gva, u32 err);
 	void (*free)(struct kvm_vcpu *vcpu);
 	gpa_t (*gva_to_gpa)(struct kvm_vcpu *vcpu, gva_t gva, u32 access,
 			    u32 *error);
 	void (*prefetch_page)(struct kvm_vcpu *vcpu,
 			      struct kvm_mmu_page *page);
 	int (*sync_page)(struct kvm_vcpu *vcpu,
 			 struct kvm_mmu_page *sp, bool clear_unsync);
 	void (*invlpg)(struct kvm_vcpu *vcpu, gva_t gva);
 	hpa_t root_hpa;
 	int root_level;
 	int shadow_root_level;
 	union kvm_mmu_page_role base_role;
 	u64 *pae_root;
 	u64 rsvd_bits_mask[2][4];
 };
 struct kvm_vcpu_arch {
 	u64 host_tsc;
 	/*
 	 * rip and regs accesses must go through
 	 * kvm_{register,rip}_{read,write} functions.
 	 */
 	unsigned long regs[NR_VCPU_REGS];
 	u32 regs_avail;
 	u32 regs_dirty;
 	unsigned long cr0;
 	unsigned long cr0_guest_owned_bits;
 	unsigned long cr2;
 	unsigned long cr3;
 	unsigned long cr4;
 	unsigned long cr4_guest_owned_bits;
 	unsigned long cr8;
 	u32 hflags;
 	u64 pdptrs[4]; /* pae */
 	u64 efer;
 	u64 apic_base;
 	struct kvm_lapic *apic;    /* kernel irqchip context */
 	int32_t apic_arb_prio;
 	int mp_state;
 	int sipi_vector;
 	u64 ia32_misc_enable_msr;
 	bool tpr_access_reporting;
 	struct kvm_mmu mmu;
 	/* only needed in kvm_pv_mmu_op() path, but it's hot so
 	 * put it here to avoid allocation */
 	struct kvm_pv_mmu_op_buffer mmu_op_buffer;
 	struct kvm_mmu_memory_cache mmu_pte_chain_cache;
 	struct kvm_mmu_memory_cache mmu_rmap_desc_cache;
 	struct kvm_mmu_memory_cache mmu_page_cache;
 	struct kvm_mmu_memory_cache mmu_page_header_cache;
 	gfn_t last_pt_write_gfn;
 	int   last_pt_write_count;
 	u64  *last_pte_updated;
 	gfn_t last_pte_gfn;
 	struct {
 		gfn_t gfn;	/* presumed gfn during guest pte update */
 		pfn_t pfn;	/* pfn corresponding to that gfn */
 		unsigned long mmu_seq;
 	} update_pte;
 	struct fpu guest_fpu;
 	u64 xcr0;
 	gva_t mmio_fault_cr2;
 	struct kvm_pio_request pio;
 	void *pio_data;
 	u8 event_exit_inst_len;
 	struct kvm_queued_exception {
 		bool pending;
 		bool has_error_code;
 		bool reinject;
 		u8 nr;
 		u32 error_code;
 	} exception;
 	struct kvm_queued_interrupt {
 		bool pending;
 		bool soft;
 		u8 nr;
 	} interrupt;
 	int halt_request; /* real mode on Intel only */
 	int cpuid_nent;
 	struct kvm_cpuid_entry2 cpuid_entries[KVM_MAX_CPUID_ENTRIES];
 	/* emulate context */
 	struct x86_emulate_ctxt emulate_ctxt;
 	gpa_t time;
 	struct pvclock_vcpu_time_info hv_clock;
 	unsigned int hv_clock_tsc_khz;
 	unsigned int time_offset;
 	struct page *time_page;
 	bool nmi_pending;
 	bool nmi_injected;
 	struct mtrr_state_type mtrr_state;
 	u32 pat;
 	int switch_db_regs;
 	unsigned long db[KVM_NR_DB_REGS];
 	unsigned long dr6;
 	unsigned long dr7;
 	unsigned long eff_db[KVM_NR_DB_REGS];
 	u64 mcg_cap;
 	u64 mcg_status;
 	u64 mcg_ctl;
 	u64 *mce_banks;
 	/* used for guest single stepping over the given code position */
 	unsigned long singlestep_rip;
 	/* fields used by HYPER-V emulation */
 	u64 hv_vapic;
 };
-struct kvm_mem_alias {
-	gfn_t base_gfn;
-	unsigned long npages;
-	gfn_t target_gfn;
-#define KVM_ALIAS_INVALID     1UL
-	unsigned long flags;
-};
-#define KVM_ARCH_HAS_UNALIAS_INSTANTIATION
-struct kvm_mem_aliases {
-	struct kvm_mem_alias aliases[KVM_ALIAS_SLOTS];
-	int naliases;
-};
 struct kvm_arch {
-	struct kvm_mem_aliases *aliases;
 	unsigned int n_free_mmu_pages;
 	unsigned int n_requested_mmu_pages;
 	unsigned int n_alloc_mmu_pages;
 	atomic_t invlpg_counter;
 	struct hlist_head mmu_page_hash[KVM_NUM_MMU_PAGES];
 	/*
 	 * Hash table of struct kvm_mmu_page.
 	 */
 	struct list_head active_mmu_pages;
 	struct list_head assigned_dev_head;
 	struct iommu_domain *iommu_domain;
 	int iommu_flags;
 	struct kvm_pic *vpic;
 	struct kvm_ioapic *vioapic;
 	struct kvm_pit *vpit;
 	int vapics_in_nmi_mode;
 	unsigned int tss_addr;
 	struct page *apic_access_page;
 	gpa_t wall_clock;
 	struct page *ept_identity_pagetable;
 	bool ept_identity_pagetable_done;
 	gpa_t ept_identity_map_addr;
 	unsigned long irq_sources_bitmap;
 	u64 vm_init_tsc;
 	s64 kvmclock_offset;
 	struct kvm_xen_hvm_config xen_hvm_config;
 	/* fields used by HYPER-V emulation */
 	u64 hv_guest_os_id;
 	u64 hv_hypercall;
 };
 struct kvm_vm_stat {
 	u32 mmu_shadow_zapped;
 	u32 mmu_pte_write;
 	u32 mmu_pte_updated;
 	u32 mmu_pde_zapped;
 	u32 mmu_flooded;
 	u32 mmu_recycled;
 	u32 mmu_cache_miss;
 	u32 mmu_unsync;
 	u32 remote_tlb_flush;
 	u32 lpages;
 };
 struct kvm_vcpu_stat {
 	u32 pf_fixed;
 	u32 pf_guest;
 	u32 tlb_flush;
 	u32 invlpg;
 	u32 exits;
 	u32 io_exits;
 	u32 mmio_exits;
 	u32 signal_exits;
 	u32 irq_window_exits;
 	u32 nmi_window_exits;
 	u32 halt_exits;
 	u32 halt_wakeup;
 	u32 request_irq_exits;
 	u32 irq_exits;
 	u32 host_state_reload;
 	u32 efer_reload;
 	u32 fpu_reload;
 	u32 insn_emulation;
 	u32 insn_emulation_fail;
 	u32 hypercalls;
 	u32 irq_injections;
 	u32 nmi_injections;
 };
 struct kvm_x86_ops {
 	int (*cpu_has_kvm_support)(void);          /* __init */
 	int (*disabled_by_bios)(void);             /* __init */
 	int (*hardware_enable)(void *dummy);
 	void (*hardware_disable)(void *dummy);
 	void (*check_processor_compatibility)(void *rtn);
 	int (*hardware_setup)(void);               /* __init */
 	void (*hardware_unsetup)(void);            /* __exit */
 	bool (*cpu_has_accelerated_tpr)(void);
 	void (*cpuid_update)(struct kvm_vcpu *vcpu);
 	/* Create, but do not attach this VCPU */
 	struct kvm_vcpu *(*vcpu_create)(struct kvm *kvm, unsigned id);
 	void (*vcpu_free)(struct kvm_vcpu *vcpu);
 	int (*vcpu_reset)(struct kvm_vcpu *vcpu);
 	void (*prepare_guest_switch)(struct kvm_vcpu *vcpu);
 	void (*vcpu_load)(struct kvm_vcpu *vcpu, int cpu);
 	void (*vcpu_put)(struct kvm_vcpu *vcpu);
 	void (*set_guest_debug)(struct kvm_vcpu *vcpu,
 				struct kvm_guest_debug *dbg);
 	int (*get_msr)(struct kvm_vcpu *vcpu, u32 msr_index, u64 *pdata);
 	int (*set_msr)(struct kvm_vcpu *vcpu, u32 msr_index, u64 data);
 	u64 (*get_segment_base)(struct kvm_vcpu *vcpu, int seg);
 	void (*get_segment)(struct kvm_vcpu *vcpu,
 			    struct kvm_segment *var, int seg);
 	int (*get_cpl)(struct kvm_vcpu *vcpu);
 	void (*set_segment)(struct kvm_vcpu *vcpu,
 			    struct kvm_segment *var, int seg);
 	void (*get_cs_db_l_bits)(struct kvm_vcpu *vcpu, int *db, int *l);
 	void (*decache_cr0_guest_bits)(struct kvm_vcpu *vcpu);
 	void (*decache_cr4_guest_bits)(struct kvm_vcpu *vcpu);
 	void (*set_cr0)(struct kvm_vcpu *vcpu, unsigned long cr0);
 	void (*set_cr3)(struct kvm_vcpu *vcpu, unsigned long cr3);
 	void (*set_cr4)(struct kvm_vcpu *vcpu, unsigned long cr4);
 	void (*set_efer)(struct kvm_vcpu *vcpu, u64 efer);
 	void (*get_idt)(struct kvm_vcpu *vcpu, struct desc_ptr *dt);
 	void (*set_idt)(struct kvm_vcpu *vcpu, struct desc_ptr *dt);
 	void (*get_gdt)(struct kvm_vcpu *vcpu, struct desc_ptr *dt);
 	void (*set_gdt)(struct kvm_vcpu *vcpu, struct desc_ptr *dt);
 	void (*set_dr7)(struct kvm_vcpu *vcpu, unsigned long value);
 	void (*cache_reg)(struct kvm_vcpu *vcpu, enum kvm_reg reg);
 	unsigned long (*get_rflags)(struct kvm_vcpu *vcpu);
 	void (*set_rflags)(struct kvm_vcpu *vcpu, unsigned long rflags);
 	void (*fpu_activate)(struct kvm_vcpu *vcpu);
 	void (*fpu_deactivate)(struct kvm_vcpu *vcpu);
 	void (*tlb_flush)(struct kvm_vcpu *vcpu);
 	void (*run)(struct kvm_vcpu *vcpu);
 	int (*handle_exit)(struct kvm_vcpu *vcpu);
 	void (*skip_emulated_instruction)(struct kvm_vcpu *vcpu);
 	void (*set_interrupt_shadow)(struct kvm_vcpu *vcpu, int mask);
 	u32 (*get_interrupt_shadow)(struct kvm_vcpu *vcpu, int mask);
 	void (*patch_hypercall)(struct kvm_vcpu *vcpu,
 				unsigned char *hypercall_addr);
 	void (*set_irq)(struct kvm_vcpu *vcpu);
 	void (*set_nmi)(struct kvm_vcpu *vcpu);
 	void (*queue_exception)(struct kvm_vcpu *vcpu, unsigned nr,
 				bool has_error_code, u32 error_code,
 				bool reinject);
 	int (*interrupt_allowed)(struct kvm_vcpu *vcpu);
 	int (*nmi_allowed)(struct kvm_vcpu *vcpu);
 	bool (*get_nmi_mask)(struct kvm_vcpu *vcpu);
 	void (*set_nmi_mask)(struct kvm_vcpu *vcpu, bool masked);
 	void (*enable_nmi_window)(struct kvm_vcpu *vcpu);
 	void (*enable_irq_window)(struct kvm_vcpu *vcpu);
 	void (*update_cr8_intercept)(struct kvm_vcpu *vcpu, int tpr, int irr);
 	int (*set_tss_addr)(struct kvm *kvm, unsigned int addr);
 	int (*get_tdp_level)(void);
 	u64 (*get_mt_mask)(struct kvm_vcpu *vcpu, gfn_t gfn, bool is_mmio);
 	int (*get_lpage_level)(void);
 	bool (*rdtscp_supported)(void);
 	void (*set_supported_cpuid)(u32 func, struct kvm_cpuid_entry2 *entry);
 	const struct trace_print_flags *exit_reasons_str;
 };
 extern struct kvm_x86_ops *kvm_x86_ops;
 int kvm_mmu_module_init(void);
 void kvm_mmu_module_exit(void);
 void kvm_mmu_destroy(struct kvm_vcpu *vcpu);
 int kvm_mmu_create(struct kvm_vcpu *vcpu);
 int kvm_mmu_setup(struct kvm_vcpu *vcpu);
 void kvm_mmu_set_nonpresent_ptes(u64 trap_pte, u64 notrap_pte);
 void kvm_mmu_set_base_ptes(u64 base_pte);
 void kvm_mmu_set_mask_ptes(u64 user_mask, u64 accessed_mask,
 		u64 dirty_mask, u64 nx_mask, u64 x_mask);
 int kvm_mmu_reset_context(struct kvm_vcpu *vcpu);
 void kvm_mmu_slot_remove_write_access(struct kvm *kvm, int slot);
 void kvm_mmu_zap_all(struct kvm *kvm);
 unsigned int kvm_mmu_calculate_mmu_pages(struct kvm *kvm);
 void kvm_mmu_change_mmu_pages(struct kvm *kvm, unsigned int kvm_nr_mmu_pages);
 int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3);
 int emulator_write_phys(struct kvm_vcpu *vcpu, gpa_t gpa,
 			  const void *val, int bytes);
 int kvm_pv_mmu_op(struct kvm_vcpu *vcpu, unsigned long bytes,
 		  gpa_t addr, unsigned long *ret);
 u8 kvm_get_guest_memory_type(struct kvm_vcpu *vcpu, gfn_t gfn);
 extern bool tdp_enabled;
 enum emulation_result {
 	EMULATE_DONE,       /* no further processing */
 	EMULATE_DO_MMIO,      /* kvm_run filled with mmio request */
 	EMULATE_FAIL,         /* can't emulate this instruction */
 };
 #define EMULTYPE_NO_DECODE	    (1 << 0)
 #define EMULTYPE_TRAP_UD	    (1 << 1)
 #define EMULTYPE_SKIP		    (1 << 2)
 int emulate_instruction(struct kvm_vcpu *vcpu,
 			unsigned long cr2, u16 error_code, int emulation_type);
 void realmode_lgdt(struct kvm_vcpu *vcpu, u16 size, unsigned long address);
 void realmode_lidt(struct kvm_vcpu *vcpu, u16 size, unsigned long address);
 void kvm_enable_efer_bits(u64);
 int kvm_get_msr(struct kvm_vcpu *vcpu, u32 msr_index, u64 *data);
 int kvm_set_msr(struct kvm_vcpu *vcpu, u32 msr_index, u64 data);
 struct x86_emulate_ctxt;
 int kvm_fast_pio_out(struct kvm_vcpu *vcpu, int size, unsigned short port);
 void kvm_emulate_cpuid(struct kvm_vcpu *vcpu);
 int kvm_emulate_halt(struct kvm_vcpu *vcpu);
 int emulate_invlpg(struct kvm_vcpu *vcpu, gva_t address);
 int emulate_clts(struct kvm_vcpu *vcpu);
 void kvm_get_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var, int seg);
 int kvm_load_segment_descriptor(struct kvm_vcpu *vcpu, u16 selector, int seg);
 int kvm_task_switch(struct kvm_vcpu *vcpu, u16 tss_selector, int reason,
 		    bool has_error_code, u32 error_code);
 int kvm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0);
 int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3);
 int kvm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4);
 void kvm_set_cr8(struct kvm_vcpu *vcpu, unsigned long cr8);
 int kvm_set_dr(struct kvm_vcpu *vcpu, int dr, unsigned long val);
 int kvm_get_dr(struct kvm_vcpu *vcpu, int dr, unsigned long *val);
 unsigned long kvm_get_cr8(struct kvm_vcpu *vcpu);
 void kvm_lmsw(struct kvm_vcpu *vcpu, unsigned long msw);
 void kvm_get_cs_db_l_bits(struct kvm_vcpu *vcpu, int *db, int *l);
 int kvm_set_xcr(struct kvm_vcpu *vcpu, u32 index, u64 xcr);
 int kvm_get_msr_common(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata);
 int kvm_set_msr_common(struct kvm_vcpu *vcpu, u32 msr, u64 data);
 unsigned long kvm_get_rflags(struct kvm_vcpu *vcpu);
 void kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags);
 void kvm_queue_exception(struct kvm_vcpu *vcpu, unsigned nr);
 void kvm_queue_exception_e(struct kvm_vcpu *vcpu, unsigned nr, u32 error_code);
 void kvm_requeue_exception(struct kvm_vcpu *vcpu, unsigned nr);
 void kvm_requeue_exception_e(struct kvm_vcpu *vcpu, unsigned nr, u32 error_code);
 void kvm_inject_page_fault(struct kvm_vcpu *vcpu, unsigned long cr2,
 			   u32 error_code);
 bool kvm_require_cpl(struct kvm_vcpu *vcpu, int required_cpl);
 int kvm_pic_set_irq(void *opaque, int irq, int level);
 void kvm_inject_nmi(struct kvm_vcpu *vcpu);
 int fx_init(struct kvm_vcpu *vcpu);
 void kvm_mmu_flush_tlb(struct kvm_vcpu *vcpu);
 void kvm_mmu_pte_write(struct kvm_vcpu *vcpu, gpa_t gpa,
 		       const u8 *new, int bytes,
 		       bool guest_initiated);
 int kvm_mmu_unprotect_page_virt(struct kvm_vcpu *vcpu, gva_t gva);
 void __kvm_mmu_free_some_pages(struct kvm_vcpu *vcpu);
 int kvm_mmu_load(struct kvm_vcpu *vcpu);
 void kvm_mmu_unload(struct kvm_vcpu *vcpu);
 void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu);
 gpa_t kvm_mmu_gva_to_gpa_read(struct kvm_vcpu *vcpu, gva_t gva, u32 *error);
 gpa_t kvm_mmu_gva_to_gpa_fetch(struct kvm_vcpu *vcpu, gva_t gva, u32 *error);
 gpa_t kvm_mmu_gva_to_gpa_write(struct kvm_vcpu *vcpu, gva_t gva, u32 *error);
 gpa_t kvm_mmu_gva_to_gpa_system(struct kvm_vcpu *vcpu, gva_t gva, u32 *error);
 int kvm_emulate_hypercall(struct kvm_vcpu *vcpu);
 int kvm_fix_hypercall(struct kvm_vcpu *vcpu);
 int kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gva_t gva, u32 error_code);
 void kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva);
 void kvm_enable_tdp(void);
 void kvm_disable_tdp(void);
 int complete_pio(struct kvm_vcpu *vcpu);
 bool kvm_check_iopl(struct kvm_vcpu *vcpu);
-struct kvm_memory_slot *gfn_to_memslot_unaliased(struct kvm *kvm, gfn_t gfn);
 static inline struct kvm_mmu_page *page_header(hpa_t shadow_page)
 {
 	struct page *page = pfn_to_page(shadow_page >> PAGE_SHIFT);
 	return (struct kvm_mmu_page *)page_private(page);
 }
 static inline u16 kvm_read_fs(void)
 {
 	u16 seg;
 	asm("mov %%fs, %0" : "=g"(seg));
 	return seg;
 }
 static inline u16 kvm_read_gs(void)
 {
 	u16 seg;
 	asm("mov %%gs, %0" : "=g"(seg));
 	return seg;
 }
 static inline u16 kvm_read_ldt(void)
 {
 	u16 ldt;
 	asm("sldt %0" : "=g"(ldt));
 	return ldt;
 }
 static inline void kvm_load_fs(u16 sel)
 {
 	asm("mov %0, %%fs" : : "rm"(sel));
 }
 static inline void kvm_load_gs(u16 sel)
 {
 	asm("mov %0, %%gs" : : "rm"(sel));
 }
 static inline void kvm_load_ldt(u16 sel)
 {
 	asm("lldt %0" : : "rm"(sel));
 }
 #ifdef CONFIG_X86_64
 static inline unsigned long read_msr(unsigned long msr)
 {
 	u64 value;
 	rdmsrl(msr, value);
 	return value;
 }
 #endif
 static inline u32 get_rdx_init_val(void)
 {
 	return 0x600; /* P6 family */
 }
 static inline void kvm_inject_gp(struct kvm_vcpu *vcpu, u32 error_code)
 {
 	kvm_queue_exception_e(vcpu, GP_VECTOR, error_code);
 }
 #define TSS_IOPB_BASE_OFFSET 0x66
 #define TSS_BASE_SIZE 0x68
 #define TSS_IOPB_SIZE (65536 / 8)
 #define TSS_REDIRECTION_SIZE (256 / 8)
 #define RMODE_TSS_SIZE							\
 	(TSS_BASE_SIZE + TSS_REDIRECTION_SIZE + TSS_IOPB_SIZE + 1)
 enum {
 	TASK_SWITCH_CALL = 0,
 	TASK_SWITCH_IRET = 1,
 	TASK_SWITCH_JMP = 2,
 	TASK_SWITCH_GATE = 3,
 };
 #define HF_GIF_MASK		(1 << 0)
 #define HF_HIF_MASK		(1 << 1)
 #define HF_VINTR_MASK		(1 << 2)
 #define HF_NMI_MASK		(1 << 3)
 #define HF_IRET_MASK		(1 << 4)
 /*
  * Hardware virtualization extension instructions may fault if a
  * reboot turns off virtualization while processes are running.
  * Trap the fault and ignore the instruction if that happens.
  */
 asmlinkage void kvm_handle_fault_on_reboot(void);
 #define __kvm_handle_fault_on_reboot(insn) \
 	"666: " insn "\n\t" \
 	".pushsection .fixup, \"ax\" \n" \
 	"667: \n\t" \
 	__ASM_SIZE(push) " $666b \n\t"	      \
 	"jmp kvm_handle_fault_on_reboot \n\t" \
 	".popsection \n\t" \
 	".pushsection __ex_table, \"a\" \n\t" \
 	_ASM_PTR " 666b, 667b \n\t" \
 	".popsection"
 #define KVM_ARCH_WANT_MMU_NOTIFIER
 int kvm_unmap_hva(struct kvm *kvm, unsigned long hva);
 int kvm_age_hva(struct kvm *kvm, unsigned long hva);
 void kvm_set_spte_hva(struct kvm *kvm, unsigned long hva, pte_t pte);
 int cpuid_maxphyaddr(struct kvm_vcpu *vcpu);
 int kvm_cpu_has_interrupt(struct kvm_vcpu *vcpu);
 int kvm_arch_interrupt_allowed(struct kvm_vcpu *vcpu);
 int kvm_cpu_get_interrupt(struct kvm_vcpu *v);
 void kvm_define_shared_msr(unsigned index, u32 msr);
 void kvm_set_shared_msr(unsigned index, u64 val, u64 mask);
 bool kvm_is_linear_rip(struct kvm_vcpu *vcpu, unsigned long linear_rip);
 #endif /* _ASM_X86_KVM_HOST_H */

arch/x86/kvm/mmu.c

Diff comments View file @ a1f4d39

 /*
  * Kernel-based Virtual Machine driver for Linux
  *
  * This module enables machines with Intel VT-x extensions to run virtual
  * machines without emulation or binary translation.
  *
  * MMU support
  *
  * Copyright (C) 2006 Qumranet, Inc.
  * Copyright 2010 Red Hat, Inc. and/or its affilates.
  *
  * Authors:
  *   Yaniv Kamay  <yaniv@qumranet.com>
  *   Avi Kivity   <avi@qumranet.com>
  *
  * This work is licensed under the terms of the GNU GPL, version 2.  See
  * the COPYING file in the top-level directory.
  *
  */
 #include "mmu.h"
 #include "x86.h"
 #include "kvm_cache_regs.h"
 #include <linux/kvm_host.h>
 #include <linux/types.h>
 #include <linux/string.h>
 #include <linux/mm.h>
 #include <linux/highmem.h>
 #include <linux/module.h>
 #include <linux/swap.h>
 #include <linux/hugetlb.h>
 #include <linux/compiler.h>
 #include <linux/srcu.h>
 #include <linux/slab.h>
 #include <linux/uaccess.h>
 #include <asm/page.h>
 #include <asm/cmpxchg.h>
 #include <asm/io.h>
 #include <asm/vmx.h>
 /*
  * When setting this variable to true it enables Two-Dimensional-Paging
  * where the hardware walks 2 page tables:
  * 1. the guest-virtual to guest-physical
  * 2. while doing 1. it walks guest-physical to host-physical
  * If the hardware supports that we don't need to do shadow paging.
  */
 bool tdp_enabled = false;
 #undef MMU_DEBUG
 #undef AUDIT
 #ifdef AUDIT
 static void kvm_mmu_audit(struct kvm_vcpu *vcpu, const char *msg);
 #else
 static void kvm_mmu_audit(struct kvm_vcpu *vcpu, const char *msg) {}
 #endif
 #ifdef MMU_DEBUG
 #define pgprintk(x...) do { if (dbg) printk(x); } while (0)
 #define rmap_printk(x...) do { if (dbg) printk(x); } while (0)
 #else
 #define pgprintk(x...) do { } while (0)
 #define rmap_printk(x...) do { } while (0)
 #endif
 #if defined(MMU_DEBUG) || defined(AUDIT)
 static int dbg = 0;
 module_param(dbg, bool, 0644);
 #endif
 static int oos_shadow = 1;
 module_param(oos_shadow, bool, 0644);
 #ifndef MMU_DEBUG
 #define ASSERT(x) do { } while (0)
 #else
 #define ASSERT(x)							\
 	if (!(x)) {							\
 		printk(KERN_WARNING "assertion failed %s:%d: %s\n",	\
 		       __FILE__, __LINE__, #x);				\
 	}
 #endif
 #define PT_FIRST_AVAIL_BITS_SHIFT 9
 #define PT64_SECOND_AVAIL_BITS_SHIFT 52
 #define VALID_PAGE(x) ((x) != INVALID_PAGE)
 #define PT64_LEVEL_BITS 9
 #define PT64_LEVEL_SHIFT(level) \
 		(PAGE_SHIFT + (level - 1) * PT64_LEVEL_BITS)
 #define PT64_LEVEL_MASK(level) \
 		(((1ULL << PT64_LEVEL_BITS) - 1) << PT64_LEVEL_SHIFT(level))
 #define PT64_INDEX(address, level)\
 	(((address) >> PT64_LEVEL_SHIFT(level)) & ((1 << PT64_LEVEL_BITS) - 1))
 #define PT32_LEVEL_BITS 10
 #define PT32_LEVEL_SHIFT(level) \
 		(PAGE_SHIFT + (level - 1) * PT32_LEVEL_BITS)
 #define PT32_LEVEL_MASK(level) \
 		(((1ULL << PT32_LEVEL_BITS) - 1) << PT32_LEVEL_SHIFT(level))
 #define PT32_LVL_OFFSET_MASK(level) \
 	(PT32_BASE_ADDR_MASK & ((1ULL << (PAGE_SHIFT + (((level) - 1) \
 						* PT32_LEVEL_BITS))) - 1))
 #define PT32_INDEX(address, level)\
 	(((address) >> PT32_LEVEL_SHIFT(level)) & ((1 << PT32_LEVEL_BITS) - 1))
 #define PT64_BASE_ADDR_MASK (((1ULL << 52) - 1) & ~(u64)(PAGE_SIZE-1))
 #define PT64_DIR_BASE_ADDR_MASK \
 	(PT64_BASE_ADDR_MASK & ~((1ULL << (PAGE_SHIFT + PT64_LEVEL_BITS)) - 1))
 #define PT64_LVL_ADDR_MASK(level) \
 	(PT64_BASE_ADDR_MASK & ~((1ULL << (PAGE_SHIFT + (((level) - 1) \
 						* PT64_LEVEL_BITS))) - 1))
 #define PT64_LVL_OFFSET_MASK(level) \
 	(PT64_BASE_ADDR_MASK & ((1ULL << (PAGE_SHIFT + (((level) - 1) \
 						* PT64_LEVEL_BITS))) - 1))
 #define PT32_BASE_ADDR_MASK PAGE_MASK
 #define PT32_DIR_BASE_ADDR_MASK \
 	(PAGE_MASK & ~((1ULL << (PAGE_SHIFT + PT32_LEVEL_BITS)) - 1))
 #define PT32_LVL_ADDR_MASK(level) \
 	(PAGE_MASK & ~((1ULL << (PAGE_SHIFT + (((level) - 1) \
 					    * PT32_LEVEL_BITS))) - 1))
 #define PT64_PERM_MASK (PT_PRESENT_MASK | PT_WRITABLE_MASK | PT_USER_MASK \
 			| PT64_NX_MASK)
 #define RMAP_EXT 4
 #define ACC_EXEC_MASK    1
 #define ACC_WRITE_MASK   PT_WRITABLE_MASK
 #define ACC_USER_MASK    PT_USER_MASK
 #define ACC_ALL          (ACC_EXEC_MASK | ACC_WRITE_MASK | ACC_USER_MASK)
 #include <trace/events/kvm.h>
 #define CREATE_TRACE_POINTS
 #include "mmutrace.h"
 #define SPTE_HOST_WRITEABLE (1ULL << PT_FIRST_AVAIL_BITS_SHIFT)
 #define SHADOW_PT_INDEX(addr, level) PT64_INDEX(addr, level)
 struct kvm_rmap_desc {
 	u64 *sptes[RMAP_EXT];
 	struct kvm_rmap_desc *more;
 };
 struct kvm_shadow_walk_iterator {
 	u64 addr;
 	hpa_t shadow_addr;
 	int level;
 	u64 *sptep;
 	unsigned index;
 };
 #define for_each_shadow_entry(_vcpu, _addr, _walker)    \
 	for (shadow_walk_init(&(_walker), _vcpu, _addr);	\
 	     shadow_walk_okay(&(_walker));			\
 	     shadow_walk_next(&(_walker)))
 typedef void (*mmu_parent_walk_fn) (struct kvm_mmu_page *sp, u64 *spte);
 static struct kmem_cache *pte_chain_cache;
 static struct kmem_cache *rmap_desc_cache;
 static struct kmem_cache *mmu_page_header_cache;
 static u64 __read_mostly shadow_trap_nonpresent_pte;
 static u64 __read_mostly shadow_notrap_nonpresent_pte;
 static u64 __read_mostly shadow_base_present_pte;
 static u64 __read_mostly shadow_nx_mask;
 static u64 __read_mostly shadow_x_mask;	/* mutual exclusive with nx_mask */
 static u64 __read_mostly shadow_user_mask;
 static u64 __read_mostly shadow_accessed_mask;
 static u64 __read_mostly shadow_dirty_mask;
 static inline u64 rsvd_bits(int s, int e)
 {
 	return ((1ULL << (e - s + 1)) - 1) << s;
 }
 void kvm_mmu_set_nonpresent_ptes(u64 trap_pte, u64 notrap_pte)
 {
 	shadow_trap_nonpresent_pte = trap_pte;
 	shadow_notrap_nonpresent_pte = notrap_pte;
 }
 EXPORT_SYMBOL_GPL(kvm_mmu_set_nonpresent_ptes);
 void kvm_mmu_set_base_ptes(u64 base_pte)
 {
 	shadow_base_present_pte = base_pte;
 }
 EXPORT_SYMBOL_GPL(kvm_mmu_set_base_ptes);
 void kvm_mmu_set_mask_ptes(u64 user_mask, u64 accessed_mask,
 		u64 dirty_mask, u64 nx_mask, u64 x_mask)
 {
 	shadow_user_mask = user_mask;
 	shadow_accessed_mask = accessed_mask;
 	shadow_dirty_mask = dirty_mask;
 	shadow_nx_mask = nx_mask;
 	shadow_x_mask = x_mask;
 }
 EXPORT_SYMBOL_GPL(kvm_mmu_set_mask_ptes);
 static bool is_write_protection(struct kvm_vcpu *vcpu)
 {
 	return kvm_read_cr0_bits(vcpu, X86_CR0_WP);
 }
 static int is_cpuid_PSE36(void)
 {
 	return 1;
 }
 static int is_nx(struct kvm_vcpu *vcpu)
 {
 	return vcpu->arch.efer & EFER_NX;
 }
 static int is_shadow_present_pte(u64 pte)
 {
 	return pte != shadow_trap_nonpresent_pte
 		&& pte != shadow_notrap_nonpresent_pte;
 }
 static int is_large_pte(u64 pte)
 {
 	return pte & PT_PAGE_SIZE_MASK;
 }
 static int is_writable_pte(unsigned long pte)
 {
 	return pte & PT_WRITABLE_MASK;
 }
 static int is_dirty_gpte(unsigned long pte)
 {
 	return pte & PT_DIRTY_MASK;
 }
 static int is_rmap_spte(u64 pte)
 {
 	return is_shadow_present_pte(pte);
 }
 static int is_last_spte(u64 pte, int level)
 {
 	if (level == PT_PAGE_TABLE_LEVEL)
 		return 1;
 	if (is_large_pte(pte))
 		return 1;
 	return 0;
 }
 static pfn_t spte_to_pfn(u64 pte)
 {
 	return (pte & PT64_BASE_ADDR_MASK) >> PAGE_SHIFT;
 }
 static gfn_t pse36_gfn_delta(u32 gpte)
 {
 	int shift = 32 - PT32_DIR_PSE36_SHIFT - PAGE_SHIFT;
 	return (gpte & PT32_DIR_PSE36_MASK) << shift;
 }
 static void __set_spte(u64 *sptep, u64 spte)
 {
 #ifdef CONFIG_X86_64
 	set_64bit((unsigned long *)sptep, spte);
 #else
 	set_64bit((unsigned long long *)sptep, spte);
 #endif
 }
 static int mmu_topup_memory_cache(struct kvm_mmu_memory_cache *cache,
 				  struct kmem_cache *base_cache, int min)
 {
 	void *obj;
 	if (cache->nobjs >= min)
 		return 0;
 	while (cache->nobjs < ARRAY_SIZE(cache->objects)) {
 		obj = kmem_cache_zalloc(base_cache, GFP_KERNEL);
 		if (!obj)
 			return -ENOMEM;
 		cache->objects[cache->nobjs++] = obj;
 	}
 	return 0;
 }
 static void mmu_free_memory_cache(struct kvm_mmu_memory_cache *mc,
 				  struct kmem_cache *cache)
 {
 	while (mc->nobjs)
 		kmem_cache_free(cache, mc->objects[--mc->nobjs]);
 }
 static int mmu_topup_memory_cache_page(struct kvm_mmu_memory_cache *cache,
 				       int min)
 {
 	struct page *page;
 	if (cache->nobjs >= min)
 		return 0;
 	while (cache->nobjs < ARRAY_SIZE(cache->objects)) {
 		page = alloc_page(GFP_KERNEL);
 		if (!page)
 			return -ENOMEM;
 		cache->objects[cache->nobjs++] = page_address(page);
 	}
 	return 0;
 }
 static void mmu_free_memory_cache_page(struct kvm_mmu_memory_cache *mc)
 {
 	while (mc->nobjs)
 		free_page((unsigned long)mc->objects[--mc->nobjs]);
 }
 static int mmu_topup_memory_caches(struct kvm_vcpu *vcpu)
 {
 	int r;
 	r = mmu_topup_memory_cache(&vcpu->arch.mmu_pte_chain_cache,
 				   pte_chain_cache, 4);
 	if (r)
 		goto out;
 	r = mmu_topup_memory_cache(&vcpu->arch.mmu_rmap_desc_cache,
 				   rmap_desc_cache, 4);
 	if (r)
 		goto out;
 	r = mmu_topup_memory_cache_page(&vcpu->arch.mmu_page_cache, 8);
 	if (r)
 		goto out;
 	r = mmu_topup_memory_cache(&vcpu->arch.mmu_page_header_cache,
 				   mmu_page_header_cache, 4);
 out:
 	return r;
 }
 static void mmu_free_memory_caches(struct kvm_vcpu *vcpu)
 {
 	mmu_free_memory_cache(&vcpu->arch.mmu_pte_chain_cache, pte_chain_cache);
 	mmu_free_memory_cache(&vcpu->arch.mmu_rmap_desc_cache, rmap_desc_cache);
 	mmu_free_memory_cache_page(&vcpu->arch.mmu_page_cache);
 	mmu_free_memory_cache(&vcpu->arch.mmu_page_header_cache,
 				mmu_page_header_cache);
 }
 static void *mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc,
 				    size_t size)
 {
 	void *p;
 	BUG_ON(!mc->nobjs);
 	p = mc->objects[--mc->nobjs];
 	return p;
 }
 static struct kvm_pte_chain *mmu_alloc_pte_chain(struct kvm_vcpu *vcpu)
 {
 	return mmu_memory_cache_alloc(&vcpu->arch.mmu_pte_chain_cache,
 				      sizeof(struct kvm_pte_chain));
 }
 static void mmu_free_pte_chain(struct kvm_pte_chain *pc)
 {
 	kmem_cache_free(pte_chain_cache, pc);
 }
 static struct kvm_rmap_desc *mmu_alloc_rmap_desc(struct kvm_vcpu *vcpu)
 {
 	return mmu_memory_cache_alloc(&vcpu->arch.mmu_rmap_desc_cache,
 				      sizeof(struct kvm_rmap_desc));
 }
 static void mmu_free_rmap_desc(struct kvm_rmap_desc *rd)
 {
 	kmem_cache_free(rmap_desc_cache, rd);
 }
 static gfn_t kvm_mmu_page_get_gfn(struct kvm_mmu_page *sp, int index)
 {
 	if (!sp->role.direct)
 		return sp->gfns[index];
 	return sp->gfn + (index << ((sp->role.level - 1) * PT64_LEVEL_BITS));
 }
 static void kvm_mmu_page_set_gfn(struct kvm_mmu_page *sp, int index, gfn_t gfn)
 {
 	if (sp->role.direct)
 		BUG_ON(gfn != kvm_mmu_page_get_gfn(sp, index));
 	else
 		sp->gfns[index] = gfn;
 }
 /*
  * Return the pointer to the largepage write count for a given
  * gfn, handling slots that are not large page aligned.
  */
 static int *slot_largepage_idx(gfn_t gfn,
 			       struct kvm_memory_slot *slot,
 			       int level)
 {
 	unsigned long idx;
 	idx = (gfn / KVM_PAGES_PER_HPAGE(level)) -
 	      (slot->base_gfn / KVM_PAGES_PER_HPAGE(level));
 	return &slot->lpage_info[level - 2][idx].write_count;
 }
 static void account_shadowed(struct kvm *kvm, gfn_t gfn)
 {
 	struct kvm_memory_slot *slot;
 	int *write_count;
 	int i;
-	gfn = unalias_gfn(kvm, gfn);
+	slot = gfn_to_memslot(kvm, gfn);
-	slot = gfn_to_memslot_unaliased(kvm, gfn);
 	for (i = PT_DIRECTORY_LEVEL;
 	     i < PT_PAGE_TABLE_LEVEL + KVM_NR_PAGE_SIZES; ++i) {
 		write_count   = slot_largepage_idx(gfn, slot, i);
 		*write_count += 1;
 	}
 }
 static void unaccount_shadowed(struct kvm *kvm, gfn_t gfn)
 {
 	struct kvm_memory_slot *slot;
 	int *write_count;
 	int i;
-	gfn = unalias_gfn(kvm, gfn);
+	slot = gfn_to_memslot(kvm, gfn);
-	slot = gfn_to_memslot_unaliased(kvm, gfn);
 	for (i = PT_DIRECTORY_LEVEL;
 	     i < PT_PAGE_TABLE_LEVEL + KVM_NR_PAGE_SIZES; ++i) {
 		write_count   = slot_largepage_idx(gfn, slot, i);
 		*write_count -= 1;
 		WARN_ON(*write_count < 0);
 	}
 }
 static int has_wrprotected_page(struct kvm *kvm,
 				gfn_t gfn,
 				int level)
 {
 	struct kvm_memory_slot *slot;
 	int *largepage_idx;
-	gfn = unalias_gfn(kvm, gfn);
+	slot = gfn_to_memslot(kvm, gfn);
-	slot = gfn_to_memslot_unaliased(kvm, gfn);
 	if (slot) {
 		largepage_idx = slot_largepage_idx(gfn, slot, level);
 		return *largepage_idx;
 	}
 	return 1;
 }
 static int host_mapping_level(struct kvm *kvm, gfn_t gfn)
 {
 	unsigned long page_size;
 	int i, ret = 0;
 	page_size = kvm_host_page_size(kvm, gfn);
 	for (i = PT_PAGE_TABLE_LEVEL;
 	     i < (PT_PAGE_TABLE_LEVEL + KVM_NR_PAGE_SIZES); ++i) {
 		if (page_size >= KVM_HPAGE_SIZE(i))
 			ret = i;
 		else
 			break;
 	}
 	return ret;
 }
 static int mapping_level(struct kvm_vcpu *vcpu, gfn_t large_gfn)
 {
 	struct kvm_memory_slot *slot;
 	int host_level, level, max_level;
 	slot = gfn_to_memslot(vcpu->kvm, large_gfn);
 	if (slot && slot->dirty_bitmap)
 		return PT_PAGE_TABLE_LEVEL;
 	host_level = host_mapping_level(vcpu->kvm, large_gfn);
 	if (host_level == PT_PAGE_TABLE_LEVEL)
 		return host_level;
 	max_level = kvm_x86_ops->get_lpage_level() < host_level ?
 		kvm_x86_ops->get_lpage_level() : host_level;
 	for (level = PT_DIRECTORY_LEVEL; level <= max_level; ++level)
 		if (has_wrprotected_page(vcpu->kvm, large_gfn, level))
 			break;
 	return level - 1;
 }
 /*
  * Take gfn and return the reverse mapping to it.
- * Note: gfn must be unaliased before this function get called
  */
 static unsigned long *gfn_to_rmap(struct kvm *kvm, gfn_t gfn, int level)
 {
 	struct kvm_memory_slot *slot;
 	unsigned long idx;
 	slot = gfn_to_memslot(kvm, gfn);
 	if (likely(level == PT_PAGE_TABLE_LEVEL))
 		return &slot->rmap[gfn - slot->base_gfn];
 	idx = (gfn / KVM_PAGES_PER_HPAGE(level)) -
 		(slot->base_gfn / KVM_PAGES_PER_HPAGE(level));
 	return &slot->lpage_info[level - 2][idx].rmap_pde;
 }
 /*
  * Reverse mapping data structures:
  *
  * If rmapp bit zero is zero, then rmapp point to the shadw page table entry
  * that points to page_address(page).
  *
  * If rmapp bit zero is one, (then rmap & ~1) points to a struct kvm_rmap_desc
  * containing more mappings.
  *
  * Returns the number of rmap entries before the spte was added or zero if
  * the spte was not added.
  *
  */
 static int rmap_add(struct kvm_vcpu *vcpu, u64 *spte, gfn_t gfn)
 {
 	struct kvm_mmu_page *sp;
 	struct kvm_rmap_desc *desc;
 	unsigned long *rmapp;
 	int i, count = 0;
 	if (!is_rmap_spte(*spte))
 		return count;
-	gfn = unalias_gfn(vcpu->kvm, gfn);
 	sp = page_header(__pa(spte));
 	kvm_mmu_page_set_gfn(sp, spte - sp->spt, gfn);
 	rmapp = gfn_to_rmap(vcpu->kvm, gfn, sp->role.level);
 	if (!*rmapp) {
 		rmap_printk("rmap_add: %p %llx 0->1\n", spte, *spte);
 		*rmapp = (unsigned long)spte;
 	} else if (!(*rmapp & 1)) {
 		rmap_printk("rmap_add: %p %llx 1->many\n", spte, *spte);
 		desc = mmu_alloc_rmap_desc(vcpu);
 		desc->sptes[0] = (u64 *)*rmapp;
 		desc->sptes[1] = spte;
 		*rmapp = (unsigned long)desc | 1;
 	} else {
 		rmap_printk("rmap_add: %p %llx many->many\n", spte, *spte);
 		desc = (struct kvm_rmap_desc *)(*rmapp & ~1ul);
 		while (desc->sptes[RMAP_EXT-1] && desc->more) {
 			desc = desc->more;
 			count += RMAP_EXT;
 		}
 		if (desc->sptes[RMAP_EXT-1]) {
 			desc->more = mmu_alloc_rmap_desc(vcpu);
 			desc = desc->more;
 		}
 		for (i = 0; desc->sptes[i]; ++i)
 			;
 		desc->sptes[i] = spte;
 	}
 	return count;
 }
 static void rmap_desc_remove_entry(unsigned long *rmapp,
 				   struct kvm_rmap_desc *desc,
 				   int i,
 				   struct kvm_rmap_desc *prev_desc)
 {
 	int j;
 	for (j = RMAP_EXT - 1; !desc->sptes[j] && j > i; --j)
 		;
 	desc->sptes[i] = desc->sptes[j];
 	desc->sptes[j] = NULL;
 	if (j != 0)
 		return;
 	if (!prev_desc && !desc->more)
 		*rmapp = (unsigned long)desc->sptes[0];
 	else
 		if (prev_desc)
 			prev_desc->more = desc->more;
 		else
 			*rmapp = (unsigned long)desc->more | 1;
 	mmu_free_rmap_desc(desc);
 }
 static void rmap_remove(struct kvm *kvm, u64 *spte)
 {
 	struct kvm_rmap_desc *desc;
 	struct kvm_rmap_desc *prev_desc;
 	struct kvm_mmu_page *sp;
 	pfn_t pfn;
 	gfn_t gfn;
 	unsigned long *rmapp;
 	int i;
 	if (!is_rmap_spte(*spte))
 		return;
 	sp = page_header(__pa(spte));
 	pfn = spte_to_pfn(*spte);
 	if (*spte & shadow_accessed_mask)
 		kvm_set_pfn_accessed(pfn);
 	if (is_writable_pte(*spte))
 		kvm_set_pfn_dirty(pfn);
 	gfn = kvm_mmu_page_get_gfn(sp, spte - sp->spt);
 	rmapp = gfn_to_rmap(kvm, gfn, sp->role.level);
 	if (!*rmapp) {
 		printk(KERN_ERR "rmap_remove: %p %llx 0->BUG\n", spte, *spte);
 		BUG();
 	} else if (!(*rmapp & 1)) {
 		rmap_printk("rmap_remove:  %p %llx 1->0\n", spte, *spte);
 		if ((u64 *)*rmapp != spte) {
 			printk(KERN_ERR "rmap_remove:  %p %llx 1->BUG\n",
 			       spte, *spte);
 			BUG();
 		}
 		*rmapp = 0;
 	} else {
 		rmap_printk("rmap_remove:  %p %llx many->many\n", spte, *spte);
 		desc = (struct kvm_rmap_desc *)(*rmapp & ~1ul);
 		prev_desc = NULL;
 		while (desc) {
 			for (i = 0; i < RMAP_EXT && desc->sptes[i]; ++i)
 				if (desc->sptes[i] == spte) {
 					rmap_desc_remove_entry(rmapp,
 							       desc, i,
 							       prev_desc);
 					return;
 				}
 			prev_desc = desc;
 			desc = desc->more;
 		}
 		pr_err("rmap_remove: %p %llx many->many\n", spte, *spte);
 		BUG();
 	}
 }
 static u64 *rmap_next(struct kvm *kvm, unsigned long *rmapp, u64 *spte)
 {
 	struct kvm_rmap_desc *desc;
 	u64 *prev_spte;
 	int i;
 	if (!*rmapp)
 		return NULL;
 	else if (!(*rmapp & 1)) {
 		if (!spte)
 			return (u64 *)*rmapp;
 		return NULL;
 	}
 	desc = (struct kvm_rmap_desc *)(*rmapp & ~1ul);
 	prev_spte = NULL;
 	while (desc) {
 		for (i = 0; i < RMAP_EXT && desc->sptes[i]; ++i) {
 			if (prev_spte == spte)
 				return desc->sptes[i];
 			prev_spte = desc->sptes[i];
 		}
 		desc = desc->more;
 	}
 	return NULL;
 }
 static int rmap_write_protect(struct kvm *kvm, u64 gfn)
 {
 	unsigned long *rmapp;
 	u64 *spte;
 	int i, write_protected = 0;
-	gfn = unalias_gfn(kvm, gfn);
 	rmapp = gfn_to_rmap(kvm, gfn, PT_PAGE_TABLE_LEVEL);
 	spte = rmap_next(kvm, rmapp, NULL);
 	while (spte) {
 		BUG_ON(!spte);
 		BUG_ON(!(*spte & PT_PRESENT_MASK));
 		rmap_printk("rmap_write_protect: spte %p %llx\n", spte, *spte);
 		if (is_writable_pte(*spte)) {
 			__set_spte(spte, *spte & ~PT_WRITABLE_MASK);
 			write_protected = 1;
 		}
 		spte = rmap_next(kvm, rmapp, spte);
 	}
 	if (write_protected) {
 		pfn_t pfn;
 		spte = rmap_next(kvm, rmapp, NULL);
 		pfn = spte_to_pfn(*spte);
 		kvm_set_pfn_dirty(pfn);
 	}
 	/* check for huge page mappings */
 	for (i = PT_DIRECTORY_LEVEL;
 	     i < PT_PAGE_TABLE_LEVEL + KVM_NR_PAGE_SIZES; ++i) {
 		rmapp = gfn_to_rmap(kvm, gfn, i);
 		spte = rmap_next(kvm, rmapp, NULL);
 		while (spte) {
 			BUG_ON(!spte);
 			BUG_ON(!(*spte & PT_PRESENT_MASK));
 			BUG_ON((*spte & (PT_PAGE_SIZE_MASK|PT_PRESENT_MASK)) != (PT_PAGE_SIZE_MASK|PT_PRESENT_MASK));
 			pgprintk("rmap_write_protect(large): spte %p %llx %lld\n", spte, *spte, gfn);
 			if (is_writable_pte(*spte)) {
 				rmap_remove(kvm, spte);
 				--kvm->stat.lpages;
 				__set_spte(spte, shadow_trap_nonpresent_pte);
 				spte = NULL;
 				write_protected = 1;
 			}
 			spte = rmap_next(kvm, rmapp, spte);
 		}
 	}
 	return write_protected;
 }
 static int kvm_unmap_rmapp(struct kvm *kvm, unsigned long *rmapp,
 			   unsigned long data)
 {
 	u64 *spte;
 	int need_tlb_flush = 0;
 	while ((spte = rmap_next(kvm, rmapp, NULL))) {
 		BUG_ON(!(*spte & PT_PRESENT_MASK));
 		rmap_printk("kvm_rmap_unmap_hva: spte %p %llx\n", spte, *spte);
 		rmap_remove(kvm, spte);
 		__set_spte(spte, shadow_trap_nonpresent_pte);
 		need_tlb_flush = 1;
 	}
 	return need_tlb_flush;
 }
 static int kvm_set_pte_rmapp(struct kvm *kvm, unsigned long *rmapp,
 			     unsigned long data)
 {
 	int need_flush = 0;
 	u64 *spte, new_spte;
 	pte_t *ptep = (pte_t *)data;
 	pfn_t new_pfn;
 	WARN_ON(pte_huge(*ptep));
 	new_pfn = pte_pfn(*ptep);
 	spte = rmap_next(kvm, rmapp, NULL);
 	while (spte) {
 		BUG_ON(!is_shadow_present_pte(*spte));
 		rmap_printk("kvm_set_pte_rmapp: spte %p %llx\n", spte, *spte);
 		need_flush = 1;
 		if (pte_write(*ptep)) {
 			rmap_remove(kvm, spte);
 			__set_spte(spte, shadow_trap_nonpresent_pte);
 			spte = rmap_next(kvm, rmapp, NULL);
 		} else {
 			new_spte = *spte &~ (PT64_BASE_ADDR_MASK);
 			new_spte |= (u64)new_pfn << PAGE_SHIFT;
 			new_spte &= ~PT_WRITABLE_MASK;
 			new_spte &= ~SPTE_HOST_WRITEABLE;
 			if (is_writable_pte(*spte))
 				kvm_set_pfn_dirty(spte_to_pfn(*spte));
 			__set_spte(spte, new_spte);
 			spte = rmap_next(kvm, rmapp, spte);
 		}
 	}
 	if (need_flush)
 		kvm_flush_remote_tlbs(kvm);
 	return 0;
 }
 static int kvm_handle_hva(struct kvm *kvm, unsigned long hva,
 			  unsigned long data,
 			  int (*handler)(struct kvm *kvm, unsigned long *rmapp,
 					 unsigned long data))
 {
 	int i, j;
 	int ret;
 	int retval = 0;
 	struct kvm_memslots *slots;
 	slots = kvm_memslots(kvm);
 	for (i = 0; i < slots->nmemslots; i++) {
 		struct kvm_memory_slot *memslot = &slots->memslots[i];
 		unsigned long start = memslot->userspace_addr;
 		unsigned long end;
 		end = start + (memslot->npages << PAGE_SHIFT);
 		if (hva >= start && hva < end) {
 			gfn_t gfn_offset = (hva - start) >> PAGE_SHIFT;
 			ret = handler(kvm, &memslot->rmap[gfn_offset], data);
 			for (j = 0; j < KVM_NR_PAGE_SIZES - 1; ++j) {
 				int idx = gfn_offset;
 				idx /= KVM_PAGES_PER_HPAGE(PT_DIRECTORY_LEVEL + j);
 				ret |= handler(kvm,
 					&memslot->lpage_info[j][idx].rmap_pde,
 					data);
 			}
 			trace_kvm_age_page(hva, memslot, ret);
 			retval |= ret;
 		}
 	}
 	return retval;
 }
 int kvm_unmap_hva(struct kvm *kvm, unsigned long hva)
 {
 	return kvm_handle_hva(kvm, hva, 0, kvm_unmap_rmapp);
 }
 void kvm_set_spte_hva(struct kvm *kvm, unsigned long hva, pte_t pte)
 {
 	kvm_handle_hva(kvm, hva, (unsigned long)&pte, kvm_set_pte_rmapp);
 }
 static int kvm_age_rmapp(struct kvm *kvm, unsigned long *rmapp,
 			 unsigned long data)
 {
 	u64 *spte;
 	int young = 0;
 	/*
 	 * Emulate the accessed bit for EPT, by checking if this page has
 	 * an EPT mapping, and clearing it if it does. On the next access,
 	 * a new EPT mapping will be established.
 	 * This has some overhead, but not as much as the cost of swapping
 	 * out actively used pages or breaking up actively used hugepages.
 	 */
 	if (!shadow_accessed_mask)
 		return kvm_unmap_rmapp(kvm, rmapp, data);
 	spte = rmap_next(kvm, rmapp, NULL);
 	while (spte) {
 		int _young;
 		u64 _spte = *spte;
 		BUG_ON(!(_spte & PT_PRESENT_MASK));
 		_young = _spte & PT_ACCESSED_MASK;
 		if (_young) {
 			young = 1;
 			clear_bit(PT_ACCESSED_SHIFT, (unsigned long *)spte);
 		}
 		spte = rmap_next(kvm, rmapp, spte);
 	}
 	return young;
 }
 #define RMAP_RECYCLE_THRESHOLD 1000
 static void rmap_recycle(struct kvm_vcpu *vcpu, u64 *spte, gfn_t gfn)
 {
 	unsigned long *rmapp;
 	struct kvm_mmu_page *sp;
 	sp = page_header(__pa(spte));
-	gfn = unalias_gfn(vcpu->kvm, gfn);
 	rmapp = gfn_to_rmap(vcpu->kvm, gfn, sp->role.level);
 	kvm_unmap_rmapp(vcpu->kvm, rmapp, 0);
 	kvm_flush_remote_tlbs(vcpu->kvm);
 }
 int kvm_age_hva(struct kvm *kvm, unsigned long hva)
 {
 	return kvm_handle_hva(kvm, hva, 0, kvm_age_rmapp);
 }
 #ifdef MMU_DEBUG
 static int is_empty_shadow_page(u64 *spt)
 {
 	u64 *pos;
 	u64 *end;
 	for (pos = spt, end = pos + PAGE_SIZE / sizeof(u64); pos != end; pos++)
 		if (is_shadow_present_pte(*pos)) {
 			printk(KERN_ERR "%s: %p %llx\n", __func__,
 			       pos, *pos);
 			return 0;
 		}
 	return 1;
 }
 #endif
 static void kvm_mmu_free_page(struct kvm *kvm, struct kvm_mmu_page *sp)
 {
 	ASSERT(is_empty_shadow_page(sp->spt));
 	hlist_del(&sp->hash_link);
 	list_del(&sp->link);
 	__free_page(virt_to_page(sp->spt));
 	if (!sp->role.direct)
 		__free_page(virt_to_page(sp->gfns));
 	kmem_cache_free(mmu_page_header_cache, sp);
 	++kvm->arch.n_free_mmu_pages;
 }
 static unsigned kvm_page_table_hashfn(gfn_t gfn)
 {
 	return gfn & ((1 << KVM_MMU_HASH_SHIFT) - 1);
 }
 static struct kvm_mmu_page *kvm_mmu_alloc_page(struct kvm_vcpu *vcpu,
 					       u64 *parent_pte, int direct)
 {
 	struct kvm_mmu_page *sp;
 	sp = mmu_memory_cache_alloc(&vcpu->arch.mmu_page_header_cache, sizeof *sp);
 	sp->spt = mmu_memory_cache_alloc(&vcpu->arch.mmu_page_cache, PAGE_SIZE);
 	if (!direct)
 		sp->gfns = mmu_memory_cache_alloc(&vcpu->arch.mmu_page_cache,
 						  PAGE_SIZE);
 	set_page_private(virt_to_page(sp->spt), (unsigned long)sp);
 	list_add(&sp->link, &vcpu->kvm->arch.active_mmu_pages);
 	bitmap_zero(sp->slot_bitmap, KVM_MEMORY_SLOTS + KVM_PRIVATE_MEM_SLOTS);
 	sp->multimapped = 0;
 	sp->parent_pte = parent_pte;
 	--vcpu->kvm->arch.n_free_mmu_pages;
 	return sp;
 }
 static void mmu_page_add_parent_pte(struct kvm_vcpu *vcpu,
 				    struct kvm_mmu_page *sp, u64 *parent_pte)
 {
 	struct kvm_pte_chain *pte_chain;
 	struct hlist_node *node;
 	int i;
 	if (!parent_pte)
 		return;
 	if (!sp->multimapped) {
 		u64 *old = sp->parent_pte;
 		if (!old) {
 			sp->parent_pte = parent_pte;
 			return;
 		}
 		sp->multimapped = 1;
 		pte_chain = mmu_alloc_pte_chain(vcpu);
 		INIT_HLIST_HEAD(&sp->parent_ptes);
 		hlist_add_head(&pte_chain->link, &sp->parent_ptes);
 		pte_chain->parent_ptes[0] = old;
 	}
 	hlist_for_each_entry(pte_chain, node, &sp->parent_ptes, link) {
 		if (pte_chain->parent_ptes[NR_PTE_CHAIN_ENTRIES-1])
 			continue;
 		for (i = 0; i < NR_PTE_CHAIN_ENTRIES; ++i)
 			if (!pte_chain->parent_ptes[i]) {
 				pte_chain->parent_ptes[i] = parent_pte;
 				return;
 			}
 	}
 	pte_chain = mmu_alloc_pte_chain(vcpu);
 	BUG_ON(!pte_chain);
 	hlist_add_head(&pte_chain->link, &sp->parent_ptes);
 	pte_chain->parent_ptes[0] = parent_pte;
 }
 static void mmu_page_remove_parent_pte(struct kvm_mmu_page *sp,
 				       u64 *parent_pte)
 {
 	struct kvm_pte_chain *pte_chain;
 	struct hlist_node *node;
 	int i;
 	if (!sp->multimapped) {
 		BUG_ON(sp->parent_pte != parent_pte);
 		sp->parent_pte = NULL;
 		return;
 	}
 	hlist_for_each_entry(pte_chain, node, &sp->parent_ptes, link)
 		for (i = 0; i < NR_PTE_CHAIN_ENTRIES; ++i) {
 			if (!pte_chain->parent_ptes[i])
 				break;
 			if (pte_chain->parent_ptes[i] != parent_pte)
 				continue;
 			while (i + 1 < NR_PTE_CHAIN_ENTRIES
 				&& pte_chain->parent_ptes[i + 1]) {
 				pte_chain->parent_ptes[i]
 					= pte_chain->parent_ptes[i + 1];
 				++i;
 			}
 			pte_chain->parent_ptes[i] = NULL;
 			if (i == 0) {
 				hlist_del(&pte_chain->link);
 				mmu_free_pte_chain(pte_chain);
 				if (hlist_empty(&sp->parent_ptes)) {
 					sp->multimapped = 0;
 					sp->parent_pte = NULL;
 				}
 			}
 			return;
 		}
 	BUG();
 }
 static void mmu_parent_walk(struct kvm_mmu_page *sp, mmu_parent_walk_fn fn)
 {
 	struct kvm_pte_chain *pte_chain;
 	struct hlist_node *node;
 	struct kvm_mmu_page *parent_sp;
 	int i;
 	if (!sp->multimapped && sp->parent_pte) {
 		parent_sp = page_header(__pa(sp->parent_pte));
 		fn(parent_sp, sp->parent_pte);
 		return;
 	}
 	hlist_for_each_entry(pte_chain, node, &sp->parent_ptes, link)
 		for (i = 0; i < NR_PTE_CHAIN_ENTRIES; ++i) {
 			u64 *spte = pte_chain->parent_ptes[i];
 			if (!spte)
 				break;
 			parent_sp = page_header(__pa(spte));
 			fn(parent_sp, spte);
 		}
 }
 static void mark_unsync(struct kvm_mmu_page *sp, u64 *spte);
 static void kvm_mmu_mark_parents_unsync(struct kvm_mmu_page *sp)
 {
 	mmu_parent_walk(sp, mark_unsync);
 }
 static void mark_unsync(struct kvm_mmu_page *sp, u64 *spte)
 {
 	unsigned int index;
 	index = spte - sp->spt;
 	if (__test_and_set_bit(index, sp->unsync_child_bitmap))
 		return;
 	if (sp->unsync_children++)
 		return;
 	kvm_mmu_mark_parents_unsync(sp);
 }
 static void nonpaging_prefetch_page(struct kvm_vcpu *vcpu,
 				    struct kvm_mmu_page *sp)
 {
 	int i;
 	for (i = 0; i < PT64_ENT_PER_PAGE; ++i)
 		sp->spt[i] = shadow_trap_nonpresent_pte;
 }
 static int nonpaging_sync_page(struct kvm_vcpu *vcpu,
 			       struct kvm_mmu_page *sp, bool clear_unsync)
 {
 	return 1;
 }
 static void nonpaging_invlpg(struct kvm_vcpu *vcpu, gva_t gva)
 {
 }
 #define KVM_PAGE_ARRAY_NR 16
 struct kvm_mmu_pages {
 	struct mmu_page_and_offset {
 		struct kvm_mmu_page *sp;
 		unsigned int idx;
 	} page[KVM_PAGE_ARRAY_NR];
 	unsigned int nr;
 };
 #define for_each_unsync_children(bitmap, idx)		\
 	for (idx = find_first_bit(bitmap, 512);		\
 	     idx < 512;					\
 	     idx = find_next_bit(bitmap, 512, idx+1))
 static int mmu_pages_add(struct kvm_mmu_pages *pvec, struct kvm_mmu_page *sp,
 			 int idx)
 {
 	int i;
 	if (sp->unsync)
 		for (i=0; i < pvec->nr; i++)
 			if (pvec->page[i].sp == sp)
 				return 0;
 	pvec->page[pvec->nr].sp = sp;
 	pvec->page[pvec->nr].idx = idx;
 	pvec->nr++;
 	return (pvec->nr == KVM_PAGE_ARRAY_NR);
 }
 static int __mmu_unsync_walk(struct kvm_mmu_page *sp,
 			   struct kvm_mmu_pages *pvec)
 {
 	int i, ret, nr_unsync_leaf = 0;
 	for_each_unsync_children(sp->unsync_child_bitmap, i) {
 		struct kvm_mmu_page *child;
 		u64 ent = sp->spt[i];
 		if (!is_shadow_present_pte(ent) || is_large_pte(ent))
 			goto clear_child_bitmap;
 		child = page_header(ent & PT64_BASE_ADDR_MASK);
 		if (child->unsync_children) {
 			if (mmu_pages_add(pvec, child, i))
 				return -ENOSPC;
 			ret = __mmu_unsync_walk(child, pvec);
 			if (!ret)
 				goto clear_child_bitmap;
 			else if (ret > 0)
 				nr_unsync_leaf += ret;
 			else
 				return ret;
 		} else if (child->unsync) {
 			nr_unsync_leaf++;
 			if (mmu_pages_add(pvec, child, i))
 				return -ENOSPC;
 		} else
 			 goto clear_child_bitmap;
 		continue;
 clear_child_bitmap:
 		__clear_bit(i, sp->unsync_child_bitmap);
 		sp->unsync_children--;
 		WARN_ON((int)sp->unsync_children < 0);
 	}
 	return nr_unsync_leaf;
 }
 static int mmu_unsync_walk(struct kvm_mmu_page *sp,
 			   struct kvm_mmu_pages *pvec)
 {
 	if (!sp->unsync_children)
 		return 0;
 	mmu_pages_add(pvec, sp, 0);
 	return __mmu_unsync_walk(sp, pvec);
 }
 static void kvm_unlink_unsync_page(struct kvm *kvm, struct kvm_mmu_page *sp)
 {
 	WARN_ON(!sp->unsync);
 	trace_kvm_mmu_sync_page(sp);
 	sp->unsync = 0;
 	--kvm->stat.mmu_unsync;
 }
 static int kvm_mmu_prepare_zap_page(struct kvm *kvm, struct kvm_mmu_page *sp,
 				    struct list_head *invalid_list);
 static void kvm_mmu_commit_zap_page(struct kvm *kvm,
 				    struct list_head *invalid_list);
 #define for_each_gfn_sp(kvm, sp, gfn, pos)				\
   hlist_for_each_entry(sp, pos,						\
    &(kvm)->arch.mmu_page_hash[kvm_page_table_hashfn(gfn)], hash_link)	\
 	if ((sp)->gfn != (gfn)) {} else
 #define for_each_gfn_indirect_valid_sp(kvm, sp, gfn, pos)		\
   hlist_for_each_entry(sp, pos,						\
    &(kvm)->arch.mmu_page_hash[kvm_page_table_hashfn(gfn)], hash_link)	\
 		if ((sp)->gfn != (gfn) || (sp)->role.direct ||		\
 			(sp)->role.invalid) {} else
 /* @sp->gfn should be write-protected at the call site */
 static int __kvm_sync_page(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,
 			   struct list_head *invalid_list, bool clear_unsync)
 {
 	if (sp->role.cr4_pae != !!is_pae(vcpu)) {
 		kvm_mmu_prepare_zap_page(vcpu->kvm, sp, invalid_list);
 		return 1;
 	}
 	if (clear_unsync)
 		kvm_unlink_unsync_page(vcpu->kvm, sp);
 	if (vcpu->arch.mmu.sync_page(vcpu, sp, clear_unsync)) {
 		kvm_mmu_prepare_zap_page(vcpu->kvm, sp, invalid_list);
 		return 1;
 	}
 	kvm_mmu_flush_tlb(vcpu);
 	return 0;
 }
 static int kvm_sync_page_transient(struct kvm_vcpu *vcpu,
 				   struct kvm_mmu_page *sp)
 {
 	LIST_HEAD(invalid_list);
 	int ret;
 	ret = __kvm_sync_page(vcpu, sp, &invalid_list, false);
 	if (ret)
 		kvm_mmu_commit_zap_page(vcpu->kvm, &invalid_list);
 	return ret;
 }
 static int kvm_sync_page(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,
 			 struct list_head *invalid_list)
 {
 	return __kvm_sync_page(vcpu, sp, invalid_list, true);
 }
 /* @gfn should be write-protected at the call site */
 static void kvm_sync_pages(struct kvm_vcpu *vcpu,  gfn_t gfn)
 {
 	struct kvm_mmu_page *s;
 	struct hlist_node *node;
 	LIST_HEAD(invalid_list);
 	bool flush = false;
 	for_each_gfn_indirect_valid_sp(vcpu->kvm, s, gfn, node) {
 		if (!s->unsync)
 			continue;
 		WARN_ON(s->role.level != PT_PAGE_TABLE_LEVEL);
 		if ((s->role.cr4_pae != !!is_pae(vcpu)) ||
 			(vcpu->arch.mmu.sync_page(vcpu, s, true))) {
 			kvm_mmu_prepare_zap_page(vcpu->kvm, s, &invalid_list);
 			continue;
 		}
 		kvm_unlink_unsync_page(vcpu->kvm, s);
 		flush = true;
 	}
 	kvm_mmu_commit_zap_page(vcpu->kvm, &invalid_list);
 	if (flush)
 		kvm_mmu_flush_tlb(vcpu);
 }
 struct mmu_page_path {
 	struct kvm_mmu_page *parent[PT64_ROOT_LEVEL-1];
 	unsigned int idx[PT64_ROOT_LEVEL-1];
 };
 #define for_each_sp(pvec, sp, parents, i)			\
 		for (i = mmu_pages_next(&pvec, &parents, -1),	\
 			sp = pvec.page[i].sp;			\
 			i < pvec.nr && ({ sp = pvec.page[i].sp; 1;});	\
 			i = mmu_pages_next(&pvec, &parents, i))
 static int mmu_pages_next(struct kvm_mmu_pages *pvec,
 			  struct mmu_page_path *parents,
 			  int i)
 {
 	int n;
 	for (n = i+1; n < pvec->nr; n++) {
 		struct kvm_mmu_page *sp = pvec->page[n].sp;
 		if (sp->role.level == PT_PAGE_TABLE_LEVEL) {
 			parents->idx[0] = pvec->page[n].idx;
 			return n;
 		}
 		parents->parent[sp->role.level-2] = sp;
 		parents->idx[sp->role.level-1] = pvec->page[n].idx;
 	}
 	return n;
 }
 static void mmu_pages_clear_parents(struct mmu_page_path *parents)
 {
 	struct kvm_mmu_page *sp;
 	unsigned int level = 0;
 	do {
 		unsigned int idx = parents->idx[level];
 		sp = parents->parent[level];
 		if (!sp)
 			return;
 		--sp->unsync_children;
 		WARN_ON((int)sp->unsync_children < 0);
 		__clear_bit(idx, sp->unsync_child_bitmap);
 		level++;
 	} while (level < PT64_ROOT_LEVEL-1 && !sp->unsync_children);
 }
 static void kvm_mmu_pages_init(struct kvm_mmu_page *parent,
 			       struct mmu_page_path *parents,
 			       struct kvm_mmu_pages *pvec)
 {
 	parents->parent[parent->role.level-1] = NULL;
 	pvec->nr = 0;
 }
 static void mmu_sync_children(struct kvm_vcpu *vcpu,
 			      struct kvm_mmu_page *parent)
 {
 	int i;
 	struct kvm_mmu_page *sp;
 	struct mmu_page_path parents;
 	struct kvm_mmu_pages pages;
 	LIST_HEAD(invalid_list);
 	kvm_mmu_pages_init(parent, &parents, &pages);
 	while (mmu_unsync_walk(parent, &pages)) {
 		int protected = 0;
 		for_each_sp(pages, sp, parents, i)
 			protected |= rmap_write_protect(vcpu->kvm, sp->gfn);
 		if (protected)
 			kvm_flush_remote_tlbs(vcpu->kvm);
 		for_each_sp(pages, sp, parents, i) {
 			kvm_sync_page(vcpu, sp, &invalid_list);
 			mmu_pages_clear_parents(&parents);
 		}
 		kvm_mmu_commit_zap_page(vcpu->kvm, &invalid_list);
 		cond_resched_lock(&vcpu->kvm->mmu_lock);
 		kvm_mmu_pages_init(parent, &parents, &pages);
 	}
 }
 static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu,
 					     gfn_t gfn,
 					     gva_t gaddr,
 					     unsigned level,
 					     int direct,
 					     unsigned access,
 					     u64 *parent_pte)
 {
 	union kvm_mmu_page_role role;
 	unsigned quadrant;
 	struct kvm_mmu_page *sp;
 	struct hlist_node *node;
 	bool need_sync = false;
 	role = vcpu->arch.mmu.base_role;
 	role.level = level;
 	role.direct = direct;
 	if (role.direct)
 		role.cr4_pae = 0;
 	role.access = access;
 	if (!tdp_enabled && vcpu->arch.mmu.root_level <= PT32_ROOT_LEVEL) {
 		quadrant = gaddr >> (PAGE_SHIFT + (PT64_PT_BITS * level));
 		quadrant &= (1 << ((PT32_PT_BITS - PT64_PT_BITS) * level)) - 1;
 		role.quadrant = quadrant;
 	}
 	for_each_gfn_sp(vcpu->kvm, sp, gfn, node) {
 		if (!need_sync && sp->unsync)
 			need_sync = true;
 		if (sp->role.word != role.word)
 			continue;
 		if (sp->unsync && kvm_sync_page_transient(vcpu, sp))
 			break;
 		mmu_page_add_parent_pte(vcpu, sp, parent_pte);
 		if (sp->unsync_children) {
 			set_bit(KVM_REQ_MMU_SYNC, &vcpu->requests);
 			kvm_mmu_mark_parents_unsync(sp);
 		} else if (sp->unsync)
 			kvm_mmu_mark_parents_unsync(sp);
 		trace_kvm_mmu_get_page(sp, false);
 		return sp;
 	}
 	++vcpu->kvm->stat.mmu_cache_miss;
 	sp = kvm_mmu_alloc_page(vcpu, parent_pte, direct);
 	if (!sp)
 		return sp;
 	sp->gfn = gfn;
 	sp->role = role;
 	hlist_add_head(&sp->hash_link,
 		&vcpu->kvm->arch.mmu_page_hash[kvm_page_table_hashfn(gfn)]);
 	if (!direct) {
 		if (rmap_write_protect(vcpu->kvm, gfn))
 			kvm_flush_remote_tlbs(vcpu->kvm);
 		if (level > PT_PAGE_TABLE_LEVEL && need_sync)
 			kvm_sync_pages(vcpu, gfn);
 		account_shadowed(vcpu->kvm, gfn);
 	}
 	if (shadow_trap_nonpresent_pte != shadow_notrap_nonpresent_pte)
 		vcpu->arch.mmu.prefetch_page(vcpu, sp);
 	else
 		nonpaging_prefetch_page(vcpu, sp);
 	trace_kvm_mmu_get_page(sp, true);
 	return sp;
 }
 static void shadow_walk_init(struct kvm_shadow_walk_iterator *iterator,
 			     struct kvm_vcpu *vcpu, u64 addr)
 {
 	iterator->addr = addr;
 	iterator->shadow_addr = vcpu->arch.mmu.root_hpa;
 	iterator->level = vcpu->arch.mmu.shadow_root_level;
 	if (iterator->level == PT32E_ROOT_LEVEL) {
 		iterator->shadow_addr
 			= vcpu->arch.mmu.pae_root[(addr >> 30) & 3];
 		iterator->shadow_addr &= PT64_BASE_ADDR_MASK;
 		--iterator->level;
 		if (!iterator->shadow_addr)
 			iterator->level = 0;
 	}
 }
 static bool shadow_walk_okay(struct kvm_shadow_walk_iterator *iterator)
 {
 	if (iterator->level < PT_PAGE_TABLE_LEVEL)
 		return false;
 	if (iterator->level == PT_PAGE_TABLE_LEVEL)
 		if (is_large_pte(*iterator->sptep))
 			return false;
 	iterator->index = SHADOW_PT_INDEX(iterator->addr, iterator->level);
 	iterator->sptep	= ((u64 *)__va(iterator->shadow_addr)) + iterator->index;
 	return true;
 }
 static void shadow_walk_next(struct kvm_shadow_walk_iterator *iterator)
 {
 	iterator->shadow_addr = *iterator->sptep & PT64_BASE_ADDR_MASK;
 	--iterator->level;
 }
 static void kvm_mmu_page_unlink_children(struct kvm *kvm,
 					 struct kvm_mmu_page *sp)
 {
 	unsigned i;
 	u64 *pt;
 	u64 ent;
 	pt = sp->spt;
 	for (i = 0; i < PT64_ENT_PER_PAGE; ++i) {
 		ent = pt[i];
 		if (is_shadow_present_pte(ent)) {
 			if (!is_last_spte(ent, sp->role.level)) {
 				ent &= PT64_BASE_ADDR_MASK;
 				mmu_page_remove_parent_pte(page_header(ent),
 							   &pt[i]);
 			} else {
 				if (is_large_pte(ent))
 					--kvm->stat.lpages;
 				rmap_remove(kvm, &pt[i]);
 			}
 		}
 		pt[i] = shadow_trap_nonpresent_pte;
 	}
 }
 static void kvm_mmu_put_page(struct kvm_mmu_page *sp, u64 *parent_pte)
 {
 	mmu_page_remove_parent_pte(sp, parent_pte);
 }
 static void kvm_mmu_reset_last_pte_updated(struct kvm *kvm)
 {
 	int i;
 	struct kvm_vcpu *vcpu;
 	kvm_for_each_vcpu(i, vcpu, kvm)
 		vcpu->arch.last_pte_updated = NULL;
 }
 static void kvm_mmu_unlink_parents(struct kvm *kvm, struct kvm_mmu_page *sp)
 {
 	u64 *parent_pte;
 	while (sp->multimapped || sp->parent_pte) {
 		if (!sp->multimapped)
 			parent_pte = sp->parent_pte;
 		else {
 			struct kvm_pte_chain *chain;
 			chain = container_of(sp->parent_ptes.first,
 					     struct kvm_pte_chain, link);
 			parent_pte = chain->parent_ptes[0];
 		}
 		BUG_ON(!parent_pte);
 		kvm_mmu_put_page(sp, parent_pte);
 		__set_spte(parent_pte, shadow_trap_nonpresent_pte);
 	}
 }
 static int mmu_zap_unsync_children(struct kvm *kvm,
 				   struct kvm_mmu_page *parent,
 				   struct list_head *invalid_list)
 {
 	int i, zapped = 0;
 	struct mmu_page_path parents;
 	struct kvm_mmu_pages pages;
 	if (parent->role.level == PT_PAGE_TABLE_LEVEL)
 		return 0;
 	kvm_mmu_pages_init(parent, &parents, &pages);
 	while (mmu_unsync_walk(parent, &pages)) {
 		struct kvm_mmu_page *sp;
 		for_each_sp(pages, sp, parents, i) {
 			kvm_mmu_prepare_zap_page(kvm, sp, invalid_list);
 			mmu_pages_clear_parents(&parents);
 			zapped++;
 		}
 		kvm_mmu_pages_init(parent, &parents, &pages);
 	}
 	return zapped;
 }
 static int kvm_mmu_prepare_zap_page(struct kvm *kvm, struct kvm_mmu_page *sp,
 				    struct list_head *invalid_list)
 {
 	int ret;
 	trace_kvm_mmu_prepare_zap_page(sp);
 	++kvm->stat.mmu_shadow_zapped;
 	ret = mmu_zap_unsync_children(kvm, sp, invalid_list);
 	kvm_mmu_page_unlink_children(kvm, sp);
 	kvm_mmu_unlink_parents(kvm, sp);
 	if (!sp->role.invalid && !sp->role.direct)
 		unaccount_shadowed(kvm, sp->gfn);
 	if (sp->unsync)
 		kvm_unlink_unsync_page(kvm, sp);
 	if (!sp->root_count) {
 		/* Count self */
 		ret++;
 		list_move(&sp->link, invalid_list);
 	} else {
 		list_move(&sp->link, &kvm->arch.active_mmu_pages);
 		kvm_reload_remote_mmus(kvm);
 	}
 	sp->role.invalid = 1;
 	kvm_mmu_reset_last_pte_updated(kvm);
 	return ret;
 }
 static void kvm_mmu_commit_zap_page(struct kvm *kvm,
 				    struct list_head *invalid_list)
 {
 	struct kvm_mmu_page *sp;
 	if (list_empty(invalid_list))
 		return;
 	kvm_flush_remote_tlbs(kvm);
 	do {
 		sp = list_first_entry(invalid_list, struct kvm_mmu_page, link);
 		WARN_ON(!sp->role.invalid || sp->root_count);
 		kvm_mmu_free_page(kvm, sp);
 	} while (!list_empty(invalid_list));
 }
 /*
  * Changing the number of mmu pages allocated to the vm
  * Note: if kvm_nr_mmu_pages is too small, you will get dead lock
  */
 void kvm_mmu_change_mmu_pages(struct kvm *kvm, unsigned int kvm_nr_mmu_pages)
 {
 	int used_pages;
 	LIST_HEAD(invalid_list);
 	used_pages = kvm->arch.n_alloc_mmu_pages - kvm->arch.n_free_mmu_pages;
 	used_pages = max(0, used_pages);
 	/*
 	 * If we set the number of mmu pages to be smaller be than the
 	 * number of actived pages , we must to free some mmu pages before we
 	 * change the value
 	 */
 	if (used_pages > kvm_nr_mmu_pages) {
 		while (used_pages > kvm_nr_mmu_pages &&
 			!list_empty(&kvm->arch.active_mmu_pages)) {
 			struct kvm_mmu_page *page;
 			page = container_of(kvm->arch.active_mmu_pages.prev,
 					    struct kvm_mmu_page, link);
 			used_pages -= kvm_mmu_prepare_zap_page(kvm, page,
 							       &invalid_list);
 		}
 		kvm_mmu_commit_zap_page(kvm, &invalid_list);
 		kvm_nr_mmu_pages = used_pages;
 		kvm->arch.n_free_mmu_pages = 0;
 	}
 	else
 		kvm->arch.n_free_mmu_pages += kvm_nr_mmu_pages
 					 - kvm->arch.n_alloc_mmu_pages;
 	kvm->arch.n_alloc_mmu_pages = kvm_nr_mmu_pages;
 }
 static int kvm_mmu_unprotect_page(struct kvm *kvm, gfn_t gfn)
 {
 	struct kvm_mmu_page *sp;
 	struct hlist_node *node;
 	LIST_HEAD(invalid_list);
 	int r;
 	pgprintk("%s: looking for gfn %lx\n", __func__, gfn);
 	r = 0;
 	for_each_gfn_indirect_valid_sp(kvm, sp, gfn, node) {
 		pgprintk("%s: gfn %lx role %x\n", __func__, gfn,
 			 sp->role.word);
 		r = 1;
 		kvm_mmu_prepare_zap_page(kvm, sp, &invalid_list);
 	}
 	kvm_mmu_commit_zap_page(kvm, &invalid_list);
 	return r;
 }
 static void mmu_unshadow(struct kvm *kvm, gfn_t gfn)
 {
 	struct kvm_mmu_page *sp;
 	struct hlist_node *node;
 	LIST_HEAD(invalid_list);
 	for_each_gfn_indirect_valid_sp(kvm, sp, gfn, node) {
 		pgprintk("%s: zap %lx %x\n",
 			 __func__, gfn, sp->role.word);
 		kvm_mmu_prepare_zap_page(kvm, sp, &invalid_list);
 	}
 	kvm_mmu_commit_zap_page(kvm, &invalid_list);
 }
 static void page_header_update_slot(struct kvm *kvm, void *pte, gfn_t gfn)
 {
 	int slot = memslot_id(kvm, gfn);
 	struct kvm_mmu_page *sp = page_header(__pa(pte));
 	__set_bit(slot, sp->slot_bitmap);
 }
 static void mmu_convert_notrap(struct kvm_mmu_page *sp)
 {
 	int i;
 	u64 *pt = sp->spt;
 	if (shadow_trap_nonpresent_pte == shadow_notrap_nonpresent_pte)
 		return;
 	for (i = 0; i < PT64_ENT_PER_PAGE; ++i) {
 		if (pt[i] == shadow_notrap_nonpresent_pte)
 			__set_spte(&pt[i], shadow_trap_nonpresent_pte);
 	}
 }
 /*
  * The function is based on mtrr_type_lookup() in
  * arch/x86/kernel/cpu/mtrr/generic.c
  */
 static int get_mtrr_type(struct mtrr_state_type *mtrr_state,
 			 u64 start, u64 end)
 {
 	int i;
 	u64 base, mask;
 	u8 prev_match, curr_match;
 	int num_var_ranges = KVM_NR_VAR_MTRR;
 	if (!mtrr_state->enabled)
 		return 0xFF;
 	/* Make end inclusive end, instead of exclusive */
 	end--;
 	/* Look in fixed ranges. Just return the type as per start */
 	if (mtrr_state->have_fixed && (start < 0x100000)) {
 		int idx;
 		if (start < 0x80000) {
 			idx = 0;
 			idx += (start >> 16);
 			return mtrr_state->fixed_ranges[idx];
 		} else if (start < 0xC0000) {
 			idx = 1 * 8;
 			idx += ((start - 0x80000) >> 14);
 			return mtrr_state->fixed_ranges[idx];
 		} else if (start < 0x1000000) {
 			idx = 3 * 8;
 			idx += ((start - 0xC0000) >> 12);
 			return mtrr_state->fixed_ranges[idx];
 		}
 	}
 	/*
 	 * Look in variable ranges
 	 * Look of multiple ranges matching this address and pick type
 	 * as per MTRR precedence
 	 */
 	if (!(mtrr_state->enabled & 2))
 		return mtrr_state->def_type;
 	prev_match = 0xFF;
 	for (i = 0; i < num_var_ranges; ++i) {
 		unsigned short start_state, end_state;
 		if (!(mtrr_state->var_ranges[i].mask_lo & (1 << 11)))
 			continue;
 		base = (((u64)mtrr_state->var_ranges[i].base_hi) << 32) +
 		       (mtrr_state->var_ranges[i].base_lo & PAGE_MASK);
 		mask = (((u64)mtrr_state->var_ranges[i].mask_hi) << 32) +
 		       (mtrr_state->var_ranges[i].mask_lo & PAGE_MASK);
 		start_state = ((start & mask) == (base & mask));
 		end_state = ((end & mask) == (base & mask));
 		if (start_state != end_state)
 			return 0xFE;
 		if ((start & mask) != (base & mask))
 			continue;
 		curr_match = mtrr_state->var_ranges[i].base_lo & 0xff;
 		if (prev_match == 0xFF) {
 			prev_match = curr_match;
 			continue;
 		}
 		if (prev_match == MTRR_TYPE_UNCACHABLE ||
 		    curr_match == MTRR_TYPE_UNCACHABLE)
 			return MTRR_TYPE_UNCACHABLE;
 		if ((prev_match == MTRR_TYPE_WRBACK &&
 		     curr_match == MTRR_TYPE_WRTHROUGH) ||
 		    (prev_match == MTRR_TYPE_WRTHROUGH &&
 		     curr_match == MTRR_TYPE_WRBACK)) {
 			prev_match = MTRR_TYPE_WRTHROUGH;
 			curr_match = MTRR_TYPE_WRTHROUGH;
 		}
 		if (prev_match != curr_match)
 			return MTRR_TYPE_UNCACHABLE;
 	}
 	if (prev_match != 0xFF)
 		return prev_match;
 	return mtrr_state->def_type;
 }
 u8 kvm_get_guest_memory_type(struct kvm_vcpu *vcpu, gfn_t gfn)
 {
 	u8 mtrr;
 	mtrr = get_mtrr_type(&vcpu->arch.mtrr_state, gfn << PAGE_SHIFT,
 			     (gfn << PAGE_SHIFT) + PAGE_SIZE);
 	if (mtrr == 0xfe || mtrr == 0xff)
 		mtrr = MTRR_TYPE_WRBACK;
 	return mtrr;
 }
 EXPORT_SYMBOL_GPL(kvm_get_guest_memory_type);
 static void __kvm_unsync_page(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp)
 {
 	trace_kvm_mmu_unsync_page(sp);
 	++vcpu->kvm->stat.mmu_unsync;
 	sp->unsync = 1;
 	kvm_mmu_mark_parents_unsync(sp);
 	mmu_convert_notrap(sp);
 }
 static void kvm_unsync_pages(struct kvm_vcpu *vcpu,  gfn_t gfn)
 {
 	struct kvm_mmu_page *s;
 	struct hlist_node *node;
 	for_each_gfn_indirect_valid_sp(vcpu->kvm, s, gfn, node) {
 		if (s->unsync)
 			continue;
 		WARN_ON(s->role.level != PT_PAGE_TABLE_LEVEL);
 		__kvm_unsync_page(vcpu, s);
 	}
 }
 static int mmu_need_write_protect(struct kvm_vcpu *vcpu, gfn_t gfn,
 				  bool can_unsync)
 {
 	struct kvm_mmu_page *s;
 	struct hlist_node *node;
 	bool need_unsync = false;
 	for_each_gfn_indirect_valid_sp(vcpu->kvm, s, gfn, node) {
 		if (s->role.level != PT_PAGE_TABLE_LEVEL)
 			return 1;
 		if (!need_unsync && !s->unsync) {
 			if (!can_unsync || !oos_shadow)
 				return 1;
 			need_unsync = true;
 		}
 	}
 	if (need_unsync)
 		kvm_unsync_pages(vcpu, gfn);
 	return 0;
 }
 static int set_spte(struct kvm_vcpu *vcpu, u64 *sptep,
 		    unsigned pte_access, int user_fault,
 		    int write_fault, int dirty, int level,
 		    gfn_t gfn, pfn_t pfn, bool speculative,
 		    bool can_unsync, bool reset_host_protection)
 {
 	u64 spte;
 	int ret = 0;
 	/*
 	 * We don't set the accessed bit, since we sometimes want to see
 	 * whether the guest actually used the pte (in order to detect
 	 * demand paging).
 	 */
 	spte = shadow_base_present_pte | shadow_dirty_mask;
 	if (!speculative)
 		spte |= shadow_accessed_mask;
 	if (!dirty)
 		pte_access &= ~ACC_WRITE_MASK;
 	if (pte_access & ACC_EXEC_MASK)
 		spte |= shadow_x_mask;
 	else
 		spte |= shadow_nx_mask;
 	if (pte_access & ACC_USER_MASK)
 		spte |= shadow_user_mask;
 	if (level > PT_PAGE_TABLE_LEVEL)
 		spte |= PT_PAGE_SIZE_MASK;
 	if (tdp_enabled)
 		spte |= kvm_x86_ops->get_mt_mask(vcpu, gfn,
 			kvm_is_mmio_pfn(pfn));
 	if (reset_host_protection)
 		spte |= SPTE_HOST_WRITEABLE;
 	spte |= (u64)pfn << PAGE_SHIFT;
 	if ((pte_access & ACC_WRITE_MASK)
 	    || (!tdp_enabled && write_fault && !is_write_protection(vcpu)
 		&& !user_fault)) {
 		if (level > PT_PAGE_TABLE_LEVEL &&
 		    has_wrprotected_page(vcpu->kvm, gfn, level)) {
 			ret = 1;
 			rmap_remove(vcpu->kvm, sptep);
 			spte = shadow_trap_nonpresent_pte;
 			goto set_pte;
 		}
 		spte |= PT_WRITABLE_MASK;
 		if (!tdp_enabled && !(pte_access & ACC_WRITE_MASK))
 			spte &= ~PT_USER_MASK;
 		/*
 		 * Optimization: for pte sync, if spte was writable the hash
 		 * lookup is unnecessary (and expensive). Write protection
 		 * is responsibility of mmu_get_page / kvm_sync_page.
 		 * Same reasoning can be applied to dirty page accounting.
 		 */
 		if (!can_unsync && is_writable_pte(*sptep))
 			goto set_pte;
 		if (mmu_need_write_protect(vcpu, gfn, can_unsync)) {
 			pgprintk("%s: found shadow page for %lx, marking ro\n",
 				 __func__, gfn);
 			ret = 1;
 			pte_access &= ~ACC_WRITE_MASK;
 			if (is_writable_pte(spte))
 				spte &= ~PT_WRITABLE_MASK;
 		}
 	}
 	if (pte_access & ACC_WRITE_MASK)
 		mark_page_dirty(vcpu->kvm, gfn);
 set_pte:
 	__set_spte(sptep, spte);
 	return ret;
 }
 static void mmu_set_spte(struct kvm_vcpu *vcpu, u64 *sptep,
 			 unsigned pt_access, unsigned pte_access,
 			 int user_fault, int write_fault, int dirty,
 			 int *ptwrite, int level, gfn_t gfn,
 			 pfn_t pfn, bool speculative,
 			 bool reset_host_protection)
 {
 	int was_rmapped = 0;
 	int was_writable = is_writable_pte(*sptep);
 	int rmap_count;
 	pgprintk("%s: spte %llx access %x write_fault %d"
 		 " user_fault %d gfn %lx\n",
 		 __func__, *sptep, pt_access,
 		 write_fault, user_fault, gfn);
 	if (is_rmap_spte(*sptep)) {
 		/*
 		 * If we overwrite a PTE page pointer with a 2MB PMD, unlink
 		 * the parent of the now unreachable PTE.
 		 */
 		if (level > PT_PAGE_TABLE_LEVEL &&
 		    !is_large_pte(*sptep)) {
 			struct kvm_mmu_page *child;
 			u64 pte = *sptep;
 			child = page_header(pte & PT64_BASE_ADDR_MASK);
 			mmu_page_remove_parent_pte(child, sptep);
 			__set_spte(sptep, shadow_trap_nonpresent_pte);
 			kvm_flush_remote_tlbs(vcpu->kvm);
 		} else if (pfn != spte_to_pfn(*sptep)) {
 			pgprintk("hfn old %lx new %lx\n",
 				 spte_to_pfn(*sptep), pfn);
 			rmap_remove(vcpu->kvm, sptep);
 			__set_spte(sptep, shadow_trap_nonpresent_pte);
 			kvm_flush_remote_tlbs(vcpu->kvm);
 		} else
 			was_rmapped = 1;
 	}
 	if (set_spte(vcpu, sptep, pte_access, user_fault, write_fault,
 		      dirty, level, gfn, pfn, speculative, true,
 		      reset_host_protection)) {
 		if (write_fault)
 			*ptwrite = 1;
 		kvm_mmu_flush_tlb(vcpu);
 	}
 	pgprintk("%s: setting spte %llx\n", __func__, *sptep);
 	pgprintk("instantiating %s PTE (%s) at %ld (%llx) addr %p\n",
 		 is_large_pte(*sptep)? "2MB" : "4kB",
 		 *sptep & PT_PRESENT_MASK ?"RW":"R", gfn,
 		 *sptep, sptep);
 	if (!was_rmapped && is_large_pte(*sptep))
 		++vcpu->kvm->stat.lpages;
 	page_header_update_slot(vcpu->kvm, sptep, gfn);
 	if (!was_rmapped) {
 		rmap_count = rmap_add(vcpu, sptep, gfn);
 		kvm_release_pfn_clean(pfn);
 		if (rmap_count > RMAP_RECYCLE_THRESHOLD)
 			rmap_recycle(vcpu, sptep, gfn);
 	} else {
 		if (was_writable)
 			kvm_release_pfn_dirty(pfn);
 		else
 			kvm_release_pfn_clean(pfn);
 	}
 	if (speculative) {
 		vcpu->arch.last_pte_updated = sptep;
 		vcpu->arch.last_pte_gfn = gfn;
 	}
 }
 static void nonpaging_new_cr3(struct kvm_vcpu *vcpu)
 {
 }
 static int __direct_map(struct kvm_vcpu *vcpu, gpa_t v, int write,
 			int level, gfn_t gfn, pfn_t pfn)
 {
 	struct kvm_shadow_walk_iterator iterator;
 	struct kvm_mmu_page *sp;
 	int pt_write = 0;
 	gfn_t pseudo_gfn;
 	for_each_shadow_entry(vcpu, (u64)gfn << PAGE_SHIFT, iterator) {
 		if (iterator.level == level) {
 			mmu_set_spte(vcpu, iterator.sptep, ACC_ALL, ACC_ALL,
 				     0, write, 1, &pt_write,
 				     level, gfn, pfn, false, true);
 			++vcpu->stat.pf_fixed;
 			break;
 		}
 		if (*iterator.sptep == shadow_trap_nonpresent_pte) {
 			u64 base_addr = iterator.addr;
 			base_addr &= PT64_LVL_ADDR_MASK(iterator.level);
 			pseudo_gfn = base_addr >> PAGE_SHIFT;
 			sp = kvm_mmu_get_page(vcpu, pseudo_gfn, iterator.addr,
 					      iterator.level - 1,
 					      1, ACC_ALL, iterator.sptep);
 			if (!sp) {
 				pgprintk("nonpaging_map: ENOMEM\n");
 				kvm_release_pfn_clean(pfn);
 				return -ENOMEM;
 			}
 			__set_spte(iterator.sptep,
 				   __pa(sp->spt)
 				   | PT_PRESENT_MASK | PT_WRITABLE_MASK
 				   | shadow_user_mask | shadow_x_mask);
 		}
 	}
 	return pt_write;
 }
 static void kvm_send_hwpoison_signal(struct kvm *kvm, gfn_t gfn)
 {
 	char buf[1];
 	void __user *hva;
 	int r;
 	/* Touch the page, so send SIGBUS */
 	hva = (void __user *)gfn_to_hva(kvm, gfn);
 	r = copy_from_user(buf, hva, 1);
 }
 static int kvm_handle_bad_page(struct kvm *kvm, gfn_t gfn, pfn_t pfn)
 {
 	kvm_release_pfn_clean(pfn);
 	if (is_hwpoison_pfn(pfn)) {
 		kvm_send_hwpoison_signal(kvm, gfn);
 		return 0;
 	}
 	return 1;
 }
 static int nonpaging_map(struct kvm_vcpu *vcpu, gva_t v, int write, gfn_t gfn)
 {
 	int r;
 	int level;
 	pfn_t pfn;
 	unsigned long mmu_seq;
 	level = mapping_level(vcpu, gfn);
 	/*
 	 * This path builds a PAE pagetable - so we can map 2mb pages at
 	 * maximum. Therefore check if the level is larger than that.
 	 */
 	if (level > PT_DIRECTORY_LEVEL)
 		level = PT_DIRECTORY_LEVEL;
 	gfn &= ~(KVM_PAGES_PER_HPAGE(level) - 1);
 	mmu_seq = vcpu->kvm->mmu_notifier_seq;
 	smp_rmb();
 	pfn = gfn_to_pfn(vcpu->kvm, gfn);
 	/* mmio */
 	if (is_error_pfn(pfn))
 		return kvm_handle_bad_page(vcpu->kvm, gfn, pfn);
 	spin_lock(&vcpu->kvm->mmu_lock);
 	if (mmu_notifier_retry(vcpu, mmu_seq))
 		goto out_unlock;
 	kvm_mmu_free_some_pages(vcpu);
 	r = __direct_map(vcpu, v, write, level, gfn, pfn);
 	spin_unlock(&vcpu->kvm->mmu_lock);
 	return r;
 out_unlock:
 	spin_unlock(&vcpu->kvm->mmu_lock);
 	kvm_release_pfn_clean(pfn);
 	return 0;
 }
 static void mmu_free_roots(struct kvm_vcpu *vcpu)
 {
 	int i;
 	struct kvm_mmu_page *sp;
 	LIST_HEAD(invalid_list);
 	if (!VALID_PAGE(vcpu->arch.mmu.root_hpa))
 		return;
 	spin_lock(&vcpu->kvm->mmu_lock);
 	if (vcpu->arch.mmu.shadow_root_level == PT64_ROOT_LEVEL) {
 		hpa_t root = vcpu->arch.mmu.root_hpa;
 		sp = page_header(root);
 		--sp->root_count;
 		if (!sp->root_count && sp->role.invalid) {
 			kvm_mmu_prepare_zap_page(vcpu->kvm, sp, &invalid_list);
 			kvm_mmu_commit_zap_page(vcpu->kvm, &invalid_list);
 		}
 		vcpu->arch.mmu.root_hpa = INVALID_PAGE;
 		spin_unlock(&vcpu->kvm->mmu_lock);
 		return;
 	}
 	for (i = 0; i < 4; ++i) {
 		hpa_t root = vcpu->arch.mmu.pae_root[i];
 		if (root) {
 			root &= PT64_BASE_ADDR_MASK;
 			sp = page_header(root);
 			--sp->root_count;
 			if (!sp->root_count && sp->role.invalid)
 				kvm_mmu_prepare_zap_page(vcpu->kvm, sp,
 							 &invalid_list);
 		}
 		vcpu->arch.mmu.pae_root[i] = INVALID_PAGE;
 	}
 	kvm_mmu_commit_zap_page(vcpu->kvm, &invalid_list);
 	spin_unlock(&vcpu->kvm->mmu_lock);
 	vcpu->arch.mmu.root_hpa = INVALID_PAGE;
 }
 static int mmu_check_root(struct kvm_vcpu *vcpu, gfn_t root_gfn)
 {
 	int ret = 0;
 	if (!kvm_is_visible_gfn(vcpu->kvm, root_gfn)) {
 		set_bit(KVM_REQ_TRIPLE_FAULT, &vcpu->requests);
 		ret = 1;
 	}
 	return ret;
 }
 static int mmu_alloc_roots(struct kvm_vcpu *vcpu)
 {
 	int i;
 	gfn_t root_gfn;
 	struct kvm_mmu_page *sp;
 	int direct = 0;
 	u64 pdptr;
 	root_gfn = vcpu->arch.cr3 >> PAGE_SHIFT;
 	if (vcpu->arch.mmu.shadow_root_level == PT64_ROOT_LEVEL) {
 		hpa_t root = vcpu->arch.mmu.root_hpa;
 		ASSERT(!VALID_PAGE(root));
 		if (mmu_check_root(vcpu, root_gfn))
 			return 1;
 		if (tdp_enabled) {
 			direct = 1;
 			root_gfn = 0;
 		}
 		spin_lock(&vcpu->kvm->mmu_lock);
 		kvm_mmu_free_some_pages(vcpu);
 		sp = kvm_mmu_get_page(vcpu, root_gfn, 0,
 				      PT64_ROOT_LEVEL, direct,
 				      ACC_ALL, NULL);
 		root = __pa(sp->spt);
 		++sp->root_count;
 		spin_unlock(&vcpu->kvm->mmu_lock);
 		vcpu->arch.mmu.root_hpa = root;
 		return 0;
 	}
 	direct = !is_paging(vcpu);
 	for (i = 0; i < 4; ++i) {
 		hpa_t root = vcpu->arch.mmu.pae_root[i];
 		ASSERT(!VALID_PAGE(root));
 		if (vcpu->arch.mmu.root_level == PT32E_ROOT_LEVEL) {
 			pdptr = kvm_pdptr_read(vcpu, i);
 			if (!is_present_gpte(pdptr)) {
 				vcpu->arch.mmu.pae_root[i] = 0;
 				continue;
 			}
 			root_gfn = pdptr >> PAGE_SHIFT;
 		} else if (vcpu->arch.mmu.root_level == 0)
 			root_gfn = 0;
 		if (mmu_check_root(vcpu, root_gfn))
 			return 1;
 		if (tdp_enabled) {
 			direct = 1;
 			root_gfn = i << 30;
 		}
 		spin_lock(&vcpu->kvm->mmu_lock);
 		kvm_mmu_free_some_pages(vcpu);
 		sp = kvm_mmu_get_page(vcpu, root_gfn, i << 30,
 				      PT32_ROOT_LEVEL, direct,
 				      ACC_ALL, NULL);
 		root = __pa(sp->spt);
 		++sp->root_count;
 		spin_unlock(&vcpu->kvm->mmu_lock);
 		vcpu->arch.mmu.pae_root[i] = root | PT_PRESENT_MASK;
 	}
 	vcpu->arch.mmu.root_hpa = __pa(vcpu->arch.mmu.pae_root);
 	return 0;
 }
 static void mmu_sync_roots(struct kvm_vcpu *vcpu)
 {
 	int i;
 	struct kvm_mmu_page *sp;
 	if (!VALID_PAGE(vcpu->arch.mmu.root_hpa))
 		return;
 	if (vcpu->arch.mmu.shadow_root_level == PT64_ROOT_LEVEL) {
 		hpa_t root = vcpu->arch.mmu.root_hpa;
 		sp = page_header(root);
 		mmu_sync_children(vcpu, sp);
 		return;
 	}
 	for (i = 0; i < 4; ++i) {
 		hpa_t root = vcpu->arch.mmu.pae_root[i];
 		if (root && VALID_PAGE(root)) {
 			root &= PT64_BASE_ADDR_MASK;
 			sp = page_header(root);
 			mmu_sync_children(vcpu, sp);
 		}
 	}
 }
 void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu)
 {
 	spin_lock(&vcpu->kvm->mmu_lock);
 	mmu_sync_roots(vcpu);
 	spin_unlock(&vcpu->kvm->mmu_lock);
 }
 static gpa_t nonpaging_gva_to_gpa(struct kvm_vcpu *vcpu, gva_t vaddr,
 				  u32 access, u32 *error)
 {
 	if (error)
 		*error = 0;
 	return vaddr;
 }
 static int nonpaging_page_fault(struct kvm_vcpu *vcpu, gva_t gva,
 				u32 error_code)
 {
 	gfn_t gfn;
 	int r;
 	pgprintk("%s: gva %lx error %x\n", __func__, gva, error_code);
 	r = mmu_topup_memory_caches(vcpu);
 	if (r)
 		return r;
 	ASSERT(vcpu);
 	ASSERT(VALID_PAGE(vcpu->arch.mmu.root_hpa));
 	gfn = gva >> PAGE_SHIFT;
 	return nonpaging_map(vcpu, gva & PAGE_MASK,
 			     error_code & PFERR_WRITE_MASK, gfn);
 }
 static int tdp_page_fault(struct kvm_vcpu *vcpu, gva_t gpa,
 				u32 error_code)
 {
 	pfn_t pfn;
 	int r;
 	int level;
 	gfn_t gfn = gpa >> PAGE_SHIFT;
 	unsigned long mmu_seq;
 	ASSERT(vcpu);
 	ASSERT(VALID_PAGE(vcpu->arch.mmu.root_hpa));
 	r = mmu_topup_memory_caches(vcpu);
 	if (r)
 		return r;
 	level = mapping_level(vcpu, gfn);
 	gfn &= ~(KVM_PAGES_PER_HPAGE(level) - 1);
 	mmu_seq = vcpu->kvm->mmu_notifier_seq;
 	smp_rmb();
 	pfn = gfn_to_pfn(vcpu->kvm, gfn);
 	if (is_error_pfn(pfn))
 		return kvm_handle_bad_page(vcpu->kvm, gfn, pfn);
 	spin_lock(&vcpu->kvm->mmu_lock);
 	if (mmu_notifier_retry(vcpu, mmu_seq))
 		goto out_unlock;
 	kvm_mmu_free_some_pages(vcpu);
 	r = __direct_map(vcpu, gpa, error_code & PFERR_WRITE_MASK,
 			 level, gfn, pfn);
 	spin_unlock(&vcpu->kvm->mmu_lock);
 	return r;
 out_unlock:
 	spin_unlock(&vcpu->kvm->mmu_lock);
 	kvm_release_pfn_clean(pfn);
 	return 0;
 }
 static void nonpaging_free(struct kvm_vcpu *vcpu)
 {
 	mmu_free_roots(vcpu);
 }
 static int nonpaging_init_context(struct kvm_vcpu *vcpu)
 {
 	struct kvm_mmu *context = &vcpu->arch.mmu;
 	context->new_cr3 = nonpaging_new_cr3;
 	context->page_fault = nonpaging_page_fault;
 	context->gva_to_gpa = nonpaging_gva_to_gpa;
 	context->free = nonpaging_free;
 	context->prefetch_page = nonpaging_prefetch_page;
 	context->sync_page = nonpaging_sync_page;
 	context->invlpg = nonpaging_invlpg;
 	context->root_level = 0;
 	context->shadow_root_level = PT32E_ROOT_LEVEL;
 	context->root_hpa = INVALID_PAGE;
 	return 0;
 }
 void kvm_mmu_flush_tlb(struct kvm_vcpu *vcpu)
 {
 	++vcpu->stat.tlb_flush;
 	set_bit(KVM_REQ_TLB_FLUSH, &vcpu->requests);
 }
 static void paging_new_cr3(struct kvm_vcpu *vcpu)
 {
 	pgprintk("%s: cr3 %lx\n", __func__, vcpu->arch.cr3);
 	mmu_free_roots(vcpu);
 }
 static void inject_page_fault(struct kvm_vcpu *vcpu,
 			      u64 addr,
 			      u32 err_code)
 {
 	kvm_inject_page_fault(vcpu, addr, err_code);
 }
 static void paging_free(struct kvm_vcpu *vcpu)
 {
 	nonpaging_free(vcpu);
 }
 static bool is_rsvd_bits_set(struct kvm_vcpu *vcpu, u64 gpte, int level)
 {
 	int bit7;
 	bit7 = (gpte >> 7) & 1;
 	return (gpte & vcpu->arch.mmu.rsvd_bits_mask[bit7][level-1]) != 0;
 }
 #define PTTYPE 64
 #include "paging_tmpl.h"
 #undef PTTYPE
 #define PTTYPE 32
 #include "paging_tmpl.h"
 #undef PTTYPE
 static void reset_rsvds_bits_mask(struct kvm_vcpu *vcpu, int level)
 {
 	struct kvm_mmu *context = &vcpu->arch.mmu;
 	int maxphyaddr = cpuid_maxphyaddr(vcpu);
 	u64 exb_bit_rsvd = 0;
 	if (!is_nx(vcpu))
 		exb_bit_rsvd = rsvd_bits(63, 63);
 	switch (level) {
 	case PT32_ROOT_LEVEL:
 		/* no rsvd bits for 2 level 4K page table entries */
 		context->rsvd_bits_mask[0][1] = 0;
 		context->rsvd_bits_mask[0][0] = 0;
 		context->rsvd_bits_mask[1][0] = context->rsvd_bits_mask[0][0];
 		if (!is_pse(vcpu)) {
 			context->rsvd_bits_mask[1][1] = 0;
 			break;
 		}
 		if (is_cpuid_PSE36())
 			/* 36bits PSE 4MB page */
 			context->rsvd_bits_mask[1][1] = rsvd_bits(17, 21);
 		else
 			/* 32 bits PSE 4MB page */
 			context->rsvd_bits_mask[1][1] = rsvd_bits(13, 21);
 		break;
 	case PT32E_ROOT_LEVEL:
 		context->rsvd_bits_mask[0][2] =
 			rsvd_bits(maxphyaddr, 63) |
 			rsvd_bits(7, 8) | rsvd_bits(1, 2);	/* PDPTE */
 		context->rsvd_bits_mask[0][1] = exb_bit_rsvd |
 			rsvd_bits(maxphyaddr, 62);	/* PDE */
 		context->rsvd_bits_mask[0][0] = exb_bit_rsvd |
 			rsvd_bits(maxphyaddr, 62); 	/* PTE */
 		context->rsvd_bits_mask[1][1] = exb_bit_rsvd |
 			rsvd_bits(maxphyaddr, 62) |
 			rsvd_bits(13, 20);		/* large page */
 		context->rsvd_bits_mask[1][0] = context->rsvd_bits_mask[0][0];
 		break;
 	case PT64_ROOT_LEVEL:
 		context->rsvd_bits_mask[0][3] = exb_bit_rsvd |
 			rsvd_bits(maxphyaddr, 51) | rsvd_bits(7, 8);
 		context->rsvd_bits_mask[0][2] = exb_bit_rsvd |
 			rsvd_bits(maxphyaddr, 51) | rsvd_bits(7, 8);
 		context->rsvd_bits_mask[0][1] = exb_bit_rsvd |
 			rsvd_bits(maxphyaddr, 51);
 		context->rsvd_bits_mask[0][0] = exb_bit_rsvd |
 			rsvd_bits(maxphyaddr, 51);
 		context->rsvd_bits_mask[1][3] = context->rsvd_bits_mask[0][3];
 		context->rsvd_bits_mask[1][2] = exb_bit_rsvd |
 			rsvd_bits(maxphyaddr, 51) |
 			rsvd_bits(13, 29);
 		context->rsvd_bits_mask[1][1] = exb_bit_rsvd |
 			rsvd_bits(maxphyaddr, 51) |
 			rsvd_bits(13, 20);		/* large page */
 		context->rsvd_bits_mask[1][0] = context->rsvd_bits_mask[0][0];
 		break;
 	}
 }
 static int paging64_init_context_common(struct kvm_vcpu *vcpu, int level)
 {
 	struct kvm_mmu *context = &vcpu->arch.mmu;
 	ASSERT(is_pae(vcpu));
 	context->new_cr3 = paging_new_cr3;
 	context->page_fault = paging64_page_fault;
 	context->gva_to_gpa = paging64_gva_to_gpa;
 	context->prefetch_page = paging64_prefetch_page;
 	context->sync_page = paging64_sync_page;
 	context->invlpg = paging64_invlpg;
 	context->free = paging_free;
 	context->root_level = level;
 	context->shadow_root_level = level;
 	context->root_hpa = INVALID_PAGE;
 	return 0;
 }
 static int paging64_init_context(struct kvm_vcpu *vcpu)
 {
 	reset_rsvds_bits_mask(vcpu, PT64_ROOT_LEVEL);
 	return paging64_init_context_common(vcpu, PT64_ROOT_LEVEL);
 }
 static int paging32_init_context(struct kvm_vcpu *vcpu)
 {
 	struct kvm_mmu *context = &vcpu->arch.mmu;
 	reset_rsvds_bits_mask(vcpu, PT32_ROOT_LEVEL);
 	context->new_cr3 = paging_new_cr3;
 	context->page_fault = paging32_page_fault;
 	context->gva_to_gpa = paging32_gva_to_gpa;
 	context->free = paging_free;
 	context->prefetch_page = paging32_prefetch_page;
 	context->sync_page = paging32_sync_page;
 	context->invlpg = paging32_invlpg;
 	context->root_level = PT32_ROOT_LEVEL;
 	context->shadow_root_level = PT32E_ROOT_LEVEL;
 	context->root_hpa = INVALID_PAGE;
 	return 0;
 }
 static int paging32E_init_context(struct kvm_vcpu *vcpu)
 {
 	reset_rsvds_bits_mask(vcpu, PT32E_ROOT_LEVEL);
 	return paging64_init_context_common(vcpu, PT32E_ROOT_LEVEL);
 }
 static int init_kvm_tdp_mmu(struct kvm_vcpu *vcpu)
 {
 	struct kvm_mmu *context = &vcpu->arch.mmu;
 	context->new_cr3 = nonpaging_new_cr3;
 	context->page_fault = tdp_page_fault;
 	context->free = nonpaging_free;
 	context->prefetch_page = nonpaging_prefetch_page;
 	context->sync_page = nonpaging_sync_page;
 	context->invlpg = nonpaging_invlpg;
 	context->shadow_root_level = kvm_x86_ops->get_tdp_level();
 	context->root_hpa = INVALID_PAGE;
 	if (!is_paging(vcpu)) {
 		context->gva_to_gpa = nonpaging_gva_to_gpa;
 		context->root_level = 0;
 	} else if (is_long_mode(vcpu)) {
 		reset_rsvds_bits_mask(vcpu, PT64_ROOT_LEVEL);
 		context->gva_to_gpa = paging64_gva_to_gpa;
 		context->root_level = PT64_ROOT_LEVEL;
 	} else if (is_pae(vcpu)) {
 		reset_rsvds_bits_mask(vcpu, PT32E_ROOT_LEVEL);
 		context->gva_to_gpa = paging64_gva_to_gpa;
 		context->root_level = PT32E_ROOT_LEVEL;
 	} else {
 		reset_rsvds_bits_mask(vcpu, PT32_ROOT_LEVEL);
 		context->gva_to_gpa = paging32_gva_to_gpa;
 		context->root_level = PT32_ROOT_LEVEL;
 	}
 	return 0;
 }
 static int init_kvm_softmmu(struct kvm_vcpu *vcpu)
 {
 	int r;
 	ASSERT(vcpu);
 	ASSERT(!VALID_PAGE(vcpu->arch.mmu.root_hpa));
 	if (!is_paging(vcpu))
 		r = nonpaging_init_context(vcpu);
 	else if (is_long_mode(vcpu))
 		r = paging64_init_context(vcpu);
 	else if (is_pae(vcpu))
 		r = paging32E_init_context(vcpu);
 	else
 		r = paging32_init_context(vcpu);
 	vcpu->arch.mmu.base_role.cr4_pae = !!is_pae(vcpu);
 	vcpu->arch.mmu.base_role.cr0_wp = is_write_protection(vcpu);
 	return r;
 }
 static int init_kvm_mmu(struct kvm_vcpu *vcpu)
 {
 	vcpu->arch.update_pte.pfn = bad_pfn;
 	if (tdp_enabled)
 		return init_kvm_tdp_mmu(vcpu);
 	else
 		return init_kvm_softmmu(vcpu);
 }
 static void destroy_kvm_mmu(struct kvm_vcpu *vcpu)
 {
 	ASSERT(vcpu);
 	if (VALID_PAGE(vcpu->arch.mmu.root_hpa))
 		/* mmu.free() should set root_hpa = INVALID_PAGE */
 		vcpu->arch.mmu.free(vcpu);
 }
 int kvm_mmu_reset_context(struct kvm_vcpu *vcpu)
 {
 	destroy_kvm_mmu(vcpu);
 	return init_kvm_mmu(vcpu);
 }
 EXPORT_SYMBOL_GPL(kvm_mmu_reset_context);
 int kvm_mmu_load(struct kvm_vcpu *vcpu)
 {
 	int r;
 	r = mmu_topup_memory_caches(vcpu);
 	if (r)
 		goto out;
 	r = mmu_alloc_roots(vcpu);
 	spin_lock(&vcpu->kvm->mmu_lock);
 	mmu_sync_roots(vcpu);
 	spin_unlock(&vcpu->kvm->mmu_lock);
 	if (r)
 		goto out;
 	/* set_cr3() should ensure TLB has been flushed */
 	kvm_x86_ops->set_cr3(vcpu, vcpu->arch.mmu.root_hpa);
 out:
 	return r;
 }
 EXPORT_SYMBOL_GPL(kvm_mmu_load);
 void kvm_mmu_unload(struct kvm_vcpu *vcpu)
 {
 	mmu_free_roots(vcpu);
 }
 static void mmu_pte_write_zap_pte(struct kvm_vcpu *vcpu,
 				  struct kvm_mmu_page *sp,
 				  u64 *spte)
 {
 	u64 pte;
 	struct kvm_mmu_page *child;
 	pte = *spte;
 	if (is_shadow_present_pte(pte)) {
 		if (is_last_spte(pte, sp->role.level))
 			rmap_remove(vcpu->kvm, spte);
 		else {
 			child = page_header(pte & PT64_BASE_ADDR_MASK);
 			mmu_page_remove_parent_pte(child, spte);
 		}
 	}
 	__set_spte(spte, shadow_trap_nonpresent_pte);
 	if (is_large_pte(pte))
 		--vcpu->kvm->stat.lpages;
 }
 static void mmu_pte_write_new_pte(struct kvm_vcpu *vcpu,
 				  struct kvm_mmu_page *sp,
 				  u64 *spte,
 				  const void *new)
 {
 	if (sp->role.level != PT_PAGE_TABLE_LEVEL) {
 		++vcpu->kvm->stat.mmu_pde_zapped;
 		return;
         }
 	++vcpu->kvm->stat.mmu_pte_updated;
 	if (!sp->role.cr4_pae)
 		paging32_update_pte(vcpu, sp, spte, new);
 	else
 		paging64_update_pte(vcpu, sp, spte, new);
 }
 static bool need_remote_flush(u64 old, u64 new)
 {
 	if (!is_shadow_present_pte(old))
 		return false;
 	if (!is_shadow_present_pte(new))
 		return true;
 	if ((old ^ new) & PT64_BASE_ADDR_MASK)
 		return true;
 	old ^= PT64_NX_MASK;
 	new ^= PT64_NX_MASK;
 	return (old & ~new & PT64_PERM_MASK) != 0;
 }
 static void mmu_pte_write_flush_tlb(struct kvm_vcpu *vcpu, bool zap_page,
 				    bool remote_flush, bool local_flush)
 {
 	if (zap_page)
 		return;
 	if (remote_flush)
 		kvm_flush_remote_tlbs(vcpu->kvm);
 	else if (local_flush)
 		kvm_mmu_flush_tlb(vcpu);
 }
 static bool last_updated_pte_accessed(struct kvm_vcpu *vcpu)
 {
 	u64 *spte = vcpu->arch.last_pte_updated;
 	return !!(spte && (*spte & shadow_accessed_mask));
 }
 static void mmu_guess_page_from_pte_write(struct kvm_vcpu *vcpu, gpa_t gpa,
 					  u64 gpte)
 {
 	gfn_t gfn;
 	pfn_t pfn;
 	if (!is_present_gpte(gpte))
 		return;
 	gfn = (gpte & PT64_BASE_ADDR_MASK) >> PAGE_SHIFT;
 	vcpu->arch.update_pte.mmu_seq = vcpu->kvm->mmu_notifier_seq;
 	smp_rmb();
 	pfn = gfn_to_pfn(vcpu->kvm, gfn);
 	if (is_error_pfn(pfn)) {
 		kvm_release_pfn_clean(pfn);
 		return;
 	}
 	vcpu->arch.update_pte.gfn = gfn;
 	vcpu->arch.update_pte.pfn = pfn;
 }
 static void kvm_mmu_access_page(struct kvm_vcpu *vcpu, gfn_t gfn)
 {
 	u64 *spte = vcpu->arch.last_pte_updated;
 	if (spte
 	    && vcpu->arch.last_pte_gfn == gfn
 	    && shadow_accessed_mask
 	    && !(*spte & shadow_accessed_mask)
 	    && is_shadow_present_pte(*spte))
 		set_bit(PT_ACCESSED_SHIFT, (unsigned long *)spte);
 }
 void kvm_mmu_pte_write(struct kvm_vcpu *vcpu, gpa_t gpa,
 		       const u8 *new, int bytes,
 		       bool guest_initiated)
 {
 	gfn_t gfn = gpa >> PAGE_SHIFT;
 	struct kvm_mmu_page *sp;
 	struct hlist_node *node;
 	LIST_HEAD(invalid_list);
 	u64 entry, gentry;
 	u64 *spte;
 	unsigned offset = offset_in_page(gpa);
 	unsigned pte_size;
 	unsigned page_offset;
 	unsigned misaligned;
 	unsigned quadrant;
 	int level;
 	int flooded = 0;
 	int npte;
 	int r;
 	int invlpg_counter;
 	bool remote_flush, local_flush, zap_page;
 	zap_page = remote_flush = local_flush = false;
 	pgprintk("%s: gpa %llx bytes %d\n", __func__, gpa, bytes);
 	invlpg_counter = atomic_read(&vcpu->kvm->arch.invlpg_counter);
 	/*
 	 * Assume that the pte write on a page table of the same type
 	 * as the current vcpu paging mode.  This is nearly always true
 	 * (might be false while changing modes).  Note it is verified later
 	 * by update_pte().
 	 */
 	if ((is_pae(vcpu) && bytes == 4) || !new) {
 		/* Handle a 32-bit guest writing two halves of a 64-bit gpte */
 		if (is_pae(vcpu)) {
 			gpa &= ~(gpa_t)7;
 			bytes = 8;
 		}
 		r = kvm_read_guest(vcpu->kvm, gpa, &gentry, min(bytes, 8));
 		if (r)
 			gentry = 0;
 		new = (const u8 *)&gentry;
 	}
 	switch (bytes) {
 	case 4:
 		gentry = *(const u32 *)new;
 		break;
 	case 8:
 		gentry = *(const u64 *)new;
 		break;
 	default:
 		gentry = 0;
 		break;
 	}
 	mmu_guess_page_from_pte_write(vcpu, gpa, gentry);
 	spin_lock(&vcpu->kvm->mmu_lock);
 	if (atomic_read(&vcpu->kvm->arch.invlpg_counter) != invlpg_counter)
 		gentry = 0;
 	kvm_mmu_access_page(vcpu, gfn);
 	kvm_mmu_free_some_pages(vcpu);
 	++vcpu->kvm->stat.mmu_pte_write;
 	kvm_mmu_audit(vcpu, "pre pte write");
 	if (guest_initiated) {
 		if (gfn == vcpu->arch.last_pt_write_gfn
 		    && !last_updated_pte_accessed(vcpu)) {
 			++vcpu->arch.last_pt_write_count;
 			if (vcpu->arch.last_pt_write_count >= 3)
 				flooded = 1;
 		} else {
 			vcpu->arch.last_pt_write_gfn = gfn;
 			vcpu->arch.last_pt_write_count = 1;
 			vcpu->arch.last_pte_updated = NULL;
 		}
 	}
 	for_each_gfn_indirect_valid_sp(vcpu->kvm, sp, gfn, node) {
 		pte_size = sp->role.cr4_pae ? 8 : 4;
 		misaligned = (offset ^ (offset + bytes - 1)) & ~(pte_size - 1);
 		misaligned |= bytes < 4;
 		if (misaligned || flooded) {
 			/*
 			 * Misaligned accesses are too much trouble to fix
 			 * up; also, they usually indicate a page is not used
 			 * as a page table.
 			 *
 			 * If we're seeing too many writes to a page,
 			 * it may no longer be a page table, or we may be
 			 * forking, in which case it is better to unmap the
 			 * page.
 			 */
 			pgprintk("misaligned: gpa %llx bytes %d role %x\n",
 				 gpa, bytes, sp->role.word);
 			zap_page |= !!kvm_mmu_prepare_zap_page(vcpu->kvm, sp,
 						     &invalid_list);
 			++vcpu->kvm->stat.mmu_flooded;
 			continue;
 		}
 		page_offset = offset;
 		level = sp->role.level;
 		npte = 1;
 		if (!sp->role.cr4_pae) {
 			page_offset <<= 1;	/* 32->64 */
 			/*
 			 * A 32-bit pde maps 4MB while the shadow pdes map
 			 * only 2MB.  So we need to double the offset again
 			 * and zap two pdes instead of one.
 			 */
 			if (level == PT32_ROOT_LEVEL) {
 				page_offset &= ~7; /* kill rounding error */
 				page_offset <<= 1;
 				npte = 2;
 			}
 			quadrant = page_offset >> PAGE_SHIFT;
 			page_offset &= ~PAGE_MASK;
 			if (quadrant != sp->role.quadrant)
 				continue;
 		}
 		local_flush = true;
 		spte = &sp->spt[page_offset / sizeof(*spte)];
 		while (npte--) {
 			entry = *spte;
 			mmu_pte_write_zap_pte(vcpu, sp, spte);
 			if (gentry)
 				mmu_pte_write_new_pte(vcpu, sp, spte, &gentry);
 			if (!remote_flush && need_remote_flush(entry, *spte))
 				remote_flush = true;
 			++spte;
 		}
 	}
 	mmu_pte_write_flush_tlb(vcpu, zap_page, remote_flush, local_flush);
 	kvm_mmu_commit_zap_page(vcpu->kvm, &invalid_list);
 	kvm_mmu_audit(vcpu, "post pte write");
 	spin_unlock(&vcpu->kvm->mmu_lock);
 	if (!is_error_pfn(vcpu->arch.update_pte.pfn)) {
 		kvm_release_pfn_clean(vcpu->arch.update_pte.pfn);
 		vcpu->arch.update_pte.pfn = bad_pfn;
 	}
 }
 int kvm_mmu_unprotect_page_virt(struct kvm_vcpu *vcpu, gva_t gva)
 {
 	gpa_t gpa;
 	int r;
 	if (tdp_enabled)
 		return 0;
 	gpa = kvm_mmu_gva_to_gpa_read(vcpu, gva, NULL);
 	spin_lock(&vcpu->kvm->mmu_lock);
 	r = kvm_mmu_unprotect_page(vcpu->kvm, gpa >> PAGE_SHIFT);
 	spin_unlock(&vcpu->kvm->mmu_lock);
 	return r;
 }
 EXPORT_SYMBOL_GPL(kvm_mmu_unprotect_page_virt);
 void __kvm_mmu_free_some_pages(struct kvm_vcpu *vcpu)
 {
 	int free_pages;
 	LIST_HEAD(invalid_list);
 	free_pages = vcpu->kvm->arch.n_free_mmu_pages;
 	while (free_pages < KVM_REFILL_PAGES &&
 	       !list_empty(&vcpu->kvm->arch.active_mmu_pages)) {
 		struct kvm_mmu_page *sp;
 		sp = container_of(vcpu->kvm->arch.active_mmu_pages.prev,
 				  struct kvm_mmu_page, link);
 		free_pages += kvm_mmu_prepare_zap_page(vcpu->kvm, sp,
 						       &invalid_list);
 		++vcpu->kvm->stat.mmu_recycled;
 	}
 	kvm_mmu_commit_zap_page(vcpu->kvm, &invalid_list);
 }
 int kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gva_t cr2, u32 error_code)
 {
 	int r;
 	enum emulation_result er;
 	r = vcpu->arch.mmu.page_fault(vcpu, cr2, error_code);
 	if (r < 0)
 		goto out;
 	if (!r) {
 		r = 1;
 		goto out;
 	}
 	r = mmu_topup_memory_caches(vcpu);
 	if (r)
 		goto out;
 	er = emulate_instruction(vcpu, cr2, error_code, 0);
 	switch (er) {
 	case EMULATE_DONE:
 		return 1;
 	case EMULATE_DO_MMIO:
 		++vcpu->stat.mmio_exits;
 		/* fall through */
 	case EMULATE_FAIL:
 		return 0;
 	default:
 		BUG();
 	}
 out:
 	return r;
 }
 EXPORT_SYMBOL_GPL(kvm_mmu_page_fault);
 void kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva)
 {
 	vcpu->arch.mmu.invlpg(vcpu, gva);
 	kvm_mmu_flush_tlb(vcpu);
 	++vcpu->stat.invlpg;
 }
 EXPORT_SYMBOL_GPL(kvm_mmu_invlpg);
 void kvm_enable_tdp(void)
 {
 	tdp_enabled = true;
 }
 EXPORT_SYMBOL_GPL(kvm_enable_tdp);
 void kvm_disable_tdp(void)
 {
 	tdp_enabled = false;
 }
 EXPORT_SYMBOL_GPL(kvm_disable_tdp);
 static void free_mmu_pages(struct kvm_vcpu *vcpu)
 {
 	free_page((unsigned long)vcpu->arch.mmu.pae_root);
 }
 static int alloc_mmu_pages(struct kvm_vcpu *vcpu)
 {
 	struct page *page;
 	int i;
 	ASSERT(vcpu);
 	/*
 	 * When emulating 32-bit mode, cr3 is only 32 bits even on x86_64.
 	 * Therefore we need to allocate shadow page tables in the first
 	 * 4GB of memory, which happens to fit the DMA32 zone.
 	 */
 	page = alloc_page(GFP_KERNEL | __GFP_DMA32);
 	if (!page)
 		return -ENOMEM;
 	vcpu->arch.mmu.pae_root = page_address(page);
 	for (i = 0; i < 4; ++i)
 		vcpu->arch.mmu.pae_root[i] = INVALID_PAGE;
 	return 0;
 }
 int kvm_mmu_create(struct kvm_vcpu *vcpu)
 {
 	ASSERT(vcpu);
 	ASSERT(!VALID_PAGE(vcpu->arch.mmu.root_hpa));
 	return alloc_mmu_pages(vcpu);
 }
 int kvm_mmu_setup(struct kvm_vcpu *vcpu)
 {
 	ASSERT(vcpu);
 	ASSERT(!VALID_PAGE(vcpu->arch.mmu.root_hpa));
 	return init_kvm_mmu(vcpu);
 }
 void kvm_mmu_destroy(struct kvm_vcpu *vcpu)
 {
 	ASSERT(vcpu);
 	destroy_kvm_mmu(vcpu);
 	free_mmu_pages(vcpu);
 	mmu_free_memory_caches(vcpu);
 }
 void kvm_mmu_slot_remove_write_access(struct kvm *kvm, int slot)
 {
 	struct kvm_mmu_page *sp;
 	list_for_each_entry(sp, &kvm->arch.active_mmu_pages, link) {
 		int i;
 		u64 *pt;
 		if (!test_bit(slot, sp->slot_bitmap))
 			continue;
 		pt = sp->spt;
 		for (i = 0; i < PT64_ENT_PER_PAGE; ++i)
 			/* avoid RMW */
 			if (is_writable_pte(pt[i]))
 				pt[i] &= ~PT_WRITABLE_MASK;
 	}
 	kvm_flush_remote_tlbs(kvm);
 }
 void kvm_mmu_zap_all(struct kvm *kvm)
 {
 	struct kvm_mmu_page *sp, *node;
 	LIST_HEAD(invalid_list);
 	spin_lock(&kvm->mmu_lock);
 restart:
 	list_for_each_entry_safe(sp, node, &kvm->arch.active_mmu_pages, link)
 		if (kvm_mmu_prepare_zap_page(kvm, sp, &invalid_list))
 			goto restart;
 	kvm_mmu_commit_zap_page(kvm, &invalid_list);
 	spin_unlock(&kvm->mmu_lock);
 }
 static int kvm_mmu_remove_some_alloc_mmu_pages(struct kvm *kvm,
 					       struct list_head *invalid_list)
 {
 	struct kvm_mmu_page *page;
 	page = container_of(kvm->arch.active_mmu_pages.prev,
 			    struct kvm_mmu_page, link);
 	return kvm_mmu_prepare_zap_page(kvm, page, invalid_list);
 }
 static int mmu_shrink(struct shrinker *shrink, int nr_to_scan, gfp_t gfp_mask)
 {
 	struct kvm *kvm;
 	struct kvm *kvm_freed = NULL;
 	int cache_count = 0;
 	spin_lock(&kvm_lock);
 	list_for_each_entry(kvm, &vm_list, vm_list) {
 		int npages, idx, freed_pages;
 		LIST_HEAD(invalid_list);
 		idx = srcu_read_lock(&kvm->srcu);
 		spin_lock(&kvm->mmu_lock);
 		npages = kvm->arch.n_alloc_mmu_pages -
 			 kvm->arch.n_free_mmu_pages;
 		cache_count += npages;
 		if (!kvm_freed && nr_to_scan > 0 && npages > 0) {
 			freed_pages = kvm_mmu_remove_some_alloc_mmu_pages(kvm,
 							  &invalid_list);
 			cache_count -= freed_pages;
 			kvm_freed = kvm;
 		}
 		nr_to_scan--;
 		kvm_mmu_commit_zap_page(kvm, &invalid_list);
 		spin_unlock(&kvm->mmu_lock);
 		srcu_read_unlock(&kvm->srcu, idx);
 	}
 	if (kvm_freed)
 		list_move_tail(&kvm_freed->vm_list, &vm_list);
 	spin_unlock(&kvm_lock);
 	return cache_count;
 }
 static struct shrinker mmu_shrinker = {
 	.shrink = mmu_shrink,
 	.seeks = DEFAULT_SEEKS * 10,
 };
 static void mmu_destroy_caches(void)
 {
 	if (pte_chain_cache)
 		kmem_cache_destroy(pte_chain_cache);
 	if (rmap_desc_cache)
 		kmem_cache_destroy(rmap_desc_cache);
 	if (mmu_page_header_cache)
 		kmem_cache_destroy(mmu_page_header_cache);
 }
 void kvm_mmu_module_exit(void)
 {
 	mmu_destroy_caches();
 	unregister_shrinker(&mmu_shrinker);
 }
 int kvm_mmu_module_init(void)
 {
 	pte_chain_cache = kmem_cache_create("kvm_pte_chain",
 					    sizeof(struct kvm_pte_chain),
 					    0, 0, NULL);
 	if (!pte_chain_cache)
 		goto nomem;
 	rmap_desc_cache = kmem_cache_create("kvm_rmap_desc",
 					    sizeof(struct kvm_rmap_desc),
 					    0, 0, NULL);
 	if (!rmap_desc_cache)
 		goto nomem;
 	mmu_page_header_cache = kmem_cache_create("kvm_mmu_page_header",
 						  sizeof(struct kvm_mmu_page),
 						  0, 0, NULL);
 	if (!mmu_page_header_cache)
 		goto nomem;
 	register_shrinker(&mmu_shrinker);
 	return 0;
 nomem:
 	mmu_destroy_caches();
 	return -ENOMEM;
 }
 /*
  * Caculate mmu pages needed for kvm.
  */
 unsigned int kvm_mmu_calculate_mmu_pages(struct kvm *kvm)
 {
 	int i;
 	unsigned int nr_mmu_pages;
 	unsigned int  nr_pages = 0;
 	struct kvm_memslots *slots;
 	slots = kvm_memslots(kvm);
 	for (i = 0; i < slots->nmemslots; i++)
 		nr_pages += slots->memslots[i].npages;
 	nr_mmu_pages = nr_pages * KVM_PERMILLE_MMU_PAGES / 1000;
 	nr_mmu_pages = max(nr_mmu_pages,
 			(unsigned int) KVM_MIN_ALLOC_MMU_PAGES);
 	return nr_mmu_pages;
 }
 static void *pv_mmu_peek_buffer(struct kvm_pv_mmu_op_buffer *buffer,
 				unsigned len)
 {
 	if (len > buffer->len)
 		return NULL;
 	return buffer->ptr;
 }
 static void *pv_mmu_read_buffer(struct kvm_pv_mmu_op_buffer *buffer,
 				unsigned len)
 {
 	void *ret;
 	ret = pv_mmu_peek_buffer(buffer, len);
 	if (!ret)
 		return ret;
 	buffer->ptr += len;
 	buffer->len -= len;
 	buffer->processed += len;
 	return ret;
 }
 static int kvm_pv_mmu_write(struct kvm_vcpu *vcpu,
 			     gpa_t addr, gpa_t value)
 {
 	int bytes = 8;
 	int r;
 	if (!is_long_mode(vcpu) && !is_pae(vcpu))
 		bytes = 4;
 	r = mmu_topup_memory_caches(vcpu);
 	if (r)
 		return r;
 	if (!emulator_write_phys(vcpu, addr, &value, bytes))
 		return -EFAULT;
 	return 1;
 }
 static int kvm_pv_mmu_flush_tlb(struct kvm_vcpu *vcpu)
 {
 	(void)kvm_set_cr3(vcpu, vcpu->arch.cr3);
 	return 1;
 }
 static int kvm_pv_mmu_release_pt(struct kvm_vcpu *vcpu, gpa_t addr)
 {
 	spin_lock(&vcpu->kvm->mmu_lock);
 	mmu_unshadow(vcpu->kvm, addr >> PAGE_SHIFT);
 	spin_unlock(&vcpu->kvm->mmu_lock);
 	return 1;
 }
 static int kvm_pv_mmu_op_one(struct kvm_vcpu *vcpu,
 			     struct kvm_pv_mmu_op_buffer *buffer)
 {
 	struct kvm_mmu_op_header *header;
 	header = pv_mmu_peek_buffer(buffer, sizeof *header);
 	if (!header)
 		return 0;
 	switch (header->op) {
 	case KVM_MMU_OP_WRITE_PTE: {
 		struct kvm_mmu_op_write_pte *wpte;
 		wpte = pv_mmu_read_buffer(buffer, sizeof *wpte);
 		if (!wpte)
 			return 0;
 		return kvm_pv_mmu_write(vcpu, wpte->pte_phys,
 					wpte->pte_val);
 	}
 	case KVM_MMU_OP_FLUSH_TLB: {
 		struct kvm_mmu_op_flush_tlb *ftlb;
 		ftlb = pv_mmu_read_buffer(buffer, sizeof *ftlb);
 		if (!ftlb)
 			return 0;
 		return kvm_pv_mmu_flush_tlb(vcpu);
 	}
 	case KVM_MMU_OP_RELEASE_PT: {
 		struct kvm_mmu_op_release_pt *rpt;
 		rpt = pv_mmu_read_buffer(buffer, sizeof *rpt);
 		if (!rpt)
 			return 0;
 		return kvm_pv_mmu_release_pt(vcpu, rpt->pt_phys);
 	}
 	default: return 0;
 	}
 }
 int kvm_pv_mmu_op(struct kvm_vcpu *vcpu, unsigned long bytes,
 		  gpa_t addr, unsigned long *ret)
 {
 	int r;
 	struct kvm_pv_mmu_op_buffer *buffer = &vcpu->arch.mmu_op_buffer;
 	buffer->ptr = buffer->buf;
 	buffer->len = min_t(unsigned long, bytes, sizeof buffer->buf);
 	buffer->processed = 0;
 	r = kvm_read_guest(vcpu->kvm, addr, buffer->buf, buffer->len);
 	if (r)
 		goto out;
 	while (buffer->len) {
 		r = kvm_pv_mmu_op_one(vcpu, buffer);
 		if (r < 0)
 			goto out;
 		if (r == 0)
 			break;
 	}
 	r = 1;
 out:
 	*ret = buffer->processed;
 	return r;
 }
 int kvm_mmu_get_spte_hierarchy(struct kvm_vcpu *vcpu, u64 addr, u64 sptes[4])
 {
 	struct kvm_shadow_walk_iterator iterator;
 	int nr_sptes = 0;
 	spin_lock(&vcpu->kvm->mmu_lock);
 	for_each_shadow_entry(vcpu, addr, iterator) {
 		sptes[iterator.level-1] = *iterator.sptep;
 		nr_sptes++;
 		if (!is_shadow_present_pte(*iterator.sptep))
 			break;
 	}
 	spin_unlock(&vcpu->kvm->mmu_lock);
 	return nr_sptes;
 }
 EXPORT_SYMBOL_GPL(kvm_mmu_get_spte_hierarchy);
 #ifdef AUDIT
 static const char *audit_msg;
 static gva_t canonicalize(gva_t gva)
 {
 #ifdef CONFIG_X86_64
 	gva = (long long)(gva << 16) >> 16;
 #endif
 	return gva;
 }
 typedef void (*inspect_spte_fn) (struct kvm *kvm, u64 *sptep);
 static void __mmu_spte_walk(struct kvm *kvm, struct kvm_mmu_page *sp,
 			    inspect_spte_fn fn)
 {
 	int i;
 	for (i = 0; i < PT64_ENT_PER_PAGE; ++i) {
 		u64 ent = sp->spt[i];
 		if (is_shadow_present_pte(ent)) {
 			if (!is_last_spte(ent, sp->role.level)) {
 				struct kvm_mmu_page *child;
 				child = page_header(ent & PT64_BASE_ADDR_MASK);
 				__mmu_spte_walk(kvm, child, fn);
 			} else
 				fn(kvm, &sp->spt[i]);
 		}
 	}
 }
 static void mmu_spte_walk(struct kvm_vcpu *vcpu, inspect_spte_fn fn)
 {
 	int i;
 	struct kvm_mmu_page *sp;
 	if (!VALID_PAGE(vcpu->arch.mmu.root_hpa))
 		return;
 	if (vcpu->arch.mmu.shadow_root_level == PT64_ROOT_LEVEL) {
 		hpa_t root = vcpu->arch.mmu.root_hpa;
 		sp = page_header(root);
 		__mmu_spte_walk(vcpu->kvm, sp, fn);
 		return;
 	}
 	for (i = 0; i < 4; ++i) {
 		hpa_t root = vcpu->arch.mmu.pae_root[i];
 		if (root && VALID_PAGE(root)) {
 			root &= PT64_BASE_ADDR_MASK;
 			sp = page_header(root);
 			__mmu_spte_walk(vcpu->kvm, sp, fn);
 		}
 	}
 	return;
 }
 static void audit_mappings_page(struct kvm_vcpu *vcpu, u64 page_pte,
 				gva_t va, int level)
 {
 	u64 *pt = __va(page_pte & PT64_BASE_ADDR_MASK);
 	int i;
 	gva_t va_delta = 1ul << (PAGE_SHIFT + 9 * (level - 1));
 	for (i = 0; i < PT64_ENT_PER_PAGE; ++i, va += va_delta) {
 		u64 ent = pt[i];
 		if (ent == shadow_trap_nonpresent_pte)
 			continue;
 		va = canonicalize(va);
 		if (is_shadow_present_pte(ent) && !is_last_spte(ent, level))
 			audit_mappings_page(vcpu, ent, va, level - 1);
 		else {
 			gpa_t gpa = kvm_mmu_gva_to_gpa_read(vcpu, va, NULL);
 			gfn_t gfn = gpa >> PAGE_SHIFT;
 			pfn_t pfn = gfn_to_pfn(vcpu->kvm, gfn);
 			hpa_t hpa = (hpa_t)pfn << PAGE_SHIFT;
 			if (is_error_pfn(pfn)) {
 				kvm_release_pfn_clean(pfn);
 				continue;
 			}
 			if (is_shadow_present_pte(ent)
 			    && (ent & PT64_BASE_ADDR_MASK) != hpa)
 				printk(KERN_ERR "xx audit error: (%s) levels %d"
 				       " gva %lx gpa %llx hpa %llx ent %llx %d\n",
 				       audit_msg, vcpu->arch.mmu.root_level,
 				       va, gpa, hpa, ent,
 				       is_shadow_present_pte(ent));
 			else if (ent == shadow_notrap_nonpresent_pte
 				 && !is_error_hpa(hpa))
 				printk(KERN_ERR "audit: (%s) notrap shadow,"
 				       " valid guest gva %lx\n", audit_msg, va);
 			kvm_release_pfn_clean(pfn);
 		}
 	}
 }
 static void audit_mappings(struct kvm_vcpu *vcpu)
 {
 	unsigned i;
 	if (vcpu->arch.mmu.root_level == 4)
 		audit_mappings_page(vcpu, vcpu->arch.mmu.root_hpa, 0, 4);
 	else
 		for (i = 0; i < 4; ++i)
 			if (vcpu->arch.mmu.pae_root[i] & PT_PRESENT_MASK)
 				audit_mappings_page(vcpu,
 						    vcpu->arch.mmu.pae_root[i],
 						    i << 30,
 						    2);
 }
 static int count_rmaps(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
 	struct kvm_memslots *slots;
 	int nmaps = 0;
 	int i, j, k, idx;
 	idx = srcu_read_lock(&kvm->srcu);
 	slots = kvm_memslots(kvm);
 	for (i = 0; i < KVM_MEMORY_SLOTS; ++i) {
 		struct kvm_memory_slot *m = &slots->memslots[i];
 		struct kvm_rmap_desc *d;
 		for (j = 0; j < m->npages; ++j) {
 			unsigned long *rmapp = &m->rmap[j];
 			if (!*rmapp)
 				continue;
 			if (!(*rmapp & 1)) {
 				++nmaps;
 				continue;
 			}
 			d = (struct kvm_rmap_desc *)(*rmapp & ~1ul);
 			while (d) {
 				for (k = 0; k < RMAP_EXT; ++k)
 					if (d->sptes[k])
 						++nmaps;
 					else
 						break;
 				d = d->more;
 			}
 		}
 	}
 	srcu_read_unlock(&kvm->srcu, idx);
 	return nmaps;
 }
 void inspect_spte_has_rmap(struct kvm *kvm, u64 *sptep)
 {
 	unsigned long *rmapp;
 	struct kvm_mmu_page *rev_sp;
 	gfn_t gfn;
 	if (is_writable_pte(*sptep)) {
 		rev_sp = page_header(__pa(sptep));
 		gfn = kvm_mmu_page_get_gfn(rev_sp, sptep - rev_sp->spt);
 		if (!gfn_to_memslot(kvm, gfn)) {
 			if (!printk_ratelimit())
 				return;
 			printk(KERN_ERR "%s: no memslot for gfn %ld\n",
 					 audit_msg, gfn);
 			printk(KERN_ERR "%s: index %ld of sp (gfn=%lx)\n",
 			       audit_msg, (long int)(sptep - rev_sp->spt),
 					rev_sp->gfn);
 			dump_stack();
 			return;
 		}
 		rmapp = gfn_to_rmap(kvm, gfn, rev_sp->role.level);
 		if (!*rmapp) {
 			if (!printk_ratelimit())
 				return;
 			printk(KERN_ERR "%s: no rmap for writable spte %llx\n",
 					 audit_msg, *sptep);
 			dump_stack();
 		}
 	}
 }
 void audit_writable_sptes_have_rmaps(struct kvm_vcpu *vcpu)
 {
 	mmu_spte_walk(vcpu, inspect_spte_has_rmap);
 }
 static void check_writable_mappings_rmap(struct kvm_vcpu *vcpu)
 {
 	struct kvm_mmu_page *sp;
 	int i;
 	list_for_each_entry(sp, &vcpu->kvm->arch.active_mmu_pages, link) {
 		u64 *pt = sp->spt;
 		if (sp->role.level != PT_PAGE_TABLE_LEVEL)
 			continue;
 		for (i = 0; i < PT64_ENT_PER_PAGE; ++i) {
 			u64 ent = pt[i];
 			if (!(ent & PT_PRESENT_MASK))
 				continue;
 			if (!is_writable_pte(ent))
 				continue;
 			inspect_spte_has_rmap(vcpu->kvm, &pt[i]);
 		}
 	}
 	return;
 }
 static void audit_rmap(struct kvm_vcpu *vcpu)
 {
 	check_writable_mappings_rmap(vcpu);
 	count_rmaps(vcpu);
 }
 static void audit_write_protection(struct kvm_vcpu *vcpu)
 {
 	struct kvm_mmu_page *sp;
 	struct kvm_memory_slot *slot;
 	unsigned long *rmapp;
 	u64 *spte;
 	gfn_t gfn;
 	list_for_each_entry(sp, &vcpu->kvm->arch.active_mmu_pages, link) {
 		if (sp->role.direct)
 			continue;
 		if (sp->unsync)
 			continue;
-		gfn = unalias_gfn(vcpu->kvm, sp->gfn);
+		slot = gfn_to_memslot(vcpu->kvm, sp->gfn);
-		slot = gfn_to_memslot_unaliased(vcpu->kvm, sp->gfn);
 		rmapp = &slot->rmap[gfn - slot->base_gfn];
 		spte = rmap_next(vcpu->kvm, rmapp, NULL);
 		while (spte) {
 			if (is_writable_pte(*spte))
 				printk(KERN_ERR "%s: (%s) shadow page has "
 				"writable mappings: gfn %lx role %x\n",
 			       __func__, audit_msg, sp->gfn,
 			       sp->role.word);
 			spte = rmap_next(vcpu->kvm, rmapp, spte);
 		}
 	}
 }
 static void kvm_mmu_audit(struct kvm_vcpu *vcpu, const char *msg)
 {
 	int olddbg = dbg;
 	dbg = 0;
 	audit_msg = msg;
 	audit_rmap(vcpu);
 	audit_write_protection(vcpu);
 	if (strcmp("pre pte write", audit_msg) != 0)
 		audit_mappings(vcpu);
 	audit_writable_sptes_have_rmaps(vcpu);
 	dbg = olddbg;
 }
 #endif

arch/x86/kvm/paging_tmpl.h

Diff comments View file @ a1f4d39

 /*
  * Kernel-based Virtual Machine driver for Linux
  *
  * This module enables machines with Intel VT-x extensions to run virtual
  * machines without emulation or binary translation.
  *
  * MMU support
  *
  * Copyright (C) 2006 Qumranet, Inc.
  * Copyright 2010 Red Hat, Inc. and/or its affilates.
  *
  * Authors:
  *   Yaniv Kamay  <yaniv@qumranet.com>
  *   Avi Kivity   <avi@qumranet.com>
  *
  * This work is licensed under the terms of the GNU GPL, version 2.  See
  * the COPYING file in the top-level directory.
  *
  */
 /*
  * We need the mmu code to access both 32-bit and 64-bit guest ptes,
  * so the code in this file is compiled twice, once per pte size.
  */
 #if PTTYPE == 64
 	#define pt_element_t u64
 	#define guest_walker guest_walker64
 	#define FNAME(name) paging##64_##name
 	#define PT_BASE_ADDR_MASK PT64_BASE_ADDR_MASK
 	#define PT_LVL_ADDR_MASK(lvl) PT64_LVL_ADDR_MASK(lvl)
 	#define PT_LVL_OFFSET_MASK(lvl) PT64_LVL_OFFSET_MASK(lvl)
 	#define PT_INDEX(addr, level) PT64_INDEX(addr, level)
 	#define PT_LEVEL_MASK(level) PT64_LEVEL_MASK(level)
 	#define PT_LEVEL_BITS PT64_LEVEL_BITS
 	#ifdef CONFIG_X86_64
 	#define PT_MAX_FULL_LEVELS 4
 	#define CMPXCHG cmpxchg
 	#else
 	#define CMPXCHG cmpxchg64
 	#define PT_MAX_FULL_LEVELS 2
 	#endif
 #elif PTTYPE == 32
 	#define pt_element_t u32
 	#define guest_walker guest_walker32
 	#define FNAME(name) paging##32_##name
 	#define PT_BASE_ADDR_MASK PT32_BASE_ADDR_MASK
 	#define PT_LVL_ADDR_MASK(lvl) PT32_LVL_ADDR_MASK(lvl)
 	#define PT_LVL_OFFSET_MASK(lvl) PT32_LVL_OFFSET_MASK(lvl)
 	#define PT_INDEX(addr, level) PT32_INDEX(addr, level)
 	#define PT_LEVEL_MASK(level) PT32_LEVEL_MASK(level)
 	#define PT_LEVEL_BITS PT32_LEVEL_BITS
 	#define PT_MAX_FULL_LEVELS 2
 	#define CMPXCHG cmpxchg
 #else
 	#error Invalid PTTYPE value
 #endif
 #define gpte_to_gfn_lvl FNAME(gpte_to_gfn_lvl)
 #define gpte_to_gfn(pte) gpte_to_gfn_lvl((pte), PT_PAGE_TABLE_LEVEL)
 /*
  * The guest_walker structure emulates the behavior of the hardware page
  * table walker.
  */
 struct guest_walker {
 	int level;
 	gfn_t table_gfn[PT_MAX_FULL_LEVELS];
 	pt_element_t ptes[PT_MAX_FULL_LEVELS];
 	gpa_t pte_gpa[PT_MAX_FULL_LEVELS];
 	unsigned pt_access;
 	unsigned pte_access;
 	gfn_t gfn;
 	u32 error_code;
 };
 static gfn_t gpte_to_gfn_lvl(pt_element_t gpte, int lvl)
 {
 	return (gpte & PT_LVL_ADDR_MASK(lvl)) >> PAGE_SHIFT;
 }
 static bool FNAME(cmpxchg_gpte)(struct kvm *kvm,
 			 gfn_t table_gfn, unsigned index,
 			 pt_element_t orig_pte, pt_element_t new_pte)
 {
 	pt_element_t ret;
 	pt_element_t *table;
 	struct page *page;
 	page = gfn_to_page(kvm, table_gfn);
 	table = kmap_atomic(page, KM_USER0);
 	ret = CMPXCHG(&table[index], orig_pte, new_pte);
 	kunmap_atomic(table, KM_USER0);
 	kvm_release_page_dirty(page);
 	return (ret != orig_pte);
 }
 static unsigned FNAME(gpte_access)(struct kvm_vcpu *vcpu, pt_element_t gpte)
 {
 	unsigned access;
 	access = (gpte & (PT_WRITABLE_MASK | PT_USER_MASK)) | ACC_EXEC_MASK;
 #if PTTYPE == 64
 	if (is_nx(vcpu))
 		access &= ~(gpte >> PT64_NX_SHIFT);
 #endif
 	return access;
 }
 /*
  * Fetch a guest pte for a guest virtual address
  */
 static int FNAME(walk_addr)(struct guest_walker *walker,
 			    struct kvm_vcpu *vcpu, gva_t addr,
 			    int write_fault, int user_fault, int fetch_fault)
 {
 	pt_element_t pte;
 	gfn_t table_gfn;
 	unsigned index, pt_access, pte_access;
 	gpa_t pte_gpa;
 	int rsvd_fault = 0;
 	trace_kvm_mmu_pagetable_walk(addr, write_fault, user_fault,
 				     fetch_fault);
 walk:
 	walker->level = vcpu->arch.mmu.root_level;
 	pte = vcpu->arch.cr3;
 #if PTTYPE == 64
 	if (!is_long_mode(vcpu)) {
 		pte = kvm_pdptr_read(vcpu, (addr >> 30) & 3);
 		trace_kvm_mmu_paging_element(pte, walker->level);
 		if (!is_present_gpte(pte))
 			goto not_present;
 		--walker->level;
 	}
 #endif
 	ASSERT((!is_long_mode(vcpu) && is_pae(vcpu)) ||
 	       (vcpu->arch.cr3 & CR3_NONPAE_RESERVED_BITS) == 0);
 	pt_access = ACC_ALL;
 	for (;;) {
 		index = PT_INDEX(addr, walker->level);
 		table_gfn = gpte_to_gfn(pte);
 		pte_gpa = gfn_to_gpa(table_gfn);
 		pte_gpa += index * sizeof(pt_element_t);
 		walker->table_gfn[walker->level - 1] = table_gfn;
 		walker->pte_gpa[walker->level - 1] = pte_gpa;
 		if (kvm_read_guest(vcpu->kvm, pte_gpa, &pte, sizeof(pte)))
 			goto not_present;
 		trace_kvm_mmu_paging_element(pte, walker->level);
 		if (!is_present_gpte(pte))
 			goto not_present;
 		rsvd_fault = is_rsvd_bits_set(vcpu, pte, walker->level);
 		if (rsvd_fault)
 			goto access_error;
 		if (write_fault && !is_writable_pte(pte))
 			if (user_fault || is_write_protection(vcpu))
 				goto access_error;
 		if (user_fault && !(pte & PT_USER_MASK))
 			goto access_error;
 #if PTTYPE == 64
 		if (fetch_fault && (pte & PT64_NX_MASK))
 			goto access_error;
 #endif
 		if (!(pte & PT_ACCESSED_MASK)) {
 			trace_kvm_mmu_set_accessed_bit(table_gfn, index,
 						       sizeof(pte));
 			if (FNAME(cmpxchg_gpte)(vcpu->kvm, table_gfn,
 			    index, pte, pte|PT_ACCESSED_MASK))
 				goto walk;
 			mark_page_dirty(vcpu->kvm, table_gfn);
 			pte |= PT_ACCESSED_MASK;
 		}
 		pte_access = pt_access & FNAME(gpte_access)(vcpu, pte);
 		walker->ptes[walker->level - 1] = pte;
 		if ((walker->level == PT_PAGE_TABLE_LEVEL) ||
 		    ((walker->level == PT_DIRECTORY_LEVEL) &&
 				is_large_pte(pte) &&
 				(PTTYPE == 64 || is_pse(vcpu))) ||
 		    ((walker->level == PT_PDPE_LEVEL) &&
 				is_large_pte(pte) &&
 				is_long_mode(vcpu))) {
 			int lvl = walker->level;
 			walker->gfn = gpte_to_gfn_lvl(pte, lvl);
 			walker->gfn += (addr & PT_LVL_OFFSET_MASK(lvl))
 					>> PAGE_SHIFT;
 			if (PTTYPE == 32 &&
 			    walker->level == PT_DIRECTORY_LEVEL &&
 			    is_cpuid_PSE36())
 				walker->gfn += pse36_gfn_delta(pte);
 			break;
 		}
 		pt_access = pte_access;
 		--walker->level;
 	}
 	if (write_fault && !is_dirty_gpte(pte)) {
 		bool ret;
 		trace_kvm_mmu_set_dirty_bit(table_gfn, index, sizeof(pte));
 		ret = FNAME(cmpxchg_gpte)(vcpu->kvm, table_gfn, index, pte,
 			    pte|PT_DIRTY_MASK);
 		if (ret)
 			goto walk;
 		mark_page_dirty(vcpu->kvm, table_gfn);
 		pte |= PT_DIRTY_MASK;
 		walker->ptes[walker->level - 1] = pte;
 	}
 	walker->pt_access = pt_access;
 	walker->pte_access = pte_access;
 	pgprintk("%s: pte %llx pte_access %x pt_access %x\n",
 		 __func__, (u64)pte, pte_access, pt_access);
 	return 1;
 not_present:
 	walker->error_code = 0;
 	goto err;
 access_error:
 	walker->error_code = PFERR_PRESENT_MASK;
 err:
 	if (write_fault)
 		walker->error_code |= PFERR_WRITE_MASK;
 	if (user_fault)
 		walker->error_code |= PFERR_USER_MASK;
 	if (fetch_fault)
 		walker->error_code |= PFERR_FETCH_MASK;
 	if (rsvd_fault)
 		walker->error_code |= PFERR_RSVD_MASK;
 	trace_kvm_mmu_walker_error(walker->error_code);
 	return 0;
 }
 static void FNAME(update_pte)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,
 			      u64 *spte, const void *pte)
 {
 	pt_element_t gpte;
 	unsigned pte_access;
 	pfn_t pfn;
 	u64 new_spte;
 	gpte = *(const pt_element_t *)pte;
 	if (~gpte & (PT_PRESENT_MASK | PT_ACCESSED_MASK)) {
 		if (!is_present_gpte(gpte)) {
 			if (sp->unsync)
 				new_spte = shadow_trap_nonpresent_pte;
 			else
 				new_spte = shadow_notrap_nonpresent_pte;
 			__set_spte(spte, new_spte);
 		}
 		return;
 	}
 	pgprintk("%s: gpte %llx spte %p\n", __func__, (u64)gpte, spte);
 	pte_access = sp->role.access & FNAME(gpte_access)(vcpu, gpte);
 	if (gpte_to_gfn(gpte) != vcpu->arch.update_pte.gfn)
 		return;
 	pfn = vcpu->arch.update_pte.pfn;
 	if (is_error_pfn(pfn))
 		return;
 	if (mmu_notifier_retry(vcpu, vcpu->arch.update_pte.mmu_seq))
 		return;
 	kvm_get_pfn(pfn);
 	/*
 	 * we call mmu_set_spte() with reset_host_protection = true beacuse that
 	 * vcpu->arch.update_pte.pfn was fetched from get_user_pages(write = 1).
 	 */
 	mmu_set_spte(vcpu, spte, sp->role.access, pte_access, 0, 0,
 		     is_dirty_gpte(gpte), NULL, PT_PAGE_TABLE_LEVEL,
 		     gpte_to_gfn(gpte), pfn, true, true);
 }
 /*
  * Fetch a shadow pte for a specific level in the paging hierarchy.
  */
 static u64 *FNAME(fetch)(struct kvm_vcpu *vcpu, gva_t addr,
 			 struct guest_walker *gw,
 			 int user_fault, int write_fault, int hlevel,
 			 int *ptwrite, pfn_t pfn)
 {
 	unsigned access = gw->pt_access;
 	struct kvm_mmu_page *sp;
 	u64 spte, *sptep = NULL;
 	int direct;
 	gfn_t table_gfn;
 	int r;
 	int level;
 	pt_element_t curr_pte;
 	struct kvm_shadow_walk_iterator iterator;
 	if (!is_present_gpte(gw->ptes[gw->level - 1]))
 		return NULL;
 	for_each_shadow_entry(vcpu, addr, iterator) {
 		level = iterator.level;
 		sptep = iterator.sptep;
 		if (iterator.level == hlevel) {
 			mmu_set_spte(vcpu, sptep, access,
 				     gw->pte_access & access,
 				     user_fault, write_fault,
 				     is_dirty_gpte(gw->ptes[gw->level-1]),
 				     ptwrite, level,
 				     gw->gfn, pfn, false, true);
 			break;
 		}
 		if (is_shadow_present_pte(*sptep) && !is_large_pte(*sptep))
 			continue;
 		if (is_large_pte(*sptep)) {
 			rmap_remove(vcpu->kvm, sptep);
 			__set_spte(sptep, shadow_trap_nonpresent_pte);
 			kvm_flush_remote_tlbs(vcpu->kvm);
 		}
 		if (level <= gw->level) {
 			int delta = level - gw->level + 1;
 			direct = 1;
 			if (!is_dirty_gpte(gw->ptes[level - delta]))
 				access &= ~ACC_WRITE_MASK;
 			/*
 			 * It is a large guest pages backed by small host pages,
 			 * So we set @direct(@sp->role.direct)=1, and set
 			 * @table_gfn(@sp->gfn)=the base page frame for linear
 			 * translations.
 			 */
 			table_gfn = gw->gfn & ~(KVM_PAGES_PER_HPAGE(level) - 1);
 			access &= gw->pte_access;
 		} else {
 			direct = 0;
 			table_gfn = gw->table_gfn[level - 2];
 		}
 		sp = kvm_mmu_get_page(vcpu, table_gfn, addr, level-1,
 					       direct, access, sptep);
 		if (!direct) {
 			r = kvm_read_guest_atomic(vcpu->kvm,
 						  gw->pte_gpa[level - 2],
 						  &curr_pte, sizeof(curr_pte));
 			if (r || curr_pte != gw->ptes[level - 2]) {
 				kvm_mmu_put_page(sp, sptep);
 				kvm_release_pfn_clean(pfn);
 				sptep = NULL;
 				break;
 			}
 		}
 		spte = __pa(sp->spt)
 			| PT_PRESENT_MASK | PT_ACCESSED_MASK
 			| PT_WRITABLE_MASK | PT_USER_MASK;
 		*sptep = spte;
 	}
 	return sptep;
 }
 /*
  * Page fault handler.  There are several causes for a page fault:
  *   - there is no shadow pte for the guest pte
  *   - write access through a shadow pte marked read only so that we can set
  *     the dirty bit
  *   - write access to a shadow pte marked read only so we can update the page
  *     dirty bitmap, when userspace requests it
  *   - mmio access; in this case we will never install a present shadow pte
  *   - normal guest page fault due to the guest pte marked not present, not
  *     writable, or not executable
  *
  *  Returns: 1 if we need to emulate the instruction, 0 otherwise, or
  *           a negative value on error.
  */
 static int FNAME(page_fault)(struct kvm_vcpu *vcpu, gva_t addr,
 			       u32 error_code)
 {
 	int write_fault = error_code & PFERR_WRITE_MASK;
 	int user_fault = error_code & PFERR_USER_MASK;
 	int fetch_fault = error_code & PFERR_FETCH_MASK;
 	struct guest_walker walker;
 	u64 *sptep;
 	int write_pt = 0;
 	int r;
 	pfn_t pfn;
 	int level = PT_PAGE_TABLE_LEVEL;
 	unsigned long mmu_seq;
 	pgprintk("%s: addr %lx err %x\n", __func__, addr, error_code);
 	kvm_mmu_audit(vcpu, "pre page fault");
 	r = mmu_topup_memory_caches(vcpu);
 	if (r)
 		return r;
 	/*
 	 * Look up the guest pte for the faulting address.
 	 */
 	r = FNAME(walk_addr)(&walker, vcpu, addr, write_fault, user_fault,
 			     fetch_fault);
 	/*
 	 * The page is not mapped by the guest.  Let the guest handle it.
 	 */
 	if (!r) {
 		pgprintk("%s: guest page fault\n", __func__);
 		inject_page_fault(vcpu, addr, walker.error_code);
 		vcpu->arch.last_pt_write_count = 0; /* reset fork detector */
 		return 0;
 	}
 	if (walker.level >= PT_DIRECTORY_LEVEL) {
 		level = min(walker.level, mapping_level(vcpu, walker.gfn));
 		walker.gfn = walker.gfn & ~(KVM_PAGES_PER_HPAGE(level) - 1);
 	}
 	mmu_seq = vcpu->kvm->mmu_notifier_seq;
 	smp_rmb();
 	pfn = gfn_to_pfn(vcpu->kvm, walker.gfn);
 	/* mmio */
 	if (is_error_pfn(pfn))
 		return kvm_handle_bad_page(vcpu->kvm, walker.gfn, pfn);
 	spin_lock(&vcpu->kvm->mmu_lock);
 	if (mmu_notifier_retry(vcpu, mmu_seq))
 		goto out_unlock;
 	kvm_mmu_free_some_pages(vcpu);
 	sptep = FNAME(fetch)(vcpu, addr, &walker, user_fault, write_fault,
 			     level, &write_pt, pfn);
 	(void)sptep;
 	pgprintk("%s: shadow pte %p %llx ptwrite %d\n", __func__,
 		 sptep, *sptep, write_pt);
 	if (!write_pt)
 		vcpu->arch.last_pt_write_count = 0; /* reset fork detector */
 	++vcpu->stat.pf_fixed;
 	kvm_mmu_audit(vcpu, "post page fault (fixed)");
 	spin_unlock(&vcpu->kvm->mmu_lock);
 	return write_pt;
 out_unlock:
 	spin_unlock(&vcpu->kvm->mmu_lock);
 	kvm_release_pfn_clean(pfn);
 	return 0;
 }
 static void FNAME(invlpg)(struct kvm_vcpu *vcpu, gva_t gva)
 {
 	struct kvm_shadow_walk_iterator iterator;
 	struct kvm_mmu_page *sp;
 	gpa_t pte_gpa = -1;
 	int level;
 	u64 *sptep;
 	int need_flush = 0;
 	spin_lock(&vcpu->kvm->mmu_lock);
 	for_each_shadow_entry(vcpu, gva, iterator) {
 		level = iterator.level;
 		sptep = iterator.sptep;
 		sp = page_header(__pa(sptep));
 		if (is_last_spte(*sptep, level)) {
 			int offset, shift;
 			if (!sp->unsync)
 				break;
 			shift = PAGE_SHIFT -
 				  (PT_LEVEL_BITS - PT64_LEVEL_BITS) * level;
 			offset = sp->role.quadrant << shift;
 			pte_gpa = (sp->gfn << PAGE_SHIFT) + offset;
 			pte_gpa += (sptep - sp->spt) * sizeof(pt_element_t);
 			if (is_shadow_present_pte(*sptep)) {
 				rmap_remove(vcpu->kvm, sptep);
 				if (is_large_pte(*sptep))
 					--vcpu->kvm->stat.lpages;
 				need_flush = 1;
 			}
 			__set_spte(sptep, shadow_trap_nonpresent_pte);
 			break;
 		}
 		if (!is_shadow_present_pte(*sptep) || !sp->unsync_children)
 			break;
 	}
 	if (need_flush)
 		kvm_flush_remote_tlbs(vcpu->kvm);
 	atomic_inc(&vcpu->kvm->arch.invlpg_counter);
 	spin_unlock(&vcpu->kvm->mmu_lock);
 	if (pte_gpa == -1)
 		return;
 	if (mmu_topup_memory_caches(vcpu))
 		return;
 	kvm_mmu_pte_write(vcpu, pte_gpa, NULL, sizeof(pt_element_t), 0);
 }
 static gpa_t FNAME(gva_to_gpa)(struct kvm_vcpu *vcpu, gva_t vaddr, u32 access,
 			       u32 *error)
 {
 	struct guest_walker walker;
 	gpa_t gpa = UNMAPPED_GVA;
 	int r;
 	r = FNAME(walk_addr)(&walker, vcpu, vaddr,
 			     !!(access & PFERR_WRITE_MASK),
 			     !!(access & PFERR_USER_MASK),
 			     !!(access & PFERR_FETCH_MASK));
 	if (r) {
 		gpa = gfn_to_gpa(walker.gfn);
 		gpa |= vaddr & ~PAGE_MASK;
 	} else if (error)
 		*error = walker.error_code;
 	return gpa;
 }
 static void FNAME(prefetch_page)(struct kvm_vcpu *vcpu,
 				 struct kvm_mmu_page *sp)
 {
 	int i, j, offset, r;
 	pt_element_t pt[256 / sizeof(pt_element_t)];
 	gpa_t pte_gpa;
 	if (sp->role.direct
 	    || (PTTYPE == 32 && sp->role.level > PT_PAGE_TABLE_LEVEL)) {
 		nonpaging_prefetch_page(vcpu, sp);
 		return;
 	}
 	pte_gpa = gfn_to_gpa(sp->gfn);
 	if (PTTYPE == 32) {
 		offset = sp->role.quadrant << PT64_LEVEL_BITS;
 		pte_gpa += offset * sizeof(pt_element_t);
 	}
 	for (i = 0; i < PT64_ENT_PER_PAGE; i += ARRAY_SIZE(pt)) {
 		r = kvm_read_guest_atomic(vcpu->kvm, pte_gpa, pt, sizeof pt);
 		pte_gpa += ARRAY_SIZE(pt) * sizeof(pt_element_t);
 		for (j = 0; j < ARRAY_SIZE(pt); ++j)
 			if (r || is_present_gpte(pt[j]))
 				sp->spt[i+j] = shadow_trap_nonpresent_pte;
 			else
 				sp->spt[i+j] = shadow_notrap_nonpresent_pte;
 	}
 }
 /*
  * Using the cached information from sp->gfns is safe because:
  * - The spte has a reference to the struct page, so the pfn for a given gfn
  *   can't change unless all sptes pointing to it are nuked first.
- * - Alias changes zap the entire shadow cache.
  */
 static int FNAME(sync_page)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,
 			    bool clear_unsync)
 {
 	int i, offset, nr_present;
 	bool reset_host_protection;
 	gpa_t first_pte_gpa;
 	offset = nr_present = 0;
 	/* direct kvm_mmu_page can not be unsync. */
 	BUG_ON(sp->role.direct);
 	if (PTTYPE == 32)
 		offset = sp->role.quadrant << PT64_LEVEL_BITS;
 	first_pte_gpa = gfn_to_gpa(sp->gfn) + offset * sizeof(pt_element_t);
 	for (i = 0; i < PT64_ENT_PER_PAGE; i++) {
 		unsigned pte_access;
 		pt_element_t gpte;
 		gpa_t pte_gpa;
 		gfn_t gfn;
 		if (!is_shadow_present_pte(sp->spt[i]))
 			continue;
 		pte_gpa = first_pte_gpa + i * sizeof(pt_element_t);
 		if (kvm_read_guest_atomic(vcpu->kvm, pte_gpa, &gpte,
 					  sizeof(pt_element_t)))
 			return -EINVAL;
 		gfn = gpte_to_gfn(gpte);
-		if (unalias_gfn(vcpu->kvm, gfn) != sp->gfns[i] ||
+		if (gfn != sp->gfns[i] ||
 		      !is_present_gpte(gpte) || !(gpte & PT_ACCESSED_MASK)) {
 			u64 nonpresent;
 			rmap_remove(vcpu->kvm, &sp->spt[i]);
 			if (is_present_gpte(gpte) || !clear_unsync)
 				nonpresent = shadow_trap_nonpresent_pte;
 			else
 				nonpresent = shadow_notrap_nonpresent_pte;
 			__set_spte(&sp->spt[i], nonpresent);
 			continue;
 		}
 		nr_present++;
 		pte_access = sp->role.access & FNAME(gpte_access)(vcpu, gpte);
 		if (!(sp->spt[i] & SPTE_HOST_WRITEABLE)) {
 			pte_access &= ~ACC_WRITE_MASK;
 			reset_host_protection = 0;
 		} else {
 			reset_host_protection = 1;
 		}
 		set_spte(vcpu, &sp->spt[i], pte_access, 0, 0,
 			 is_dirty_gpte(gpte), PT_PAGE_TABLE_LEVEL, gfn,
 			 spte_to_pfn(sp->spt[i]), true, false,
 			 reset_host_protection);
 	}
 	return !nr_present;
 }
 #undef pt_element_t
 #undef guest_walker
 #undef FNAME
 #undef PT_BASE_ADDR_MASK
 #undef PT_INDEX
 #undef PT_LEVEL_MASK
 #undef PT_LVL_ADDR_MASK
 #undef PT_LVL_OFFSET_MASK
 #undef PT_LEVEL_BITS
 #undef PT_MAX_FULL_LEVELS
 #undef gpte_to_gfn
 #undef gpte_to_gfn_lvl
 #undef CMPXCHG

arch/x86/kvm/x86.c

Diff comments View file @ a1f4d39

 /*
  * Kernel-based Virtual Machine driver for Linux
  *
  * derived from drivers/kvm/kvm_main.c
  *
  * Copyright (C) 2006 Qumranet, Inc.
  * Copyright (C) 2008 Qumranet, Inc.
  * Copyright IBM Corporation, 2008
  * Copyright 2010 Red Hat, Inc. and/or its affilates.
  *
  * Authors:
  *   Avi Kivity   <avi@qumranet.com>
  *   Yaniv Kamay  <yaniv@qumranet.com>
  *   Amit Shah    <amit.shah@qumranet.com>
  *   Ben-Ami Yassour <benami@il.ibm.com>
  *
  * This work is licensed under the terms of the GNU GPL, version 2.  See
  * the COPYING file in the top-level directory.
  *
  */
 #include <linux/kvm_host.h>
 #include "irq.h"
 #include "mmu.h"
 #include "i8254.h"
 #include "tss.h"
 #include "kvm_cache_regs.h"
 #include "x86.h"
 #include <linux/clocksource.h>
 #include <linux/interrupt.h>
 #include <linux/kvm.h>
 #include <linux/fs.h>
 #include <linux/vmalloc.h>
 #include <linux/module.h>
 #include <linux/mman.h>
 #include <linux/highmem.h>
 #include <linux/iommu.h>
 #include <linux/intel-iommu.h>
 #include <linux/cpufreq.h>
 #include <linux/user-return-notifier.h>
 #include <linux/srcu.h>
 #include <linux/slab.h>
 #include <linux/perf_event.h>
 #include <linux/uaccess.h>
 #include <trace/events/kvm.h>
 #define CREATE_TRACE_POINTS
 #include "trace.h"
 #include <asm/debugreg.h>
 #include <asm/msr.h>
 #include <asm/desc.h>
 #include <asm/mtrr.h>
 #include <asm/mce.h>
 #include <asm/i387.h>
 #include <asm/xcr.h>
 #define MAX_IO_MSRS 256
 #define CR0_RESERVED_BITS						\
 	(~(unsigned long)(X86_CR0_PE | X86_CR0_MP | X86_CR0_EM | X86_CR0_TS \
 			  | X86_CR0_ET | X86_CR0_NE | X86_CR0_WP | X86_CR0_AM \
 			  | X86_CR0_NW | X86_CR0_CD | X86_CR0_PG))
 #define CR4_RESERVED_BITS						\
 	(~(unsigned long)(X86_CR4_VME | X86_CR4_PVI | X86_CR4_TSD | X86_CR4_DE\
 			  | X86_CR4_PSE | X86_CR4_PAE | X86_CR4_MCE	\
 			  | X86_CR4_PGE | X86_CR4_PCE | X86_CR4_OSFXSR	\
 			  | X86_CR4_OSXSAVE \
 			  | X86_CR4_OSXMMEXCPT | X86_CR4_VMXE))
 #define CR8_RESERVED_BITS (~(unsigned long)X86_CR8_TPR)
 #define KVM_MAX_MCE_BANKS 32
 #define KVM_MCE_CAP_SUPPORTED MCG_CTL_P
 /* EFER defaults:
  * - enable syscall per default because its emulated by KVM
  * - enable LME and LMA per default on 64 bit KVM
  */
 #ifdef CONFIG_X86_64
 static u64 __read_mostly efer_reserved_bits = 0xfffffffffffffafeULL;
 #else
 static u64 __read_mostly efer_reserved_bits = 0xfffffffffffffffeULL;
 #endif
 #define VM_STAT(x) offsetof(struct kvm, stat.x), KVM_STAT_VM
 #define VCPU_STAT(x) offsetof(struct kvm_vcpu, stat.x), KVM_STAT_VCPU
 static void update_cr8_intercept(struct kvm_vcpu *vcpu);
 static int kvm_dev_ioctl_get_supported_cpuid(struct kvm_cpuid2 *cpuid,
 				    struct kvm_cpuid_entry2 __user *entries);
 struct kvm_x86_ops *kvm_x86_ops;
 EXPORT_SYMBOL_GPL(kvm_x86_ops);
 int ignore_msrs = 0;
 module_param_named(ignore_msrs, ignore_msrs, bool, S_IRUGO | S_IWUSR);
 #define KVM_NR_SHARED_MSRS 16
 struct kvm_shared_msrs_global {
 	int nr;
 	u32 msrs[KVM_NR_SHARED_MSRS];
 };
 struct kvm_shared_msrs {
 	struct user_return_notifier urn;
 	bool registered;
 	struct kvm_shared_msr_values {
 		u64 host;
 		u64 curr;
 	} values[KVM_NR_SHARED_MSRS];
 };
 static struct kvm_shared_msrs_global __read_mostly shared_msrs_global;
 static DEFINE_PER_CPU(struct kvm_shared_msrs, shared_msrs);
 struct kvm_stats_debugfs_item debugfs_entries[] = {
 	{ "pf_fixed", VCPU_STAT(pf_fixed) },
 	{ "pf_guest", VCPU_STAT(pf_guest) },
 	{ "tlb_flush", VCPU_STAT(tlb_flush) },
 	{ "invlpg", VCPU_STAT(invlpg) },
 	{ "exits", VCPU_STAT(exits) },
 	{ "io_exits", VCPU_STAT(io_exits) },
 	{ "mmio_exits", VCPU_STAT(mmio_exits) },
 	{ "signal_exits", VCPU_STAT(signal_exits) },
 	{ "irq_window", VCPU_STAT(irq_window_exits) },
 	{ "nmi_window", VCPU_STAT(nmi_window_exits) },
 	{ "halt_exits", VCPU_STAT(halt_exits) },
 	{ "halt_wakeup", VCPU_STAT(halt_wakeup) },
 	{ "hypercalls", VCPU_STAT(hypercalls) },
 	{ "request_irq", VCPU_STAT(request_irq_exits) },
 	{ "irq_exits", VCPU_STAT(irq_exits) },
 	{ "host_state_reload", VCPU_STAT(host_state_reload) },
 	{ "efer_reload", VCPU_STAT(efer_reload) },
 	{ "fpu_reload", VCPU_STAT(fpu_reload) },
 	{ "insn_emulation", VCPU_STAT(insn_emulation) },
 	{ "insn_emulation_fail", VCPU_STAT(insn_emulation_fail) },
 	{ "irq_injections", VCPU_STAT(irq_injections) },
 	{ "nmi_injections", VCPU_STAT(nmi_injections) },
 	{ "mmu_shadow_zapped", VM_STAT(mmu_shadow_zapped) },
 	{ "mmu_pte_write", VM_STAT(mmu_pte_write) },
 	{ "mmu_pte_updated", VM_STAT(mmu_pte_updated) },
 	{ "mmu_pde_zapped", VM_STAT(mmu_pde_zapped) },
 	{ "mmu_flooded", VM_STAT(mmu_flooded) },
 	{ "mmu_recycled", VM_STAT(mmu_recycled) },
 	{ "mmu_cache_miss", VM_STAT(mmu_cache_miss) },
 	{ "mmu_unsync", VM_STAT(mmu_unsync) },
 	{ "remote_tlb_flush", VM_STAT(remote_tlb_flush) },
 	{ "largepages", VM_STAT(lpages) },
 	{ NULL }
 };
 u64 __read_mostly host_xcr0;
 static inline u32 bit(int bitno)
 {
 	return 1 << (bitno & 31);
 }
 static void kvm_on_user_return(struct user_return_notifier *urn)
 {
 	unsigned slot;
 	struct kvm_shared_msrs *locals
 		= container_of(urn, struct kvm_shared_msrs, urn);
 	struct kvm_shared_msr_values *values;
 	for (slot = 0; slot < shared_msrs_global.nr; ++slot) {
 		values = &locals->values[slot];
 		if (values->host != values->curr) {
 			wrmsrl(shared_msrs_global.msrs[slot], values->host);
 			values->curr = values->host;
 		}
 	}
 	locals->registered = false;
 	user_return_notifier_unregister(urn);
 }
 static void shared_msr_update(unsigned slot, u32 msr)
 {
 	struct kvm_shared_msrs *smsr;
 	u64 value;
 	smsr = &__get_cpu_var(shared_msrs);
 	/* only read, and nobody should modify it at this time,
 	 * so don't need lock */
 	if (slot >= shared_msrs_global.nr) {
 		printk(KERN_ERR "kvm: invalid MSR slot!");
 		return;
 	}
 	rdmsrl_safe(msr, &value);
 	smsr->values[slot].host = value;
 	smsr->values[slot].curr = value;
 }
 void kvm_define_shared_msr(unsigned slot, u32 msr)
 {
 	if (slot >= shared_msrs_global.nr)
 		shared_msrs_global.nr = slot + 1;
 	shared_msrs_global.msrs[slot] = msr;
 	/* we need ensured the shared_msr_global have been updated */
 	smp_wmb();
 }
 EXPORT_SYMBOL_GPL(kvm_define_shared_msr);
 static void kvm_shared_msr_cpu_online(void)
 {
 	unsigned i;
 	for (i = 0; i < shared_msrs_global.nr; ++i)
 		shared_msr_update(i, shared_msrs_global.msrs[i]);
 }
 void kvm_set_shared_msr(unsigned slot, u64 value, u64 mask)
 {
 	struct kvm_shared_msrs *smsr = &__get_cpu_var(shared_msrs);
 	if (((value ^ smsr->values[slot].curr) & mask) == 0)
 		return;
 	smsr->values[slot].curr = value;
 	wrmsrl(shared_msrs_global.msrs[slot], value);
 	if (!smsr->registered) {
 		smsr->urn.on_user_return = kvm_on_user_return;
 		user_return_notifier_register(&smsr->urn);
 		smsr->registered = true;
 	}
 }
 EXPORT_SYMBOL_GPL(kvm_set_shared_msr);
 static void drop_user_return_notifiers(void *ignore)
 {
 	struct kvm_shared_msrs *smsr = &__get_cpu_var(shared_msrs);
 	if (smsr->registered)
 		kvm_on_user_return(&smsr->urn);
 }
 u64 kvm_get_apic_base(struct kvm_vcpu *vcpu)
 {
 	if (irqchip_in_kernel(vcpu->kvm))
 		return vcpu->arch.apic_base;
 	else
 		return vcpu->arch.apic_base;
 }
 EXPORT_SYMBOL_GPL(kvm_get_apic_base);
 void kvm_set_apic_base(struct kvm_vcpu *vcpu, u64 data)
 {
 	/* TODO: reserve bits check */
 	if (irqchip_in_kernel(vcpu->kvm))
 		kvm_lapic_set_base(vcpu, data);
 	else
 		vcpu->arch.apic_base = data;
 }
 EXPORT_SYMBOL_GPL(kvm_set_apic_base);
 #define EXCPT_BENIGN		0
 #define EXCPT_CONTRIBUTORY	1
 #define EXCPT_PF		2
 static int exception_class(int vector)
 {
 	switch (vector) {
 	case PF_VECTOR:
 		return EXCPT_PF;
 	case DE_VECTOR:
 	case TS_VECTOR:
 	case NP_VECTOR:
 	case SS_VECTOR:
 	case GP_VECTOR:
 		return EXCPT_CONTRIBUTORY;
 	default:
 		break;
 	}
 	return EXCPT_BENIGN;
 }
 static void kvm_multiple_exception(struct kvm_vcpu *vcpu,
 		unsigned nr, bool has_error, u32 error_code,
 		bool reinject)
 {
 	u32 prev_nr;
 	int class1, class2;
 	if (!vcpu->arch.exception.pending) {
 	queue:
 		vcpu->arch.exception.pending = true;
 		vcpu->arch.exception.has_error_code = has_error;
 		vcpu->arch.exception.nr = nr;
 		vcpu->arch.exception.error_code = error_code;
 		vcpu->arch.exception.reinject = reinject;
 		return;
 	}
 	/* to check exception */
 	prev_nr = vcpu->arch.exception.nr;
 	if (prev_nr == DF_VECTOR) {
 		/* triple fault -> shutdown */
 		set_bit(KVM_REQ_TRIPLE_FAULT, &vcpu->requests);
 		return;
 	}
 	class1 = exception_class(prev_nr);
 	class2 = exception_class(nr);
 	if ((class1 == EXCPT_CONTRIBUTORY && class2 == EXCPT_CONTRIBUTORY)
 		|| (class1 == EXCPT_PF && class2 != EXCPT_BENIGN)) {
 		/* generate double fault per SDM Table 5-5 */
 		vcpu->arch.exception.pending = true;
 		vcpu->arch.exception.has_error_code = true;
 		vcpu->arch.exception.nr = DF_VECTOR;
 		vcpu->arch.exception.error_code = 0;
 	} else
 		/* replace previous exception with a new one in a hope
 		   that instruction re-execution will regenerate lost
 		   exception */
 		goto queue;
 }
 void kvm_queue_exception(struct kvm_vcpu *vcpu, unsigned nr)
 {
 	kvm_multiple_exception(vcpu, nr, false, 0, false);
 }
 EXPORT_SYMBOL_GPL(kvm_queue_exception);
 void kvm_requeue_exception(struct kvm_vcpu *vcpu, unsigned nr)
 {
 	kvm_multiple_exception(vcpu, nr, false, 0, true);
 }
 EXPORT_SYMBOL_GPL(kvm_requeue_exception);
 void kvm_inject_page_fault(struct kvm_vcpu *vcpu, unsigned long addr,
 			   u32 error_code)
 {
 	++vcpu->stat.pf_guest;
 	vcpu->arch.cr2 = addr;
 	kvm_queue_exception_e(vcpu, PF_VECTOR, error_code);
 }
 void kvm_inject_nmi(struct kvm_vcpu *vcpu)
 {
 	vcpu->arch.nmi_pending = 1;
 }
 EXPORT_SYMBOL_GPL(kvm_inject_nmi);
 void kvm_queue_exception_e(struct kvm_vcpu *vcpu, unsigned nr, u32 error_code)
 {
 	kvm_multiple_exception(vcpu, nr, true, error_code, false);
 }
 EXPORT_SYMBOL_GPL(kvm_queue_exception_e);
 void kvm_requeue_exception_e(struct kvm_vcpu *vcpu, unsigned nr, u32 error_code)
 {
 	kvm_multiple_exception(vcpu, nr, true, error_code, true);
 }
 EXPORT_SYMBOL_GPL(kvm_requeue_exception_e);
 /*
  * Checks if cpl <= required_cpl; if true, return true.  Otherwise queue
  * a #GP and return false.
  */
 bool kvm_require_cpl(struct kvm_vcpu *vcpu, int required_cpl)
 {
 	if (kvm_x86_ops->get_cpl(vcpu) <= required_cpl)
 		return true;
 	kvm_queue_exception_e(vcpu, GP_VECTOR, 0);
 	return false;
 }
 EXPORT_SYMBOL_GPL(kvm_require_cpl);
 /*
  * Load the pae pdptrs.  Return true is they are all valid.
  */
 int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3)
 {
 	gfn_t pdpt_gfn = cr3 >> PAGE_SHIFT;
 	unsigned offset = ((cr3 & (PAGE_SIZE-1)) >> 5) << 2;
 	int i;
 	int ret;
 	u64 pdpte[ARRAY_SIZE(vcpu->arch.pdptrs)];
 	ret = kvm_read_guest_page(vcpu->kvm, pdpt_gfn, pdpte,
 				  offset * sizeof(u64), sizeof(pdpte));
 	if (ret < 0) {
 		ret = 0;
 		goto out;
 	}
 	for (i = 0; i < ARRAY_SIZE(pdpte); ++i) {
 		if (is_present_gpte(pdpte[i]) &&
 		    (pdpte[i] & vcpu->arch.mmu.rsvd_bits_mask[0][2])) {
 			ret = 0;
 			goto out;
 		}
 	}
 	ret = 1;
 	memcpy(vcpu->arch.pdptrs, pdpte, sizeof(vcpu->arch.pdptrs));
 	__set_bit(VCPU_EXREG_PDPTR,
 		  (unsigned long *)&vcpu->arch.regs_avail);
 	__set_bit(VCPU_EXREG_PDPTR,
 		  (unsigned long *)&vcpu->arch.regs_dirty);
 out:
 	return ret;
 }
 EXPORT_SYMBOL_GPL(load_pdptrs);
 static bool pdptrs_changed(struct kvm_vcpu *vcpu)
 {
 	u64 pdpte[ARRAY_SIZE(vcpu->arch.pdptrs)];
 	bool changed = true;
 	int r;
 	if (is_long_mode(vcpu) || !is_pae(vcpu))
 		return false;
 	if (!test_bit(VCPU_EXREG_PDPTR,
 		      (unsigned long *)&vcpu->arch.regs_avail))
 		return true;
 	r = kvm_read_guest(vcpu->kvm, vcpu->arch.cr3 & ~31u, pdpte, sizeof(pdpte));
 	if (r < 0)
 		goto out;
 	changed = memcmp(pdpte, vcpu->arch.pdptrs, sizeof(pdpte)) != 0;
 out:
 	return changed;
 }
 int kvm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0)
 {
 	unsigned long old_cr0 = kvm_read_cr0(vcpu);
 	unsigned long update_bits = X86_CR0_PG | X86_CR0_WP |
 				    X86_CR0_CD | X86_CR0_NW;
 	cr0 |= X86_CR0_ET;
 #ifdef CONFIG_X86_64
 	if (cr0 & 0xffffffff00000000UL)
 		return 1;
 #endif
 	cr0 &= ~CR0_RESERVED_BITS;
 	if ((cr0 & X86_CR0_NW) && !(cr0 & X86_CR0_CD))
 		return 1;
 	if ((cr0 & X86_CR0_PG) && !(cr0 & X86_CR0_PE))
 		return 1;
 	if (!is_paging(vcpu) && (cr0 & X86_CR0_PG)) {
 #ifdef CONFIG_X86_64
 		if ((vcpu->arch.efer & EFER_LME)) {
 			int cs_db, cs_l;
 			if (!is_pae(vcpu))
 				return 1;
 			kvm_x86_ops->get_cs_db_l_bits(vcpu, &cs_db, &cs_l);
 			if (cs_l)
 				return 1;
 		} else
 #endif
 		if (is_pae(vcpu) && !load_pdptrs(vcpu, vcpu->arch.cr3))
 			return 1;
 	}
 	kvm_x86_ops->set_cr0(vcpu, cr0);
 	if ((cr0 ^ old_cr0) & update_bits)
 		kvm_mmu_reset_context(vcpu);
 	return 0;
 }
 EXPORT_SYMBOL_GPL(kvm_set_cr0);
 void kvm_lmsw(struct kvm_vcpu *vcpu, unsigned long msw)
 {
 	(void)kvm_set_cr0(vcpu, kvm_read_cr0_bits(vcpu, ~0x0eul) | (msw & 0x0f));
 }
 EXPORT_SYMBOL_GPL(kvm_lmsw);
 int __kvm_set_xcr(struct kvm_vcpu *vcpu, u32 index, u64 xcr)
 {
 	u64 xcr0;
 	/* Only support XCR_XFEATURE_ENABLED_MASK(xcr0) now  */
 	if (index != XCR_XFEATURE_ENABLED_MASK)
 		return 1;
 	xcr0 = xcr;
 	if (kvm_x86_ops->get_cpl(vcpu) != 0)
 		return 1;
 	if (!(xcr0 & XSTATE_FP))
 		return 1;
 	if ((xcr0 & XSTATE_YMM) && !(xcr0 & XSTATE_SSE))
 		return 1;
 	if (xcr0 & ~host_xcr0)
 		return 1;
 	vcpu->arch.xcr0 = xcr0;
 	vcpu->guest_xcr0_loaded = 0;
 	return 0;
 }
 int kvm_set_xcr(struct kvm_vcpu *vcpu, u32 index, u64 xcr)
 {
 	if (__kvm_set_xcr(vcpu, index, xcr)) {
 		kvm_inject_gp(vcpu, 0);
 		return 1;
 	}
 	return 0;
 }
 EXPORT_SYMBOL_GPL(kvm_set_xcr);
 static bool guest_cpuid_has_xsave(struct kvm_vcpu *vcpu)
 {
 	struct kvm_cpuid_entry2 *best;
 	best = kvm_find_cpuid_entry(vcpu, 1, 0);
 	return best && (best->ecx & bit(X86_FEATURE_XSAVE));
 }
 static void update_cpuid(struct kvm_vcpu *vcpu)
 {
 	struct kvm_cpuid_entry2 *best;
 	best = kvm_find_cpuid_entry(vcpu, 1, 0);
 	if (!best)
 		return;
 	/* Update OSXSAVE bit */
 	if (cpu_has_xsave && best->function == 0x1) {
 		best->ecx &= ~(bit(X86_FEATURE_OSXSAVE));
 		if (kvm_read_cr4_bits(vcpu, X86_CR4_OSXSAVE))
 			best->ecx |= bit(X86_FEATURE_OSXSAVE);
 	}
 }
 int kvm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4)
 {
 	unsigned long old_cr4 = kvm_read_cr4(vcpu);
 	unsigned long pdptr_bits = X86_CR4_PGE | X86_CR4_PSE | X86_CR4_PAE;
 	if (cr4 & CR4_RESERVED_BITS)
 		return 1;
 	if (!guest_cpuid_has_xsave(vcpu) && (cr4 & X86_CR4_OSXSAVE))
 		return 1;
 	if (is_long_mode(vcpu)) {
 		if (!(cr4 & X86_CR4_PAE))
 			return 1;
 	} else if (is_paging(vcpu) && (cr4 & X86_CR4_PAE)
 		   && ((cr4 ^ old_cr4) & pdptr_bits)
 		   && !load_pdptrs(vcpu, vcpu->arch.cr3))
 		return 1;
 	if (cr4 & X86_CR4_VMXE)
 		return 1;
 	kvm_x86_ops->set_cr4(vcpu, cr4);
 	if ((cr4 ^ old_cr4) & pdptr_bits)
 		kvm_mmu_reset_context(vcpu);
 	if ((cr4 ^ old_cr4) & X86_CR4_OSXSAVE)
 		update_cpuid(vcpu);
 	return 0;
 }
 EXPORT_SYMBOL_GPL(kvm_set_cr4);
 int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3)
 {
 	if (cr3 == vcpu->arch.cr3 && !pdptrs_changed(vcpu)) {
 		kvm_mmu_sync_roots(vcpu);
 		kvm_mmu_flush_tlb(vcpu);
 		return 0;
 	}
 	if (is_long_mode(vcpu)) {
 		if (cr3 & CR3_L_MODE_RESERVED_BITS)
 			return 1;
 	} else {
 		if (is_pae(vcpu)) {
 			if (cr3 & CR3_PAE_RESERVED_BITS)
 				return 1;
 			if (is_paging(vcpu) && !load_pdptrs(vcpu, cr3))
 				return 1;
 		}
 		/*
 		 * We don't check reserved bits in nonpae mode, because
 		 * this isn't enforced, and VMware depends on this.
 		 */
 	}
 	/*
 	 * Does the new cr3 value map to physical memory? (Note, we
 	 * catch an invalid cr3 even in real-mode, because it would
 	 * cause trouble later on when we turn on paging anyway.)
 	 *
 	 * A real CPU would silently accept an invalid cr3 and would
 	 * attempt to use it - with largely undefined (and often hard
 	 * to debug) behavior on the guest side.
 	 */
 	if (unlikely(!gfn_to_memslot(vcpu->kvm, cr3 >> PAGE_SHIFT)))
 		return 1;
 	vcpu->arch.cr3 = cr3;
 	vcpu->arch.mmu.new_cr3(vcpu);
 	return 0;
 }
 EXPORT_SYMBOL_GPL(kvm_set_cr3);
 int __kvm_set_cr8(struct kvm_vcpu *vcpu, unsigned long cr8)
 {
 	if (cr8 & CR8_RESERVED_BITS)
 		return 1;
 	if (irqchip_in_kernel(vcpu->kvm))
 		kvm_lapic_set_tpr(vcpu, cr8);
 	else
 		vcpu->arch.cr8 = cr8;
 	return 0;
 }
 void kvm_set_cr8(struct kvm_vcpu *vcpu, unsigned long cr8)
 {
 	if (__kvm_set_cr8(vcpu, cr8))
 		kvm_inject_gp(vcpu, 0);
 }
 EXPORT_SYMBOL_GPL(kvm_set_cr8);
 unsigned long kvm_get_cr8(struct kvm_vcpu *vcpu)
 {
 	if (irqchip_in_kernel(vcpu->kvm))
 		return kvm_lapic_get_cr8(vcpu);
 	else
 		return vcpu->arch.cr8;
 }
 EXPORT_SYMBOL_GPL(kvm_get_cr8);
 static int __kvm_set_dr(struct kvm_vcpu *vcpu, int dr, unsigned long val)
 {
 	switch (dr) {
 	case 0 ... 3:
 		vcpu->arch.db[dr] = val;
 		if (!(vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP))
 			vcpu->arch.eff_db[dr] = val;
 		break;
 	case 4:
 		if (kvm_read_cr4_bits(vcpu, X86_CR4_DE))
 			return 1; /* #UD */
 		/* fall through */
 	case 6:
 		if (val & 0xffffffff00000000ULL)
 			return -1; /* #GP */
 		vcpu->arch.dr6 = (val & DR6_VOLATILE) | DR6_FIXED_1;
 		break;
 	case 5:
 		if (kvm_read_cr4_bits(vcpu, X86_CR4_DE))
 			return 1; /* #UD */
 		/* fall through */
 	default: /* 7 */
 		if (val & 0xffffffff00000000ULL)
 			return -1; /* #GP */
 		vcpu->arch.dr7 = (val & DR7_VOLATILE) | DR7_FIXED_1;
 		if (!(vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP)) {
 			kvm_x86_ops->set_dr7(vcpu, vcpu->arch.dr7);
 			vcpu->arch.switch_db_regs = (val & DR7_BP_EN_MASK);
 		}
 		break;
 	}
 	return 0;
 }
 int kvm_set_dr(struct kvm_vcpu *vcpu, int dr, unsigned long val)
 {
 	int res;
 	res = __kvm_set_dr(vcpu, dr, val);
 	if (res > 0)
 		kvm_queue_exception(vcpu, UD_VECTOR);
 	else if (res < 0)
 		kvm_inject_gp(vcpu, 0);
 	return res;
 }
 EXPORT_SYMBOL_GPL(kvm_set_dr);
 static int _kvm_get_dr(struct kvm_vcpu *vcpu, int dr, unsigned long *val)
 {
 	switch (dr) {
 	case 0 ... 3:
 		*val = vcpu->arch.db[dr];
 		break;
 	case 4:
 		if (kvm_read_cr4_bits(vcpu, X86_CR4_DE))
 			return 1;
 		/* fall through */
 	case 6:
 		*val = vcpu->arch.dr6;
 		break;
 	case 5:
 		if (kvm_read_cr4_bits(vcpu, X86_CR4_DE))
 			return 1;
 		/* fall through */
 	default: /* 7 */
 		*val = vcpu->arch.dr7;
 		break;
 	}
 	return 0;
 }
 int kvm_get_dr(struct kvm_vcpu *vcpu, int dr, unsigned long *val)
 {
 	if (_kvm_get_dr(vcpu, dr, val)) {
 		kvm_queue_exception(vcpu, UD_VECTOR);
 		return 1;
 	}
 	return 0;
 }
 EXPORT_SYMBOL_GPL(kvm_get_dr);
 /*
  * List of msr numbers which we expose to userspace through KVM_GET_MSRS
  * and KVM_SET_MSRS, and KVM_GET_MSR_INDEX_LIST.
  *
  * This list is modified at module load time to reflect the
  * capabilities of the host cpu. This capabilities test skips MSRs that are
  * kvm-specific. Those are put in the beginning of the list.
  */
 #define KVM_SAVE_MSRS_BEGIN	7
 static u32 msrs_to_save[] = {
 	MSR_KVM_SYSTEM_TIME, MSR_KVM_WALL_CLOCK,
 	MSR_KVM_SYSTEM_TIME_NEW, MSR_KVM_WALL_CLOCK_NEW,
 	HV_X64_MSR_GUEST_OS_ID, HV_X64_MSR_HYPERCALL,
 	HV_X64_MSR_APIC_ASSIST_PAGE,
 	MSR_IA32_SYSENTER_CS, MSR_IA32_SYSENTER_ESP, MSR_IA32_SYSENTER_EIP,
 	MSR_K6_STAR,
 #ifdef CONFIG_X86_64
 	MSR_CSTAR, MSR_KERNEL_GS_BASE, MSR_SYSCALL_MASK, MSR_LSTAR,
 #endif
 	MSR_IA32_TSC, MSR_IA32_PERF_STATUS, MSR_IA32_CR_PAT, MSR_VM_HSAVE_PA
 };
 static unsigned num_msrs_to_save;
 static u32 emulated_msrs[] = {
 	MSR_IA32_MISC_ENABLE,
 };
 static int set_efer(struct kvm_vcpu *vcpu, u64 efer)
 {
 	u64 old_efer = vcpu->arch.efer;
 	if (efer & efer_reserved_bits)
 		return 1;
 	if (is_paging(vcpu)
 	    && (vcpu->arch.efer & EFER_LME) != (efer & EFER_LME))
 		return 1;
 	if (efer & EFER_FFXSR) {
 		struct kvm_cpuid_entry2 *feat;
 		feat = kvm_find_cpuid_entry(vcpu, 0x80000001, 0);
 		if (!feat || !(feat->edx & bit(X86_FEATURE_FXSR_OPT)))
 			return 1;
 	}
 	if (efer & EFER_SVME) {
 		struct kvm_cpuid_entry2 *feat;
 		feat = kvm_find_cpuid_entry(vcpu, 0x80000001, 0);
 		if (!feat || !(feat->ecx & bit(X86_FEATURE_SVM)))
 			return 1;
 	}
 	efer &= ~EFER_LMA;
 	efer |= vcpu->arch.efer & EFER_LMA;
 	kvm_x86_ops->set_efer(vcpu, efer);
 	vcpu->arch.mmu.base_role.nxe = (efer & EFER_NX) && !tdp_enabled;
 	kvm_mmu_reset_context(vcpu);
 	/* Update reserved bits */
 	if ((efer ^ old_efer) & EFER_NX)
 		kvm_mmu_reset_context(vcpu);
 	return 0;
 }
 void kvm_enable_efer_bits(u64 mask)
 {
        efer_reserved_bits &= ~mask;
 }
 EXPORT_SYMBOL_GPL(kvm_enable_efer_bits);
 /*
  * Writes msr value into into the appropriate "register".
  * Returns 0 on success, non-0 otherwise.
  * Assumes vcpu_load() was already called.
  */
 int kvm_set_msr(struct kvm_vcpu *vcpu, u32 msr_index, u64 data)
 {
 	return kvm_x86_ops->set_msr(vcpu, msr_index, data);
 }
 /*
  * Adapt set_msr() to msr_io()'s calling convention
  */
 static int do_set_msr(struct kvm_vcpu *vcpu, unsigned index, u64 *data)
 {
 	return kvm_set_msr(vcpu, index, *data);
 }
 static void kvm_write_wall_clock(struct kvm *kvm, gpa_t wall_clock)
 {
 	int version;
 	int r;
 	struct pvclock_wall_clock wc;
 	struct timespec boot;
 	if (!wall_clock)
 		return;
 	r = kvm_read_guest(kvm, wall_clock, &version, sizeof(version));
 	if (r)
 		return;
 	if (version & 1)
 		++version;  /* first time write, random junk */
 	++version;
 	kvm_write_guest(kvm, wall_clock, &version, sizeof(version));
 	/*
 	 * The guest calculates current wall clock time by adding
 	 * system time (updated by kvm_write_guest_time below) to the
 	 * wall clock specified here.  guest system time equals host
 	 * system time for us, thus we must fill in host boot time here.
 	 */
 	getboottime(&boot);
 	wc.sec = boot.tv_sec;
 	wc.nsec = boot.tv_nsec;
 	wc.version = version;
 	kvm_write_guest(kvm, wall_clock, &wc, sizeof(wc));
 	version++;
 	kvm_write_guest(kvm, wall_clock, &version, sizeof(version));
 }
 static uint32_t div_frac(uint32_t dividend, uint32_t divisor)
 {
 	uint32_t quotient, remainder;
 	/* Don't try to replace with do_div(), this one calculates
 	 * "(dividend << 32) / divisor" */
 	__asm__ ( "divl %4"
 		  : "=a" (quotient), "=d" (remainder)
 		  : "0" (0), "1" (dividend), "r" (divisor) );
 	return quotient;
 }
 static void kvm_set_time_scale(uint32_t tsc_khz, struct pvclock_vcpu_time_info *hv_clock)
 {
 	uint64_t nsecs = 1000000000LL;
 	int32_t  shift = 0;
 	uint64_t tps64;
 	uint32_t tps32;
 	tps64 = tsc_khz * 1000LL;
 	while (tps64 > nsecs*2) {
 		tps64 >>= 1;
 		shift--;
 	}
 	tps32 = (uint32_t)tps64;
 	while (tps32 <= (uint32_t)nsecs) {
 		tps32 <<= 1;
 		shift++;
 	}
 	hv_clock->tsc_shift = shift;
 	hv_clock->tsc_to_system_mul = div_frac(nsecs, tps32);
 	pr_debug("%s: tsc_khz %u, tsc_shift %d, tsc_mul %u\n",
 		 __func__, tsc_khz, hv_clock->tsc_shift,
 		 hv_clock->tsc_to_system_mul);
 }
 static DEFINE_PER_CPU(unsigned long, cpu_tsc_khz);
 static void kvm_write_guest_time(struct kvm_vcpu *v)
 {
 	struct timespec ts;
 	unsigned long flags;
 	struct kvm_vcpu_arch *vcpu = &v->arch;
 	void *shared_kaddr;
 	unsigned long this_tsc_khz;
 	if ((!vcpu->time_page))
 		return;
 	this_tsc_khz = get_cpu_var(cpu_tsc_khz);
 	if (unlikely(vcpu->hv_clock_tsc_khz != this_tsc_khz)) {
 		kvm_set_time_scale(this_tsc_khz, &vcpu->hv_clock);
 		vcpu->hv_clock_tsc_khz = this_tsc_khz;
 	}
 	put_cpu_var(cpu_tsc_khz);
 	/* Keep irq disabled to prevent changes to the clock */
 	local_irq_save(flags);
 	kvm_get_msr(v, MSR_IA32_TSC, &vcpu->hv_clock.tsc_timestamp);
 	ktime_get_ts(&ts);
 	monotonic_to_bootbased(&ts);
 	local_irq_restore(flags);
 	/* With all the info we got, fill in the values */
 	vcpu->hv_clock.system_time = ts.tv_nsec +
 				     (NSEC_PER_SEC * (u64)ts.tv_sec) + v->kvm->arch.kvmclock_offset;
 	vcpu->hv_clock.flags = 0;
 	/*
 	 * The interface expects us to write an even number signaling that the
 	 * update is finished. Since the guest won't see the intermediate
 	 * state, we just increase by 2 at the end.
 	 */
 	vcpu->hv_clock.version += 2;
 	shared_kaddr = kmap_atomic(vcpu->time_page, KM_USER0);
 	memcpy(shared_kaddr + vcpu->time_offset, &vcpu->hv_clock,
 	       sizeof(vcpu->hv_clock));
 	kunmap_atomic(shared_kaddr, KM_USER0);
 	mark_page_dirty(v->kvm, vcpu->time >> PAGE_SHIFT);
 }
 static int kvm_request_guest_time_update(struct kvm_vcpu *v)
 {
 	struct kvm_vcpu_arch *vcpu = &v->arch;
 	if (!vcpu->time_page)
 		return 0;
 	set_bit(KVM_REQ_KVMCLOCK_UPDATE, &v->requests);
 	return 1;
 }
 static bool msr_mtrr_valid(unsigned msr)
 {
 	switch (msr) {
 	case 0x200 ... 0x200 + 2 * KVM_NR_VAR_MTRR - 1:
 	case MSR_MTRRfix64K_00000:
 	case MSR_MTRRfix16K_80000:
 	case MSR_MTRRfix16K_A0000:
 	case MSR_MTRRfix4K_C0000:
 	case MSR_MTRRfix4K_C8000:
 	case MSR_MTRRfix4K_D0000:
 	case MSR_MTRRfix4K_D8000:
 	case MSR_MTRRfix4K_E0000:
 	case MSR_MTRRfix4K_E8000:
 	case MSR_MTRRfix4K_F0000:
 	case MSR_MTRRfix4K_F8000:
 	case MSR_MTRRdefType:
 	case MSR_IA32_CR_PAT:
 		return true;
 	case 0x2f8:
 		return true;
 	}
 	return false;
 }
 static bool valid_pat_type(unsigned t)
 {
 	return t < 8 && (1 << t) & 0xf3; /* 0, 1, 4, 5, 6, 7 */
 }
 static bool valid_mtrr_type(unsigned t)
 {
 	return t < 8 && (1 << t) & 0x73; /* 0, 1, 4, 5, 6 */
 }
 static bool mtrr_valid(struct kvm_vcpu *vcpu, u32 msr, u64 data)
 {
 	int i;
 	if (!msr_mtrr_valid(msr))
 		return false;
 	if (msr == MSR_IA32_CR_PAT) {
 		for (i = 0; i < 8; i++)
 			if (!valid_pat_type((data >> (i * 8)) & 0xff))
 				return false;
 		return true;
 	} else if (msr == MSR_MTRRdefType) {
 		if (data & ~0xcff)
 			return false;
 		return valid_mtrr_type(data & 0xff);
 	} else if (msr >= MSR_MTRRfix64K_00000 && msr <= MSR_MTRRfix4K_F8000) {
 		for (i = 0; i < 8 ; i++)
 			if (!valid_mtrr_type((data >> (i * 8)) & 0xff))
 				return false;
 		return true;
 	}
 	/* variable MTRRs */
 	return valid_mtrr_type(data & 0xff);
 }
 static int set_msr_mtrr(struct kvm_vcpu *vcpu, u32 msr, u64 data)
 {
 	u64 *p = (u64 *)&vcpu->arch.mtrr_state.fixed_ranges;
 	if (!mtrr_valid(vcpu, msr, data))
 		return 1;
 	if (msr == MSR_MTRRdefType) {
 		vcpu->arch.mtrr_state.def_type = data;
 		vcpu->arch.mtrr_state.enabled = (data & 0xc00) >> 10;
 	} else if (msr == MSR_MTRRfix64K_00000)
 		p[0] = data;
 	else if (msr == MSR_MTRRfix16K_80000 || msr == MSR_MTRRfix16K_A0000)
 		p[1 + msr - MSR_MTRRfix16K_80000] = data;
 	else if (msr >= MSR_MTRRfix4K_C0000 && msr <= MSR_MTRRfix4K_F8000)
 		p[3 + msr - MSR_MTRRfix4K_C0000] = data;
 	else if (msr == MSR_IA32_CR_PAT)
 		vcpu->arch.pat = data;
 	else {	/* Variable MTRRs */
 		int idx, is_mtrr_mask;
 		u64 *pt;
 		idx = (msr - 0x200) / 2;
 		is_mtrr_mask = msr - 0x200 - 2 * idx;
 		if (!is_mtrr_mask)
 			pt =
 			  (u64 *)&vcpu->arch.mtrr_state.var_ranges[idx].base_lo;
 		else
 			pt =
 			  (u64 *)&vcpu->arch.mtrr_state.var_ranges[idx].mask_lo;
 		*pt = data;
 	}
 	kvm_mmu_reset_context(vcpu);
 	return 0;
 }
 static int set_msr_mce(struct kvm_vcpu *vcpu, u32 msr, u64 data)
 {
 	u64 mcg_cap = vcpu->arch.mcg_cap;
 	unsigned bank_num = mcg_cap & 0xff;
 	switch (msr) {
 	case MSR_IA32_MCG_STATUS:
 		vcpu->arch.mcg_status = data;
 		break;
 	case MSR_IA32_MCG_CTL:
 		if (!(mcg_cap & MCG_CTL_P))
 			return 1;
 		if (data != 0 && data != ~(u64)0)
 			return -1;
 		vcpu->arch.mcg_ctl = data;
 		break;
 	default:
 		if (msr >= MSR_IA32_MC0_CTL &&
 		    msr < MSR_IA32_MC0_CTL + 4 * bank_num) {
 			u32 offset = msr - MSR_IA32_MC0_CTL;
 			/* only 0 or all 1s can be written to IA32_MCi_CTL
 			 * some Linux kernels though clear bit 10 in bank 4 to
 			 * workaround a BIOS/GART TBL issue on AMD K8s, ignore
 			 * this to avoid an uncatched #GP in the guest
 			 */
 			if ((offset & 0x3) == 0 &&
 			    data != 0 && (data | (1 << 10)) != ~(u64)0)
 				return -1;
 			vcpu->arch.mce_banks[offset] = data;
 			break;
 		}
 		return 1;
 	}
 	return 0;
 }
 static int xen_hvm_config(struct kvm_vcpu *vcpu, u64 data)
 {
 	struct kvm *kvm = vcpu->kvm;
 	int lm = is_long_mode(vcpu);
 	u8 *blob_addr = lm ? (u8 *)(long)kvm->arch.xen_hvm_config.blob_addr_64
 		: (u8 *)(long)kvm->arch.xen_hvm_config.blob_addr_32;
 	u8 blob_size = lm ? kvm->arch.xen_hvm_config.blob_size_64
 		: kvm->arch.xen_hvm_config.blob_size_32;
 	u32 page_num = data & ~PAGE_MASK;
 	u64 page_addr = data & PAGE_MASK;
 	u8 *page;
 	int r;
 	r = -E2BIG;
 	if (page_num >= blob_size)
 		goto out;
 	r = -ENOMEM;
 	page = kzalloc(PAGE_SIZE, GFP_KERNEL);
 	if (!page)
 		goto out;
 	r = -EFAULT;
 	if (copy_from_user(page, blob_addr + (page_num * PAGE_SIZE), PAGE_SIZE))
 		goto out_free;
 	if (kvm_write_guest(kvm, page_addr, page, PAGE_SIZE))
 		goto out_free;
 	r = 0;
 out_free:
 	kfree(page);
 out:
 	return r;
 }
 static bool kvm_hv_hypercall_enabled(struct kvm *kvm)
 {
 	return kvm->arch.hv_hypercall & HV_X64_MSR_HYPERCALL_ENABLE;
 }
 static bool kvm_hv_msr_partition_wide(u32 msr)
 {
 	bool r = false;
 	switch (msr) {
 	case HV_X64_MSR_GUEST_OS_ID:
 	case HV_X64_MSR_HYPERCALL:
 		r = true;
 		break;
 	}
 	return r;
 }
 static int set_msr_hyperv_pw(struct kvm_vcpu *vcpu, u32 msr, u64 data)
 {
 	struct kvm *kvm = vcpu->kvm;
 	switch (msr) {
 	case HV_X64_MSR_GUEST_OS_ID:
 		kvm->arch.hv_guest_os_id = data;
 		/* setting guest os id to zero disables hypercall page */
 		if (!kvm->arch.hv_guest_os_id)
 			kvm->arch.hv_hypercall &= ~HV_X64_MSR_HYPERCALL_ENABLE;
 		break;
 	case HV_X64_MSR_HYPERCALL: {
 		u64 gfn;
 		unsigned long addr;
 		u8 instructions[4];
 		/* if guest os id is not set hypercall should remain disabled */
 		if (!kvm->arch.hv_guest_os_id)
 			break;
 		if (!(data & HV_X64_MSR_HYPERCALL_ENABLE)) {
 			kvm->arch.hv_hypercall = data;
 			break;
 		}
 		gfn = data >> HV_X64_MSR_HYPERCALL_PAGE_ADDRESS_SHIFT;
 		addr = gfn_to_hva(kvm, gfn);
 		if (kvm_is_error_hva(addr))
 			return 1;
 		kvm_x86_ops->patch_hypercall(vcpu, instructions);
 		((unsigned char *)instructions)[3] = 0xc3; /* ret */
 		if (copy_to_user((void __user *)addr, instructions, 4))
 			return 1;
 		kvm->arch.hv_hypercall = data;
 		break;
 	}
 	default:
 		pr_unimpl(vcpu, "HYPER-V unimplemented wrmsr: 0x%x "
 			  "data 0x%llx\n", msr, data);
 		return 1;
 	}
 	return 0;
 }
 static int set_msr_hyperv(struct kvm_vcpu *vcpu, u32 msr, u64 data)
 {
 	switch (msr) {
 	case HV_X64_MSR_APIC_ASSIST_PAGE: {
 		unsigned long addr;
 		if (!(data & HV_X64_MSR_APIC_ASSIST_PAGE_ENABLE)) {
 			vcpu->arch.hv_vapic = data;
 			break;
 		}
 		addr = gfn_to_hva(vcpu->kvm, data >>
 				  HV_X64_MSR_APIC_ASSIST_PAGE_ADDRESS_SHIFT);
 		if (kvm_is_error_hva(addr))
 			return 1;
 		if (clear_user((void __user *)addr, PAGE_SIZE))
 			return 1;
 		vcpu->arch.hv_vapic = data;
 		break;
 	}
 	case HV_X64_MSR_EOI:
 		return kvm_hv_vapic_msr_write(vcpu, APIC_EOI, data);
 	case HV_X64_MSR_ICR:
 		return kvm_hv_vapic_msr_write(vcpu, APIC_ICR, data);
 	case HV_X64_MSR_TPR:
 		return kvm_hv_vapic_msr_write(vcpu, APIC_TASKPRI, data);
 	default:
 		pr_unimpl(vcpu, "HYPER-V unimplemented wrmsr: 0x%x "
 			  "data 0x%llx\n", msr, data);
 		return 1;
 	}
 	return 0;
 }
 int kvm_set_msr_common(struct kvm_vcpu *vcpu, u32 msr, u64 data)
 {
 	switch (msr) {
 	case MSR_EFER:
 		return set_efer(vcpu, data);
 	case MSR_K7_HWCR:
 		data &= ~(u64)0x40;	/* ignore flush filter disable */
 		data &= ~(u64)0x100;	/* ignore ignne emulation enable */
 		if (data != 0) {
 			pr_unimpl(vcpu, "unimplemented HWCR wrmsr: 0x%llx\n",
 				data);
 			return 1;
 		}
 		break;
 	case MSR_FAM10H_MMIO_CONF_BASE:
 		if (data != 0) {
 			pr_unimpl(vcpu, "unimplemented MMIO_CONF_BASE wrmsr: "
 				"0x%llx\n", data);
 			return 1;
 		}
 		break;
 	case MSR_AMD64_NB_CFG:
 		break;
 	case MSR_IA32_DEBUGCTLMSR:
 		if (!data) {
 			/* We support the non-activated case already */
 			break;
 		} else if (data & ~(DEBUGCTLMSR_LBR | DEBUGCTLMSR_BTF)) {
 			/* Values other than LBR and BTF are vendor-specific,
 			   thus reserved and should throw a #GP */
 			return 1;
 		}
 		pr_unimpl(vcpu, "%s: MSR_IA32_DEBUGCTLMSR 0x%llx, nop\n",
 			__func__, data);
 		break;
 	case MSR_IA32_UCODE_REV:
 	case MSR_IA32_UCODE_WRITE:
 	case MSR_VM_HSAVE_PA:
 	case MSR_AMD64_PATCH_LOADER:
 		break;
 	case 0x200 ... 0x2ff:
 		return set_msr_mtrr(vcpu, msr, data);
 	case MSR_IA32_APICBASE:
 		kvm_set_apic_base(vcpu, data);
 		break;
 	case APIC_BASE_MSR ... APIC_BASE_MSR + 0x3ff:
 		return kvm_x2apic_msr_write(vcpu, msr, data);
 	case MSR_IA32_MISC_ENABLE:
 		vcpu->arch.ia32_misc_enable_msr = data;
 		break;
 	case MSR_KVM_WALL_CLOCK_NEW:
 	case MSR_KVM_WALL_CLOCK:
 		vcpu->kvm->arch.wall_clock = data;
 		kvm_write_wall_clock(vcpu->kvm, data);
 		break;
 	case MSR_KVM_SYSTEM_TIME_NEW:
 	case MSR_KVM_SYSTEM_TIME: {
 		if (vcpu->arch.time_page) {
 			kvm_release_page_dirty(vcpu->arch.time_page);
 			vcpu->arch.time_page = NULL;
 		}
 		vcpu->arch.time = data;
 		/* we verify if the enable bit is set... */
 		if (!(data & 1))
 			break;
 		/* ...but clean it before doing the actual write */
 		vcpu->arch.time_offset = data & ~(PAGE_MASK | 1);
 		vcpu->arch.time_page =
 				gfn_to_page(vcpu->kvm, data >> PAGE_SHIFT);
 		if (is_error_page(vcpu->arch.time_page)) {
 			kvm_release_page_clean(vcpu->arch.time_page);
 			vcpu->arch.time_page = NULL;
 		}
 		kvm_request_guest_time_update(vcpu);
 		break;
 	}
 	case MSR_IA32_MCG_CTL:
 	case MSR_IA32_MCG_STATUS:
 	case MSR_IA32_MC0_CTL ... MSR_IA32_MC0_CTL + 4 * KVM_MAX_MCE_BANKS - 1:
 		return set_msr_mce(vcpu, msr, data);
 	/* Performance counters are not protected by a CPUID bit,
 	 * so we should check all of them in the generic path for the sake of
 	 * cross vendor migration.
 	 * Writing a zero into the event select MSRs disables them,
 	 * which we perfectly emulate ;-). Any other value should be at least
 	 * reported, some guests depend on them.
 	 */
 	case MSR_P6_EVNTSEL0:
 	case MSR_P6_EVNTSEL1:
 	case MSR_K7_EVNTSEL0:
 	case MSR_K7_EVNTSEL1:
 	case MSR_K7_EVNTSEL2:
 	case MSR_K7_EVNTSEL3:
 		if (data != 0)
 			pr_unimpl(vcpu, "unimplemented perfctr wrmsr: "
 				"0x%x data 0x%llx\n", msr, data);
 		break;
 	/* at least RHEL 4 unconditionally writes to the perfctr registers,
 	 * so we ignore writes to make it happy.
 	 */
 	case MSR_P6_PERFCTR0:
 	case MSR_P6_PERFCTR1:
 	case MSR_K7_PERFCTR0:
 	case MSR_K7_PERFCTR1:
 	case MSR_K7_PERFCTR2:
 	case MSR_K7_PERFCTR3:
 		pr_unimpl(vcpu, "unimplemented perfctr wrmsr: "
 			"0x%x data 0x%llx\n", msr, data);
 		break;
 	case HV_X64_MSR_GUEST_OS_ID ... HV_X64_MSR_SINT15:
 		if (kvm_hv_msr_partition_wide(msr)) {
 			int r;
 			mutex_lock(&vcpu->kvm->lock);
 			r = set_msr_hyperv_pw(vcpu, msr, data);
 			mutex_unlock(&vcpu->kvm->lock);
 			return r;
 		} else
 			return set_msr_hyperv(vcpu, msr, data);
 		break;
 	default:
 		if (msr && (msr == vcpu->kvm->arch.xen_hvm_config.msr))
 			return xen_hvm_config(vcpu, data);
 		if (!ignore_msrs) {
 			pr_unimpl(vcpu, "unhandled wrmsr: 0x%x data %llx\n",
 				msr, data);
 			return 1;
 		} else {
 			pr_unimpl(vcpu, "ignored wrmsr: 0x%x data %llx\n",
 				msr, data);
 			break;
 		}
 	}
 	return 0;
 }
 EXPORT_SYMBOL_GPL(kvm_set_msr_common);
 /*
  * Reads an msr value (of 'msr_index') into 'pdata'.
  * Returns 0 on success, non-0 otherwise.
  * Assumes vcpu_load() was already called.
  */
 int kvm_get_msr(struct kvm_vcpu *vcpu, u32 msr_index, u64 *pdata)
 {
 	return kvm_x86_ops->get_msr(vcpu, msr_index, pdata);
 }
 static int get_msr_mtrr(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata)
 {
 	u64 *p = (u64 *)&vcpu->arch.mtrr_state.fixed_ranges;
 	if (!msr_mtrr_valid(msr))
 		return 1;
 	if (msr == MSR_MTRRdefType)
 		*pdata = vcpu->arch.mtrr_state.def_type +
 			 (vcpu->arch.mtrr_state.enabled << 10);
 	else if (msr == MSR_MTRRfix64K_00000)
 		*pdata = p[0];
 	else if (msr == MSR_MTRRfix16K_80000 || msr == MSR_MTRRfix16K_A0000)
 		*pdata = p[1 + msr - MSR_MTRRfix16K_80000];
 	else if (msr >= MSR_MTRRfix4K_C0000 && msr <= MSR_MTRRfix4K_F8000)
 		*pdata = p[3 + msr - MSR_MTRRfix4K_C0000];
 	else if (msr == MSR_IA32_CR_PAT)
 		*pdata = vcpu->arch.pat;
 	else {	/* Variable MTRRs */
 		int idx, is_mtrr_mask;
 		u64 *pt;
 		idx = (msr - 0x200) / 2;
 		is_mtrr_mask = msr - 0x200 - 2 * idx;
 		if (!is_mtrr_mask)
 			pt =
 			  (u64 *)&vcpu->arch.mtrr_state.var_ranges[idx].base_lo;
 		else
 			pt =
 			  (u64 *)&vcpu->arch.mtrr_state.var_ranges[idx].mask_lo;
 		*pdata = *pt;
 	}
 	return 0;
 }
 static int get_msr_mce(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata)
 {
 	u64 data;
 	u64 mcg_cap = vcpu->arch.mcg_cap;
 	unsigned bank_num = mcg_cap & 0xff;
 	switch (msr) {
 	case MSR_IA32_P5_MC_ADDR:
 	case MSR_IA32_P5_MC_TYPE:
 		data = 0;
 		break;
 	case MSR_IA32_MCG_CAP:
 		data = vcpu->arch.mcg_cap;
 		break;
 	case MSR_IA32_MCG_CTL:
 		if (!(mcg_cap & MCG_CTL_P))
 			return 1;
 		data = vcpu->arch.mcg_ctl;
 		break;
 	case MSR_IA32_MCG_STATUS:
 		data = vcpu->arch.mcg_status;
 		break;
 	default:
 		if (msr >= MSR_IA32_MC0_CTL &&
 		    msr < MSR_IA32_MC0_CTL + 4 * bank_num) {
 			u32 offset = msr - MSR_IA32_MC0_CTL;
 			data = vcpu->arch.mce_banks[offset];
 			break;
 		}
 		return 1;
 	}
 	*pdata = data;
 	return 0;
 }
 static int get_msr_hyperv_pw(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata)
 {
 	u64 data = 0;
 	struct kvm *kvm = vcpu->kvm;
 	switch (msr) {
 	case HV_X64_MSR_GUEST_OS_ID:
 		data = kvm->arch.hv_guest_os_id;
 		break;
 	case HV_X64_MSR_HYPERCALL:
 		data = kvm->arch.hv_hypercall;
 		break;
 	default:
 		pr_unimpl(vcpu, "Hyper-V unhandled rdmsr: 0x%x\n", msr);
 		return 1;
 	}
 	*pdata = data;
 	return 0;
 }
 static int get_msr_hyperv(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata)
 {
 	u64 data = 0;
 	switch (msr) {
 	case HV_X64_MSR_VP_INDEX: {
 		int r;
 		struct kvm_vcpu *v;
 		kvm_for_each_vcpu(r, v, vcpu->kvm)
 			if (v == vcpu)
 				data = r;
 		break;
 	}
 	case HV_X64_MSR_EOI:
 		return kvm_hv_vapic_msr_read(vcpu, APIC_EOI, pdata);
 	case HV_X64_MSR_ICR:
 		return kvm_hv_vapic_msr_read(vcpu, APIC_ICR, pdata);
 	case HV_X64_MSR_TPR:
 		return kvm_hv_vapic_msr_read(vcpu, APIC_TASKPRI, pdata);
 	default:
 		pr_unimpl(vcpu, "Hyper-V unhandled rdmsr: 0x%x\n", msr);
 		return 1;
 	}
 	*pdata = data;
 	return 0;
 }
 int kvm_get_msr_common(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata)
 {
 	u64 data;
 	switch (msr) {
 	case MSR_IA32_PLATFORM_ID:
 	case MSR_IA32_UCODE_REV:
 	case MSR_IA32_EBL_CR_POWERON:
 	case MSR_IA32_DEBUGCTLMSR:
 	case MSR_IA32_LASTBRANCHFROMIP:
 	case MSR_IA32_LASTBRANCHTOIP:
 	case MSR_IA32_LASTINTFROMIP:
 	case MSR_IA32_LASTINTTOIP:
 	case MSR_K8_SYSCFG:
 	case MSR_K7_HWCR:
 	case MSR_VM_HSAVE_PA:
 	case MSR_P6_PERFCTR0:
 	case MSR_P6_PERFCTR1:
 	case MSR_P6_EVNTSEL0:
 	case MSR_P6_EVNTSEL1:
 	case MSR_K7_EVNTSEL0:
 	case MSR_K7_PERFCTR0:
 	case MSR_K8_INT_PENDING_MSG:
 	case MSR_AMD64_NB_CFG:
 	case MSR_FAM10H_MMIO_CONF_BASE:
 		data = 0;
 		break;
 	case MSR_MTRRcap:
 		data = 0x500 | KVM_NR_VAR_MTRR;
 		break;
 	case 0x200 ... 0x2ff:
 		return get_msr_mtrr(vcpu, msr, pdata);
 	case 0xcd: /* fsb frequency */
 		data = 3;
 		break;
 	case MSR_IA32_APICBASE:
 		data = kvm_get_apic_base(vcpu);
 		break;
 	case APIC_BASE_MSR ... APIC_BASE_MSR + 0x3ff:
 		return kvm_x2apic_msr_read(vcpu, msr, pdata);
 		break;
 	case MSR_IA32_MISC_ENABLE:
 		data = vcpu->arch.ia32_misc_enable_msr;
 		break;
 	case MSR_IA32_PERF_STATUS:
 		/* TSC increment by tick */
 		data = 1000ULL;
 		/* CPU multiplier */
 		data |= (((uint64_t)4ULL) << 40);
 		break;
 	case MSR_EFER:
 		data = vcpu->arch.efer;
 		break;
 	case MSR_KVM_WALL_CLOCK:
 	case MSR_KVM_WALL_CLOCK_NEW:
 		data = vcpu->kvm->arch.wall_clock;
 		break;
 	case MSR_KVM_SYSTEM_TIME:
 	case MSR_KVM_SYSTEM_TIME_NEW:
 		data = vcpu->arch.time;
 		break;
 	case MSR_IA32_P5_MC_ADDR:
 	case MSR_IA32_P5_MC_TYPE:
 	case MSR_IA32_MCG_CAP:
 	case MSR_IA32_MCG_CTL:
 	case MSR_IA32_MCG_STATUS:
 	case MSR_IA32_MC0_CTL ... MSR_IA32_MC0_CTL + 4 * KVM_MAX_MCE_BANKS - 1:
 		return get_msr_mce(vcpu, msr, pdata);
 	case HV_X64_MSR_GUEST_OS_ID ... HV_X64_MSR_SINT15:
 		if (kvm_hv_msr_partition_wide(msr)) {
 			int r;
 			mutex_lock(&vcpu->kvm->lock);
 			r = get_msr_hyperv_pw(vcpu, msr, pdata);
 			mutex_unlock(&vcpu->kvm->lock);
 			return r;
 		} else
 			return get_msr_hyperv(vcpu, msr, pdata);
 		break;
 	default:
 		if (!ignore_msrs) {
 			pr_unimpl(vcpu, "unhandled rdmsr: 0x%x\n", msr);
 			return 1;
 		} else {
 			pr_unimpl(vcpu, "ignored rdmsr: 0x%x\n", msr);
 			data = 0;
 		}
 		break;
 	}
 	*pdata = data;
 	return 0;
 }
 EXPORT_SYMBOL_GPL(kvm_get_msr_common);
 /*
  * Read or write a bunch of msrs. All parameters are kernel addresses.
  *
  * @return number of msrs set successfully.
  */
 static int __msr_io(struct kvm_vcpu *vcpu, struct kvm_msrs *msrs,
 		    struct kvm_msr_entry *entries,
 		    int (*do_msr)(struct kvm_vcpu *vcpu,
 				  unsigned index, u64 *data))
 {
 	int i, idx;
 	idx = srcu_read_lock(&vcpu->kvm->srcu);
 	for (i = 0; i < msrs->nmsrs; ++i)
 		if (do_msr(vcpu, entries[i].index, &entries[i].data))
 			break;
 	srcu_read_unlock(&vcpu->kvm->srcu, idx);
 	return i;
 }
 /*
  * Read or write a bunch of msrs. Parameters are user addresses.
  *
  * @return number of msrs set successfully.
  */
 static int msr_io(struct kvm_vcpu *vcpu, struct kvm_msrs __user *user_msrs,
 		  int (*do_msr)(struct kvm_vcpu *vcpu,
 				unsigned index, u64 *data),
 		  int writeback)
 {
 	struct kvm_msrs msrs;
 	struct kvm_msr_entry *entries;
 	int r, n;
 	unsigned size;
 	r = -EFAULT;
 	if (copy_from_user(&msrs, user_msrs, sizeof msrs))
 		goto out;
 	r = -E2BIG;
 	if (msrs.nmsrs >= MAX_IO_MSRS)
 		goto out;
 	r = -ENOMEM;
 	size = sizeof(struct kvm_msr_entry) * msrs.nmsrs;
 	entries = kmalloc(size, GFP_KERNEL);
 	if (!entries)
 		goto out;
 	r = -EFAULT;
 	if (copy_from_user(entries, user_msrs->entries, size))
 		goto out_free;
 	r = n = __msr_io(vcpu, &msrs, entries, do_msr);
 	if (r < 0)
 		goto out_free;
 	r = -EFAULT;
 	if (writeback && copy_to_user(user_msrs->entries, entries, size))
 		goto out_free;
 	r = n;
 out_free:
 	kfree(entries);
 out:
 	return r;
 }
 int kvm_dev_ioctl_check_extension(long ext)
 {
 	int r;
 	switch (ext) {
 	case KVM_CAP_IRQCHIP:
 	case KVM_CAP_HLT:
 	case KVM_CAP_MMU_SHADOW_CACHE_CONTROL:
 	case KVM_CAP_SET_TSS_ADDR:
 	case KVM_CAP_EXT_CPUID:
 	case KVM_CAP_CLOCKSOURCE:
 	case KVM_CAP_PIT:
 	case KVM_CAP_NOP_IO_DELAY:
 	case KVM_CAP_MP_STATE:
 	case KVM_CAP_SYNC_MMU:
 	case KVM_CAP_REINJECT_CONTROL:
 	case KVM_CAP_IRQ_INJECT_STATUS:
 	case KVM_CAP_ASSIGN_DEV_IRQ:
 	case KVM_CAP_IRQFD:
 	case KVM_CAP_IOEVENTFD:
 	case KVM_CAP_PIT2:
 	case KVM_CAP_PIT_STATE2:
 	case KVM_CAP_SET_IDENTITY_MAP_ADDR:
 	case KVM_CAP_XEN_HVM:
 	case KVM_CAP_ADJUST_CLOCK:
 	case KVM_CAP_VCPU_EVENTS:
 	case KVM_CAP_HYPERV:
 	case KVM_CAP_HYPERV_VAPIC:
 	case KVM_CAP_HYPERV_SPIN:
 	case KVM_CAP_PCI_SEGMENT:
 	case KVM_CAP_DEBUGREGS:
 	case KVM_CAP_X86_ROBUST_SINGLESTEP:
 	case KVM_CAP_XSAVE:
 		r = 1;
 		break;
 	case KVM_CAP_COALESCED_MMIO:
 		r = KVM_COALESCED_MMIO_PAGE_OFFSET;
 		break;
 	case KVM_CAP_VAPIC:
 		r = !kvm_x86_ops->cpu_has_accelerated_tpr();
 		break;
 	case KVM_CAP_NR_VCPUS:
 		r = KVM_MAX_VCPUS;
 		break;
 	case KVM_CAP_NR_MEMSLOTS:
 		r = KVM_MEMORY_SLOTS;
 		break;
 	case KVM_CAP_PV_MMU:	/* obsolete */
 		r = 0;
 		break;
 	case KVM_CAP_IOMMU:
 		r = iommu_found();
 		break;
 	case KVM_CAP_MCE:
 		r = KVM_MAX_MCE_BANKS;
 		break;
 	case KVM_CAP_XCRS:
 		r = cpu_has_xsave;
 		break;
 	default:
 		r = 0;
 		break;
 	}
 	return r;
 }
 long kvm_arch_dev_ioctl(struct file *filp,
 			unsigned int ioctl, unsigned long arg)
 {
 	void __user *argp = (void __user *)arg;
 	long r;
 	switch (ioctl) {
 	case KVM_GET_MSR_INDEX_LIST: {
 		struct kvm_msr_list __user *user_msr_list = argp;
 		struct kvm_msr_list msr_list;
 		unsigned n;
 		r = -EFAULT;
 		if (copy_from_user(&msr_list, user_msr_list, sizeof msr_list))
 			goto out;
 		n = msr_list.nmsrs;
 		msr_list.nmsrs = num_msrs_to_save + ARRAY_SIZE(emulated_msrs);
 		if (copy_to_user(user_msr_list, &msr_list, sizeof msr_list))
 			goto out;
 		r = -E2BIG;
 		if (n < msr_list.nmsrs)
 			goto out;
 		r = -EFAULT;
 		if (copy_to_user(user_msr_list->indices, &msrs_to_save,
 				 num_msrs_to_save * sizeof(u32)))
 			goto out;
 		if (copy_to_user(user_msr_list->indices + num_msrs_to_save,
 				 &emulated_msrs,
 				 ARRAY_SIZE(emulated_msrs) * sizeof(u32)))
 			goto out;
 		r = 0;
 		break;
 	}
 	case KVM_GET_SUPPORTED_CPUID: {
 		struct kvm_cpuid2 __user *cpuid_arg = argp;
 		struct kvm_cpuid2 cpuid;
 		r = -EFAULT;
 		if (copy_from_user(&cpuid, cpuid_arg, sizeof cpuid))
 			goto out;
 		r = kvm_dev_ioctl_get_supported_cpuid(&cpuid,
 						      cpuid_arg->entries);
 		if (r)
 			goto out;
 		r = -EFAULT;
 		if (copy_to_user(cpuid_arg, &cpuid, sizeof cpuid))
 			goto out;
 		r = 0;
 		break;
 	}
 	case KVM_X86_GET_MCE_CAP_SUPPORTED: {
 		u64 mce_cap;
 		mce_cap = KVM_MCE_CAP_SUPPORTED;
 		r = -EFAULT;
 		if (copy_to_user(argp, &mce_cap, sizeof mce_cap))
 			goto out;
 		r = 0;
 		break;
 	}
 	default:
 		r = -EINVAL;
 	}
 out:
 	return r;
 }
 void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
 {
 	kvm_x86_ops->vcpu_load(vcpu, cpu);
 	if (unlikely(per_cpu(cpu_tsc_khz, cpu) == 0)) {
 		unsigned long khz = cpufreq_quick_get(cpu);
 		if (!khz)
 			khz = tsc_khz;
 		per_cpu(cpu_tsc_khz, cpu) = khz;
 	}
 	kvm_request_guest_time_update(vcpu);
 }
 void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
 {
 	kvm_x86_ops->vcpu_put(vcpu);
 	kvm_put_guest_fpu(vcpu);
 }
 static int is_efer_nx(void)
 {
 	unsigned long long efer = 0;
 	rdmsrl_safe(MSR_EFER, &efer);
 	return efer & EFER_NX;
 }
 static void cpuid_fix_nx_cap(struct kvm_vcpu *vcpu)
 {
 	int i;
 	struct kvm_cpuid_entry2 *e, *entry;
 	entry = NULL;
 	for (i = 0; i < vcpu->arch.cpuid_nent; ++i) {
 		e = &vcpu->arch.cpuid_entries[i];
 		if (e->function == 0x80000001) {
 			entry = e;
 			break;
 		}
 	}
 	if (entry && (entry->edx & (1 << 20)) && !is_efer_nx()) {
 		entry->edx &= ~(1 << 20);
 		printk(KERN_INFO "kvm: guest NX capability removed\n");
 	}
 }
 /* when an old userspace process fills a new kernel module */
 static int kvm_vcpu_ioctl_set_cpuid(struct kvm_vcpu *vcpu,
 				    struct kvm_cpuid *cpuid,
 				    struct kvm_cpuid_entry __user *entries)
 {
 	int r, i;
 	struct kvm_cpuid_entry *cpuid_entries;
 	r = -E2BIG;
 	if (cpuid->nent > KVM_MAX_CPUID_ENTRIES)
 		goto out;
 	r = -ENOMEM;
 	cpuid_entries = vmalloc(sizeof(struct kvm_cpuid_entry) * cpuid->nent);
 	if (!cpuid_entries)
 		goto out;
 	r = -EFAULT;
 	if (copy_from_user(cpuid_entries, entries,
 			   cpuid->nent * sizeof(struct kvm_cpuid_entry)))
 		goto out_free;
 	for (i = 0; i < cpuid->nent; i++) {
 		vcpu->arch.cpuid_entries[i].function = cpuid_entries[i].function;
 		vcpu->arch.cpuid_entries[i].eax = cpuid_entries[i].eax;
 		vcpu->arch.cpuid_entries[i].ebx = cpuid_entries[i].ebx;
 		vcpu->arch.cpuid_entries[i].ecx = cpuid_entries[i].ecx;
 		vcpu->arch.cpuid_entries[i].edx = cpuid_entries[i].edx;
 		vcpu->arch.cpuid_entries[i].index = 0;
 		vcpu->arch.cpuid_entries[i].flags = 0;
 		vcpu->arch.cpuid_entries[i].padding[0] = 0;
 		vcpu->arch.cpuid_entries[i].padding[1] = 0;
 		vcpu->arch.cpuid_entries[i].padding[2] = 0;
 	}
 	vcpu->arch.cpuid_nent = cpuid->nent;
 	cpuid_fix_nx_cap(vcpu);
 	r = 0;
 	kvm_apic_set_version(vcpu);
 	kvm_x86_ops->cpuid_update(vcpu);
 	update_cpuid(vcpu);
 out_free:
 	vfree(cpuid_entries);
 out:
 	return r;
 }
 static int kvm_vcpu_ioctl_set_cpuid2(struct kvm_vcpu *vcpu,
 				     struct kvm_cpuid2 *cpuid,
 				     struct kvm_cpuid_entry2 __user *entries)
 {
 	int r;
 	r = -E2BIG;
 	if (cpuid->nent > KVM_MAX_CPUID_ENTRIES)
 		goto out;
 	r = -EFAULT;
 	if (copy_from_user(&vcpu->arch.cpuid_entries, entries,
 			   cpuid->nent * sizeof(struct kvm_cpuid_entry2)))
 		goto out;
 	vcpu->arch.cpuid_nent = cpuid->nent;
 	kvm_apic_set_version(vcpu);
 	kvm_x86_ops->cpuid_update(vcpu);
 	update_cpuid(vcpu);
 	return 0;
 out:
 	return r;
 }
 static int kvm_vcpu_ioctl_get_cpuid2(struct kvm_vcpu *vcpu,
 				     struct kvm_cpuid2 *cpuid,
 				     struct kvm_cpuid_entry2 __user *entries)
 {
 	int r;
 	r = -E2BIG;
 	if (cpuid->nent < vcpu->arch.cpuid_nent)
 		goto out;
 	r = -EFAULT;
 	if (copy_to_user(entries, &vcpu->arch.cpuid_entries,
 			 vcpu->arch.cpuid_nent * sizeof(struct kvm_cpuid_entry2)))
 		goto out;
 	return 0;
 out:
 	cpuid->nent = vcpu->arch.cpuid_nent;
 	return r;
 }
 static void do_cpuid_1_ent(struct kvm_cpuid_entry2 *entry, u32 function,
 			   u32 index)
 {
 	entry->function = function;
 	entry->index = index;
 	cpuid_count(entry->function, entry->index,
 		    &entry->eax, &entry->ebx, &entry->ecx, &entry->edx);
 	entry->flags = 0;
 }
 #define F(x) bit(X86_FEATURE_##x)
 static void do_cpuid_ent(struct kvm_cpuid_entry2 *entry, u32 function,
 			 u32 index, int *nent, int maxnent)
 {
 	unsigned f_nx = is_efer_nx() ? F(NX) : 0;
 #ifdef CONFIG_X86_64
 	unsigned f_gbpages = (kvm_x86_ops->get_lpage_level() == PT_PDPE_LEVEL)
 				? F(GBPAGES) : 0;
 	unsigned f_lm = F(LM);
 #else
 	unsigned f_gbpages = 0;
 	unsigned f_lm = 0;
 #endif
 	unsigned f_rdtscp = kvm_x86_ops->rdtscp_supported() ? F(RDTSCP) : 0;
 	/* cpuid 1.edx */
 	const u32 kvm_supported_word0_x86_features =
 		F(FPU) | F(VME) | F(DE) | F(PSE) |
 		F(TSC) | F(MSR) | F(PAE) | F(MCE) |
 		F(CX8) | F(APIC) | 0 /* Reserved */ | F(SEP) |
 		F(MTRR) | F(PGE) | F(MCA) | F(CMOV) |
 		F(PAT) | F(PSE36) | 0 /* PSN */ | F(CLFLSH) |
 		0 /* Reserved, DS, ACPI */ | F(MMX) |
 		F(FXSR) | F(XMM) | F(XMM2) | F(SELFSNOOP) |
 		0 /* HTT, TM, Reserved, PBE */;
 	/* cpuid 0x80000001.edx */
 	const u32 kvm_supported_word1_x86_features =
 		F(FPU) | F(VME) | F(DE) | F(PSE) |
 		F(TSC) | F(MSR) | F(PAE) | F(MCE) |
 		F(CX8) | F(APIC) | 0 /* Reserved */ | F(SYSCALL) |
 		F(MTRR) | F(PGE) | F(MCA) | F(CMOV) |
 		F(PAT) | F(PSE36) | 0 /* Reserved */ |
 		f_nx | 0 /* Reserved */ | F(MMXEXT) | F(MMX) |
 		F(FXSR) | F(FXSR_OPT) | f_gbpages | f_rdtscp |
 		0 /* Reserved */ | f_lm | F(3DNOWEXT) | F(3DNOW);
 	/* cpuid 1.ecx */
 	const u32 kvm_supported_word4_x86_features =
 		F(XMM3) | 0 /* Reserved, DTES64, MONITOR */ |
 		0 /* DS-CPL, VMX, SMX, EST */ |
 		0 /* TM2 */ | F(SSSE3) | 0 /* CNXT-ID */ | 0 /* Reserved */ |
 		0 /* Reserved */ | F(CX16) | 0 /* xTPR Update, PDCM */ |
 		0 /* Reserved, DCA */ | F(XMM4_1) |
 		F(XMM4_2) | F(X2APIC) | F(MOVBE) | F(POPCNT) |
 		0 /* Reserved, AES */ | F(XSAVE) | 0 /* OSXSAVE */;
 	/* cpuid 0x80000001.ecx */
 	const u32 kvm_supported_word6_x86_features =
 		F(LAHF_LM) | F(CMP_LEGACY) | F(SVM) | 0 /* ExtApicSpace */ |
 		F(CR8_LEGACY) | F(ABM) | F(SSE4A) | F(MISALIGNSSE) |
 		F(3DNOWPREFETCH) | 0 /* OSVW */ | 0 /* IBS */ | F(SSE5) |
 		0 /* SKINIT */ | 0 /* WDT */;
 	/* all calls to cpuid_count() should be made on the same cpu */
 	get_cpu();
 	do_cpuid_1_ent(entry, function, index);
 	++*nent;
 	switch (function) {
 	case 0:
 		entry->eax = min(entry->eax, (u32)0xd);
 		break;
 	case 1:
 		entry->edx &= kvm_supported_word0_x86_features;
 		entry->ecx &= kvm_supported_word4_x86_features;
 		/* we support x2apic emulation even if host does not support
 		 * it since we emulate x2apic in software */
 		entry->ecx |= F(X2APIC);
 		break;
 	/* function 2 entries are STATEFUL. That is, repeated cpuid commands
 	 * may return different values. This forces us to get_cpu() before
 	 * issuing the first command, and also to emulate this annoying behavior
 	 * in kvm_emulate_cpuid() using KVM_CPUID_FLAG_STATE_READ_NEXT */
 	case 2: {
 		int t, times = entry->eax & 0xff;
 		entry->flags |= KVM_CPUID_FLAG_STATEFUL_FUNC;
 		entry->flags |= KVM_CPUID_FLAG_STATE_READ_NEXT;
 		for (t = 1; t < times && *nent < maxnent; ++t) {
 			do_cpuid_1_ent(&entry[t], function, 0);
 			entry[t].flags |= KVM_CPUID_FLAG_STATEFUL_FUNC;
 			++*nent;
 		}
 		break;
 	}
 	/* function 4 and 0xb have additional index. */
 	case 4: {
 		int i, cache_type;
 		entry->flags |= KVM_CPUID_FLAG_SIGNIFCANT_INDEX;
 		/* read more entries until cache_type is zero */
 		for (i = 1; *nent < maxnent; ++i) {
 			cache_type = entry[i - 1].eax & 0x1f;
 			if (!cache_type)
 				break;
 			do_cpuid_1_ent(&entry[i], function, i);
 			entry[i].flags |=
 			       KVM_CPUID_FLAG_SIGNIFCANT_INDEX;
 			++*nent;
 		}
 		break;
 	}
 	case 0xb: {
 		int i, level_type;
 		entry->flags |= KVM_CPUID_FLAG_SIGNIFCANT_INDEX;
 		/* read more entries until level_type is zero */
 		for (i = 1; *nent < maxnent; ++i) {
 			level_type = entry[i - 1].ecx & 0xff00;
 			if (!level_type)
 				break;
 			do_cpuid_1_ent(&entry[i], function, i);
 			entry[i].flags |=
 			       KVM_CPUID_FLAG_SIGNIFCANT_INDEX;
 			++*nent;
 		}
 		break;
 	}
 	case 0xd: {
 		int i;
 		entry->flags |= KVM_CPUID_FLAG_SIGNIFCANT_INDEX;
 		for (i = 1; *nent < maxnent; ++i) {
 			if (entry[i - 1].eax == 0 && i != 2)
 				break;
 			do_cpuid_1_ent(&entry[i], function, i);
 			entry[i].flags |=
 			       KVM_CPUID_FLAG_SIGNIFCANT_INDEX;
 			++*nent;
 		}
 		break;
 	}
 	case KVM_CPUID_SIGNATURE: {
 		char signature[12] = "KVMKVMKVM\0\0";
 		u32 *sigptr = (u32 *)signature;
 		entry->eax = 0;
 		entry->ebx = sigptr[0];
 		entry->ecx = sigptr[1];
 		entry->edx = sigptr[2];
 		break;
 	}
 	case KVM_CPUID_FEATURES:
 		entry->eax = (1 << KVM_FEATURE_CLOCKSOURCE) |
 			     (1 << KVM_FEATURE_NOP_IO_DELAY) |
 			     (1 << KVM_FEATURE_CLOCKSOURCE2) |
 			     (1 << KVM_FEATURE_CLOCKSOURCE_STABLE_BIT);
 		entry->ebx = 0;
 		entry->ecx = 0;
 		entry->edx = 0;
 		break;
 	case 0x80000000:
 		entry->eax = min(entry->eax, 0x8000001a);
 		break;
 	case 0x80000001:
 		entry->edx &= kvm_supported_word1_x86_features;
 		entry->ecx &= kvm_supported_word6_x86_features;
 		break;
 	}
 	kvm_x86_ops->set_supported_cpuid(function, entry);
 	put_cpu();
 }
 #undef F
 static int kvm_dev_ioctl_get_supported_cpuid(struct kvm_cpuid2 *cpuid,
 				     struct kvm_cpuid_entry2 __user *entries)
 {
 	struct kvm_cpuid_entry2 *cpuid_entries;
 	int limit, nent = 0, r = -E2BIG;
 	u32 func;
 	if (cpuid->nent < 1)
 		goto out;
 	if (cpuid->nent > KVM_MAX_CPUID_ENTRIES)
 		cpuid->nent = KVM_MAX_CPUID_ENTRIES;
 	r = -ENOMEM;
 	cpuid_entries = vmalloc(sizeof(struct kvm_cpuid_entry2) * cpuid->nent);
 	if (!cpuid_entries)
 		goto out;
 	do_cpuid_ent(&cpuid_entries[0], 0, 0, &nent, cpuid->nent);
 	limit = cpuid_entries[0].eax;
 	for (func = 1; func <= limit && nent < cpuid->nent; ++func)
 		do_cpuid_ent(&cpuid_entries[nent], func, 0,
 			     &nent, cpuid->nent);
 	r = -E2BIG;
 	if (nent >= cpuid->nent)
 		goto out_free;
 	do_cpuid_ent(&cpuid_entries[nent], 0x80000000, 0, &nent, cpuid->nent);
 	limit = cpuid_entries[nent - 1].eax;
 	for (func = 0x80000001; func <= limit && nent < cpuid->nent; ++func)
 		do_cpuid_ent(&cpuid_entries[nent], func, 0,
 			     &nent, cpuid->nent);
 	r = -E2BIG;
 	if (nent >= cpuid->nent)
 		goto out_free;
 	do_cpuid_ent(&cpuid_entries[nent], KVM_CPUID_SIGNATURE, 0, &nent,
 		     cpuid->nent);
 	r = -E2BIG;
 	if (nent >= cpuid->nent)
 		goto out_free;
 	do_cpuid_ent(&cpuid_entries[nent], KVM_CPUID_FEATURES, 0, &nent,
 		     cpuid->nent);
 	r = -E2BIG;
 	if (nent >= cpuid->nent)
 		goto out_free;
 	r = -EFAULT;
 	if (copy_to_user(entries, cpuid_entries,
 			 nent * sizeof(struct kvm_cpuid_entry2)))
 		goto out_free;
 	cpuid->nent = nent;
 	r = 0;
 out_free:
 	vfree(cpuid_entries);
 out:
 	return r;
 }
 static int kvm_vcpu_ioctl_get_lapic(struct kvm_vcpu *vcpu,
 				    struct kvm_lapic_state *s)
 {
 	memcpy(s->regs, vcpu->arch.apic->regs, sizeof *s);
 	return 0;
 }
 static int kvm_vcpu_ioctl_set_lapic(struct kvm_vcpu *vcpu,
 				    struct kvm_lapic_state *s)
 {
 	memcpy(vcpu->arch.apic->regs, s->regs, sizeof *s);
 	kvm_apic_post_state_restore(vcpu);
 	update_cr8_intercept(vcpu);
 	return 0;
 }
 static int kvm_vcpu_ioctl_interrupt(struct kvm_vcpu *vcpu,
 				    struct kvm_interrupt *irq)
 {
 	if (irq->irq < 0 || irq->irq >= 256)
 		return -EINVAL;
 	if (irqchip_in_kernel(vcpu->kvm))
 		return -ENXIO;
 	kvm_queue_interrupt(vcpu, irq->irq, false);
 	return 0;
 }
 static int kvm_vcpu_ioctl_nmi(struct kvm_vcpu *vcpu)
 {
 	kvm_inject_nmi(vcpu);
 	return 0;
 }
 static int vcpu_ioctl_tpr_access_reporting(struct kvm_vcpu *vcpu,
 					   struct kvm_tpr_access_ctl *tac)
 {
 	if (tac->flags)
 		return -EINVAL;
 	vcpu->arch.tpr_access_reporting = !!tac->enabled;
 	return 0;
 }
 static int kvm_vcpu_ioctl_x86_setup_mce(struct kvm_vcpu *vcpu,
 					u64 mcg_cap)
 {
 	int r;
 	unsigned bank_num = mcg_cap & 0xff, bank;
 	r = -EINVAL;
 	if (!bank_num || bank_num >= KVM_MAX_MCE_BANKS)
 		goto out;
 	if (mcg_cap & ~(KVM_MCE_CAP_SUPPORTED | 0xff | 0xff0000))
 		goto out;
 	r = 0;
 	vcpu->arch.mcg_cap = mcg_cap;
 	/* Init IA32_MCG_CTL to all 1s */
 	if (mcg_cap & MCG_CTL_P)
 		vcpu->arch.mcg_ctl = ~(u64)0;
 	/* Init IA32_MCi_CTL to all 1s */
 	for (bank = 0; bank < bank_num; bank++)
 		vcpu->arch.mce_banks[bank*4] = ~(u64)0;
 out:
 	return r;
 }
 static int kvm_vcpu_ioctl_x86_set_mce(struct kvm_vcpu *vcpu,
 				      struct kvm_x86_mce *mce)
 {
 	u64 mcg_cap = vcpu->arch.mcg_cap;
 	unsigned bank_num = mcg_cap & 0xff;
 	u64 *banks = vcpu->arch.mce_banks;
 	if (mce->bank >= bank_num || !(mce->status & MCI_STATUS_VAL))
 		return -EINVAL;
 	/*
 	 * if IA32_MCG_CTL is not all 1s, the uncorrected error
 	 * reporting is disabled
 	 */
 	if ((mce->status & MCI_STATUS_UC) && (mcg_cap & MCG_CTL_P) &&
 	    vcpu->arch.mcg_ctl != ~(u64)0)
 		return 0;
 	banks += 4 * mce->bank;
 	/*
 	 * if IA32_MCi_CTL is not all 1s, the uncorrected error
 	 * reporting is disabled for the bank
 	 */
 	if ((mce->status & MCI_STATUS_UC) && banks[0] != ~(u64)0)
 		return 0;
 	if (mce->status & MCI_STATUS_UC) {
 		if ((vcpu->arch.mcg_status & MCG_STATUS_MCIP) ||
 		    !kvm_read_cr4_bits(vcpu, X86_CR4_MCE)) {
 			printk(KERN_DEBUG "kvm: set_mce: "
 			       "injects mce exception while "
 			       "previous one is in progress!\n");
 			set_bit(KVM_REQ_TRIPLE_FAULT, &vcpu->requests);
 			return 0;
 		}
 		if (banks[1] & MCI_STATUS_VAL)
 			mce->status |= MCI_STATUS_OVER;
 		banks[2] = mce->addr;
 		banks[3] = mce->misc;
 		vcpu->arch.mcg_status = mce->mcg_status;
 		banks[1] = mce->status;
 		kvm_queue_exception(vcpu, MC_VECTOR);
 	} else if (!(banks[1] & MCI_STATUS_VAL)
 		   || !(banks[1] & MCI_STATUS_UC)) {
 		if (banks[1] & MCI_STATUS_VAL)
 			mce->status |= MCI_STATUS_OVER;
 		banks[2] = mce->addr;
 		banks[3] = mce->misc;
 		banks[1] = mce->status;
 	} else
 		banks[1] |= MCI_STATUS_OVER;
 	return 0;
 }
 static void kvm_vcpu_ioctl_x86_get_vcpu_events(struct kvm_vcpu *vcpu,
 					       struct kvm_vcpu_events *events)
 {
 	events->exception.injected =
 		vcpu->arch.exception.pending &&
 		!kvm_exception_is_soft(vcpu->arch.exception.nr);
 	events->exception.nr = vcpu->arch.exception.nr;
 	events->exception.has_error_code = vcpu->arch.exception.has_error_code;
 	events->exception.error_code = vcpu->arch.exception.error_code;
 	events->interrupt.injected =
 		vcpu->arch.interrupt.pending && !vcpu->arch.interrupt.soft;
 	events->interrupt.nr = vcpu->arch.interrupt.nr;
 	events->interrupt.soft = 0;
 	events->interrupt.shadow =
 		kvm_x86_ops->get_interrupt_shadow(vcpu,
 			KVM_X86_SHADOW_INT_MOV_SS | KVM_X86_SHADOW_INT_STI);
 	events->nmi.injected = vcpu->arch.nmi_injected;
 	events->nmi.pending = vcpu->arch.nmi_pending;
 	events->nmi.masked = kvm_x86_ops->get_nmi_mask(vcpu);
 	events->sipi_vector = vcpu->arch.sipi_vector;
 	events->flags = (KVM_VCPUEVENT_VALID_NMI_PENDING
 			 | KVM_VCPUEVENT_VALID_SIPI_VECTOR
 			 | KVM_VCPUEVENT_VALID_SHADOW);
 }
 static int kvm_vcpu_ioctl_x86_set_vcpu_events(struct kvm_vcpu *vcpu,
 					      struct kvm_vcpu_events *events)
 {
 	if (events->flags & ~(KVM_VCPUEVENT_VALID_NMI_PENDING
 			      | KVM_VCPUEVENT_VALID_SIPI_VECTOR
 			      | KVM_VCPUEVENT_VALID_SHADOW))
 		return -EINVAL;
 	vcpu->arch.exception.pending = events->exception.injected;
 	vcpu->arch.exception.nr = events->exception.nr;
 	vcpu->arch.exception.has_error_code = events->exception.has_error_code;
 	vcpu->arch.exception.error_code = events->exception.error_code;
 	vcpu->arch.interrupt.pending = events->interrupt.injected;
 	vcpu->arch.interrupt.nr = events->interrupt.nr;
 	vcpu->arch.interrupt.soft = events->interrupt.soft;
 	if (vcpu->arch.interrupt.pending && irqchip_in_kernel(vcpu->kvm))
 		kvm_pic_clear_isr_ack(vcpu->kvm);
 	if (events->flags & KVM_VCPUEVENT_VALID_SHADOW)
 		kvm_x86_ops->set_interrupt_shadow(vcpu,
 						  events->interrupt.shadow);
 	vcpu->arch.nmi_injected = events->nmi.injected;
 	if (events->flags & KVM_VCPUEVENT_VALID_NMI_PENDING)
 		vcpu->arch.nmi_pending = events->nmi.pending;
 	kvm_x86_ops->set_nmi_mask(vcpu, events->nmi.masked);
 	if (events->flags & KVM_VCPUEVENT_VALID_SIPI_VECTOR)
 		vcpu->arch.sipi_vector = events->sipi_vector;
 	return 0;
 }
 static void kvm_vcpu_ioctl_x86_get_debugregs(struct kvm_vcpu *vcpu,
 					     struct kvm_debugregs *dbgregs)
 {
 	memcpy(dbgregs->db, vcpu->arch.db, sizeof(vcpu->arch.db));
 	dbgregs->dr6 = vcpu->arch.dr6;
 	dbgregs->dr7 = vcpu->arch.dr7;
 	dbgregs->flags = 0;
 }
 static int kvm_vcpu_ioctl_x86_set_debugregs(struct kvm_vcpu *vcpu,
 					    struct kvm_debugregs *dbgregs)
 {
 	if (dbgregs->flags)
 		return -EINVAL;
 	memcpy(vcpu->arch.db, dbgregs->db, sizeof(vcpu->arch.db));
 	vcpu->arch.dr6 = dbgregs->dr6;
 	vcpu->arch.dr7 = dbgregs->dr7;
 	return 0;
 }
 static void kvm_vcpu_ioctl_x86_get_xsave(struct kvm_vcpu *vcpu,
 					 struct kvm_xsave *guest_xsave)
 {
 	if (cpu_has_xsave)
 		memcpy(guest_xsave->region,
 			&vcpu->arch.guest_fpu.state->xsave,
 			sizeof(struct xsave_struct));
 	else {
 		memcpy(guest_xsave->region,
 			&vcpu->arch.guest_fpu.state->fxsave,
 			sizeof(struct i387_fxsave_struct));
 		*(u64 *)&guest_xsave->region[XSAVE_HDR_OFFSET / sizeof(u32)] =
 			XSTATE_FPSSE;
 	}
 }
 static int kvm_vcpu_ioctl_x86_set_xsave(struct kvm_vcpu *vcpu,
 					struct kvm_xsave *guest_xsave)
 {
 	u64 xstate_bv =
 		*(u64 *)&guest_xsave->region[XSAVE_HDR_OFFSET / sizeof(u32)];
 	if (cpu_has_xsave)
 		memcpy(&vcpu->arch.guest_fpu.state->xsave,
 			guest_xsave->region, sizeof(struct xsave_struct));
 	else {
 		if (xstate_bv & ~XSTATE_FPSSE)
 			return -EINVAL;
 		memcpy(&vcpu->arch.guest_fpu.state->fxsave,
 			guest_xsave->region, sizeof(struct i387_fxsave_struct));
 	}
 	return 0;
 }
 static void kvm_vcpu_ioctl_x86_get_xcrs(struct kvm_vcpu *vcpu,
 					struct kvm_xcrs *guest_xcrs)
 {
 	if (!cpu_has_xsave) {
 		guest_xcrs->nr_xcrs = 0;
 		return;
 	}
 	guest_xcrs->nr_xcrs = 1;
 	guest_xcrs->flags = 0;
 	guest_xcrs->xcrs[0].xcr = XCR_XFEATURE_ENABLED_MASK;
 	guest_xcrs->xcrs[0].value = vcpu->arch.xcr0;
 }
 static int kvm_vcpu_ioctl_x86_set_xcrs(struct kvm_vcpu *vcpu,
 				       struct kvm_xcrs *guest_xcrs)
 {
 	int i, r = 0;
 	if (!cpu_has_xsave)
 		return -EINVAL;
 	if (guest_xcrs->nr_xcrs > KVM_MAX_XCRS || guest_xcrs->flags)
 		return -EINVAL;
 	for (i = 0; i < guest_xcrs->nr_xcrs; i++)
 		/* Only support XCR0 currently */
 		if (guest_xcrs->xcrs[0].xcr == XCR_XFEATURE_ENABLED_MASK) {
 			r = __kvm_set_xcr(vcpu, XCR_XFEATURE_ENABLED_MASK,
 				guest_xcrs->xcrs[0].value);
 			break;
 		}
 	if (r)
 		r = -EINVAL;
 	return r;
 }
 long kvm_arch_vcpu_ioctl(struct file *filp,
 			 unsigned int ioctl, unsigned long arg)
 {
 	struct kvm_vcpu *vcpu = filp->private_data;
 	void __user *argp = (void __user *)arg;
 	int r;
 	union {
 		struct kvm_lapic_state *lapic;
 		struct kvm_xsave *xsave;
 		struct kvm_xcrs *xcrs;
 		void *buffer;
 	} u;
 	u.buffer = NULL;
 	switch (ioctl) {
 	case KVM_GET_LAPIC: {
 		r = -EINVAL;
 		if (!vcpu->arch.apic)
 			goto out;
 		u.lapic = kzalloc(sizeof(struct kvm_lapic_state), GFP_KERNEL);
 		r = -ENOMEM;
 		if (!u.lapic)
 			goto out;
 		r = kvm_vcpu_ioctl_get_lapic(vcpu, u.lapic);
 		if (r)
 			goto out;
 		r = -EFAULT;
 		if (copy_to_user(argp, u.lapic, sizeof(struct kvm_lapic_state)))
 			goto out;
 		r = 0;
 		break;
 	}
 	case KVM_SET_LAPIC: {
 		r = -EINVAL;
 		if (!vcpu->arch.apic)
 			goto out;
 		u.lapic = kmalloc(sizeof(struct kvm_lapic_state), GFP_KERNEL);
 		r = -ENOMEM;
 		if (!u.lapic)
 			goto out;
 		r = -EFAULT;
 		if (copy_from_user(u.lapic, argp, sizeof(struct kvm_lapic_state)))
 			goto out;
 		r = kvm_vcpu_ioctl_set_lapic(vcpu, u.lapic);
 		if (r)
 			goto out;
 		r = 0;
 		break;
 	}
 	case KVM_INTERRUPT: {
 		struct kvm_interrupt irq;
 		r = -EFAULT;
 		if (copy_from_user(&irq, argp, sizeof irq))
 			goto out;
 		r = kvm_vcpu_ioctl_interrupt(vcpu, &irq);
 		if (r)
 			goto out;
 		r = 0;
 		break;
 	}
 	case KVM_NMI: {
 		r = kvm_vcpu_ioctl_nmi(vcpu);
 		if (r)
 			goto out;
 		r = 0;
 		break;
 	}
 	case KVM_SET_CPUID: {
 		struct kvm_cpuid __user *cpuid_arg = argp;
 		struct kvm_cpuid cpuid;
 		r = -EFAULT;
 		if (copy_from_user(&cpuid, cpuid_arg, sizeof cpuid))
 			goto out;
 		r = kvm_vcpu_ioctl_set_cpuid(vcpu, &cpuid, cpuid_arg->entries);
 		if (r)
 			goto out;
 		break;
 	}
 	case KVM_SET_CPUID2: {
 		struct kvm_cpuid2 __user *cpuid_arg = argp;
 		struct kvm_cpuid2 cpuid;
 		r = -EFAULT;
 		if (copy_from_user(&cpuid, cpuid_arg, sizeof cpuid))
 			goto out;
 		r = kvm_vcpu_ioctl_set_cpuid2(vcpu, &cpuid,
 					      cpuid_arg->entries);
 		if (r)
 			goto out;
 		break;
 	}
 	case KVM_GET_CPUID2: {
 		struct kvm_cpuid2 __user *cpuid_arg = argp;
 		struct kvm_cpuid2 cpuid;
 		r = -EFAULT;
 		if (copy_from_user(&cpuid, cpuid_arg, sizeof cpuid))
 			goto out;
 		r = kvm_vcpu_ioctl_get_cpuid2(vcpu, &cpuid,
 					      cpuid_arg->entries);
 		if (r)
 			goto out;
 		r = -EFAULT;
 		if (copy_to_user(cpuid_arg, &cpuid, sizeof cpuid))
 			goto out;
 		r = 0;
 		break;
 	}
 	case KVM_GET_MSRS:
 		r = msr_io(vcpu, argp, kvm_get_msr, 1);
 		break;
 	case KVM_SET_MSRS:
 		r = msr_io(vcpu, argp, do_set_msr, 0);
 		break;
 	case KVM_TPR_ACCESS_REPORTING: {
 		struct kvm_tpr_access_ctl tac;
 		r = -EFAULT;
 		if (copy_from_user(&tac, argp, sizeof tac))
 			goto out;
 		r = vcpu_ioctl_tpr_access_reporting(vcpu, &tac);
 		if (r)
 			goto out;
 		r = -EFAULT;
 		if (copy_to_user(argp, &tac, sizeof tac))
 			goto out;
 		r = 0;
 		break;
 	};
 	case KVM_SET_VAPIC_ADDR: {
 		struct kvm_vapic_addr va;
 		r = -EINVAL;
 		if (!irqchip_in_kernel(vcpu->kvm))
 			goto out;
 		r = -EFAULT;
 		if (copy_from_user(&va, argp, sizeof va))
 			goto out;
 		r = 0;
 		kvm_lapic_set_vapic_addr(vcpu, va.vapic_addr);
 		break;
 	}
 	case KVM_X86_SETUP_MCE: {
 		u64 mcg_cap;
 		r = -EFAULT;
 		if (copy_from_user(&mcg_cap, argp, sizeof mcg_cap))
 			goto out;
 		r = kvm_vcpu_ioctl_x86_setup_mce(vcpu, mcg_cap);
 		break;
 	}
 	case KVM_X86_SET_MCE: {
 		struct kvm_x86_mce mce;
 		r = -EFAULT;
 		if (copy_from_user(&mce, argp, sizeof mce))
 			goto out;
 		r = kvm_vcpu_ioctl_x86_set_mce(vcpu, &mce);
 		break;
 	}
 	case KVM_GET_VCPU_EVENTS: {
 		struct kvm_vcpu_events events;
 		kvm_vcpu_ioctl_x86_get_vcpu_events(vcpu, &events);
 		r = -EFAULT;
 		if (copy_to_user(argp, &events, sizeof(struct kvm_vcpu_events)))
 			break;
 		r = 0;
 		break;
 	}
 	case KVM_SET_VCPU_EVENTS: {
 		struct kvm_vcpu_events events;
 		r = -EFAULT;
 		if (copy_from_user(&events, argp, sizeof(struct kvm_vcpu_events)))
 			break;
 		r = kvm_vcpu_ioctl_x86_set_vcpu_events(vcpu, &events);
 		break;
 	}
 	case KVM_GET_DEBUGREGS: {
 		struct kvm_debugregs dbgregs;
 		kvm_vcpu_ioctl_x86_get_debugregs(vcpu, &dbgregs);
 		r = -EFAULT;
 		if (copy_to_user(argp, &dbgregs,
 				 sizeof(struct kvm_debugregs)))
 			break;
 		r = 0;
 		break;
 	}
 	case KVM_SET_DEBUGREGS: {
 		struct kvm_debugregs dbgregs;
 		r = -EFAULT;
 		if (copy_from_user(&dbgregs, argp,
 				   sizeof(struct kvm_debugregs)))
 			break;
 		r = kvm_vcpu_ioctl_x86_set_debugregs(vcpu, &dbgregs);
 		break;
 	}
 	case KVM_GET_XSAVE: {
 		u.xsave = kzalloc(sizeof(struct kvm_xsave), GFP_KERNEL);
 		r = -ENOMEM;
 		if (!u.xsave)
 			break;
 		kvm_vcpu_ioctl_x86_get_xsave(vcpu, u.xsave);
 		r = -EFAULT;
 		if (copy_to_user(argp, u.xsave, sizeof(struct kvm_xsave)))
 			break;
 		r = 0;
 		break;
 	}
 	case KVM_SET_XSAVE: {
 		u.xsave = kzalloc(sizeof(struct kvm_xsave), GFP_KERNEL);
 		r = -ENOMEM;
 		if (!u.xsave)
 			break;
 		r = -EFAULT;
 		if (copy_from_user(u.xsave, argp, sizeof(struct kvm_xsave)))
 			break;
 		r = kvm_vcpu_ioctl_x86_set_xsave(vcpu, u.xsave);
 		break;
 	}
 	case KVM_GET_XCRS: {
 		u.xcrs = kzalloc(sizeof(struct kvm_xcrs), GFP_KERNEL);
 		r = -ENOMEM;
 		if (!u.xcrs)
 			break;
 		kvm_vcpu_ioctl_x86_get_xcrs(vcpu, u.xcrs);
 		r = -EFAULT;
 		if (copy_to_user(argp, u.xcrs,
 				 sizeof(struct kvm_xcrs)))
 			break;
 		r = 0;
 		break;
 	}
 	case KVM_SET_XCRS: {
 		u.xcrs = kzalloc(sizeof(struct kvm_xcrs), GFP_KERNEL);
 		r = -ENOMEM;
 		if (!u.xcrs)
 			break;
 		r = -EFAULT;
 		if (copy_from_user(u.xcrs, argp,
 				   sizeof(struct kvm_xcrs)))
 			break;
 		r = kvm_vcpu_ioctl_x86_set_xcrs(vcpu, u.xcrs);
 		break;
 	}
 	default:
 		r = -EINVAL;
 	}
 out:
 	kfree(u.buffer);
 	return r;
 }
 static int kvm_vm_ioctl_set_tss_addr(struct kvm *kvm, unsigned long addr)
 {
 	int ret;
 	if (addr > (unsigned int)(-3 * PAGE_SIZE))
 		return -1;
 	ret = kvm_x86_ops->set_tss_addr(kvm, addr);
 	return ret;
 }
 static int kvm_vm_ioctl_set_identity_map_addr(struct kvm *kvm,
 					      u64 ident_addr)
 {
 	kvm->arch.ept_identity_map_addr = ident_addr;
 	return 0;
 }
 static int kvm_vm_ioctl_set_nr_mmu_pages(struct kvm *kvm,
 					  u32 kvm_nr_mmu_pages)
 {
 	if (kvm_nr_mmu_pages < KVM_MIN_ALLOC_MMU_PAGES)
 		return -EINVAL;
 	mutex_lock(&kvm->slots_lock);
 	spin_lock(&kvm->mmu_lock);
 	kvm_mmu_change_mmu_pages(kvm, kvm_nr_mmu_pages);
 	kvm->arch.n_requested_mmu_pages = kvm_nr_mmu_pages;
 	spin_unlock(&kvm->mmu_lock);
 	mutex_unlock(&kvm->slots_lock);
 	return 0;
 }
 static int kvm_vm_ioctl_get_nr_mmu_pages(struct kvm *kvm)
 {
 	return kvm->arch.n_alloc_mmu_pages;
 }
-gfn_t unalias_gfn_instantiation(struct kvm *kvm, gfn_t gfn)
-{
-	int i;
-	struct kvm_mem_alias *alias;
-	struct kvm_mem_aliases *aliases;
-	aliases = kvm_aliases(kvm);
-	for (i = 0; i < aliases->naliases; ++i) {
-		alias = &aliases->aliases[i];
-		if (alias->flags & KVM_ALIAS_INVALID)
-			continue;
-		if (gfn >= alias->base_gfn
-		    && gfn < alias->base_gfn + alias->npages)
-			return alias->target_gfn + gfn - alias->base_gfn;
-	}
-	return gfn;
-}
-gfn_t unalias_gfn(struct kvm *kvm, gfn_t gfn)
-{
-	int i;
-	struct kvm_mem_alias *alias;
-	struct kvm_mem_aliases *aliases;
-	aliases = kvm_aliases(kvm);
-	for (i = 0; i < aliases->naliases; ++i) {
-		alias = &aliases->aliases[i];
-		if (gfn >= alias->base_gfn
-		    && gfn < alias->base_gfn + alias->npages)
-			return alias->target_gfn + gfn - alias->base_gfn;
-	}
-	return gfn;
-}
-/*
- * Set a new alias region.  Aliases map a portion of physical memory into
- * another portion.  This is useful for memory windows, for example the PC
- * VGA region.
- */
-static int kvm_vm_ioctl_set_memory_alias(struct kvm *kvm,
-					 struct kvm_memory_alias *alias)
-{
-	int r, n;
-	struct kvm_mem_alias *p;
-	struct kvm_mem_aliases *aliases, *old_aliases;
-	r = -EINVAL;
-	/* General sanity checks */
-	if (alias->memory_size & (PAGE_SIZE - 1))
-		goto out;
-	if (alias->guest_phys_addr & (PAGE_SIZE - 1))
-		goto out;
-	if (alias->slot >= KVM_ALIAS_SLOTS)
-		goto out;
-	if (alias->guest_phys_addr + alias->memory_size
-	    < alias->guest_phys_addr)
-		goto out;
-	if (alias->target_phys_addr + alias->memory_size
-	    < alias->target_phys_addr)
-		goto out;
-	r = -ENOMEM;
-	aliases = kzalloc(sizeof(struct kvm_mem_aliases), GFP_KERNEL);
-	if (!aliases)
-		goto out;
-	mutex_lock(&kvm->slots_lock);
-	/* invalidate any gfn reference in case of deletion/shrinking */
-	memcpy(aliases, kvm->arch.aliases, sizeof(struct kvm_mem_aliases));
-	aliases->aliases[alias->slot].flags |= KVM_ALIAS_INVALID;
-	old_aliases = kvm->arch.aliases;
-	rcu_assign_pointer(kvm->arch.aliases, aliases);
-	synchronize_srcu_expedited(&kvm->srcu);
-	kvm_mmu_zap_all(kvm);
-	kfree(old_aliases);
-	r = -ENOMEM;
-	aliases = kzalloc(sizeof(struct kvm_mem_aliases), GFP_KERNEL);
-	if (!aliases)
-		goto out_unlock;
-	memcpy(aliases, kvm->arch.aliases, sizeof(struct kvm_mem_aliases));
-	p = &aliases->aliases[alias->slot];
-	p->base_gfn = alias->guest_phys_addr >> PAGE_SHIFT;
-	p->npages = alias->memory_size >> PAGE_SHIFT;
-	p->target_gfn = alias->target_phys_addr >> PAGE_SHIFT;
-	p->flags &= ~(KVM_ALIAS_INVALID);
-	for (n = KVM_ALIAS_SLOTS; n > 0; --n)
-		if (aliases->aliases[n - 1].npages)
-			break;
-	aliases->naliases = n;
-	old_aliases = kvm->arch.aliases;
-	rcu_assign_pointer(kvm->arch.aliases, aliases);
-	synchronize_srcu_expedited(&kvm->srcu);
-	kfree(old_aliases);
-	r = 0;
-out_unlock:
-	mutex_unlock(&kvm->slots_lock);
-out:
-	return r;
-}
 static int kvm_vm_ioctl_get_irqchip(struct kvm *kvm, struct kvm_irqchip *chip)
 {
 	int r;
 	r = 0;
 	switch (chip->chip_id) {
 	case KVM_IRQCHIP_PIC_MASTER:
 		memcpy(&chip->chip.pic,
 			&pic_irqchip(kvm)->pics[0],
 			sizeof(struct kvm_pic_state));
 		break;
 	case KVM_IRQCHIP_PIC_SLAVE:
 		memcpy(&chip->chip.pic,
 			&pic_irqchip(kvm)->pics[1],
 			sizeof(struct kvm_pic_state));
 		break;
 	case KVM_IRQCHIP_IOAPIC:
 		r = kvm_get_ioapic(kvm, &chip->chip.ioapic);
 		break;
 	default:
 		r = -EINVAL;
 		break;
 	}
 	return r;
 }
 static int kvm_vm_ioctl_set_irqchip(struct kvm *kvm, struct kvm_irqchip *chip)
 {
 	int r;
 	r = 0;
 	switch (chip->chip_id) {
 	case KVM_IRQCHIP_PIC_MASTER:
 		raw_spin_lock(&pic_irqchip(kvm)->lock);
 		memcpy(&pic_irqchip(kvm)->pics[0],
 			&chip->chip.pic,
 			sizeof(struct kvm_pic_state));
 		raw_spin_unlock(&pic_irqchip(kvm)->lock);
 		break;
 	case KVM_IRQCHIP_PIC_SLAVE:
 		raw_spin_lock(&pic_irqchip(kvm)->lock);
 		memcpy(&pic_irqchip(kvm)->pics[1],
 			&chip->chip.pic,
 			sizeof(struct kvm_pic_state));
 		raw_spin_unlock(&pic_irqchip(kvm)->lock);
 		break;
 	case KVM_IRQCHIP_IOAPIC:
 		r = kvm_set_ioapic(kvm, &chip->chip.ioapic);
 		break;
 	default:
 		r = -EINVAL;
 		break;
 	}
 	kvm_pic_update_irq(pic_irqchip(kvm));
 	return r;
 }
 static int kvm_vm_ioctl_get_pit(struct kvm *kvm, struct kvm_pit_state *ps)
 {
 	int r = 0;
 	mutex_lock(&kvm->arch.vpit->pit_state.lock);
 	memcpy(ps, &kvm->arch.vpit->pit_state, sizeof(struct kvm_pit_state));
 	mutex_unlock(&kvm->arch.vpit->pit_state.lock);
 	return r;
 }
 static int kvm_vm_ioctl_set_pit(struct kvm *kvm, struct kvm_pit_state *ps)
 {
 	int r = 0;
 	mutex_lock(&kvm->arch.vpit->pit_state.lock);
 	memcpy(&kvm->arch.vpit->pit_state, ps, sizeof(struct kvm_pit_state));
 	kvm_pit_load_count(kvm, 0, ps->channels[0].count, 0);
 	mutex_unlock(&kvm->arch.vpit->pit_state.lock);
 	return r;
 }
 static int kvm_vm_ioctl_get_pit2(struct kvm *kvm, struct kvm_pit_state2 *ps)
 {
 	int r = 0;
 	mutex_lock(&kvm->arch.vpit->pit_state.lock);
 	memcpy(ps->channels, &kvm->arch.vpit->pit_state.channels,
 		sizeof(ps->channels));
 	ps->flags = kvm->arch.vpit->pit_state.flags;
 	mutex_unlock(&kvm->arch.vpit->pit_state.lock);
 	return r;
 }
 static int kvm_vm_ioctl_set_pit2(struct kvm *kvm, struct kvm_pit_state2 *ps)
 {
 	int r = 0, start = 0;
 	u32 prev_legacy, cur_legacy;
 	mutex_lock(&kvm->arch.vpit->pit_state.lock);
 	prev_legacy = kvm->arch.vpit->pit_state.flags & KVM_PIT_FLAGS_HPET_LEGACY;
 	cur_legacy = ps->flags & KVM_PIT_FLAGS_HPET_LEGACY;
 	if (!prev_legacy && cur_legacy)
 		start = 1;
 	memcpy(&kvm->arch.vpit->pit_state.channels, &ps->channels,
 	       sizeof(kvm->arch.vpit->pit_state.channels));
 	kvm->arch.vpit->pit_state.flags = ps->flags;
 	kvm_pit_load_count(kvm, 0, kvm->arch.vpit->pit_state.channels[0].count, start);
 	mutex_unlock(&kvm->arch.vpit->pit_state.lock);
 	return r;
 }
 static int kvm_vm_ioctl_reinject(struct kvm *kvm,
 				 struct kvm_reinject_control *control)
 {
 	if (!kvm->arch.vpit)
 		return -ENXIO;
 	mutex_lock(&kvm->arch.vpit->pit_state.lock);
 	kvm->arch.vpit->pit_state.pit_timer.reinject = control->pit_reinject;
 	mutex_unlock(&kvm->arch.vpit->pit_state.lock);
 	return 0;
 }
 /*
  * Get (and clear) the dirty memory log for a memory slot.
  */
 int kvm_vm_ioctl_get_dirty_log(struct kvm *kvm,
 				      struct kvm_dirty_log *log)
 {
 	int r, i;
 	struct kvm_memory_slot *memslot;
 	unsigned long n;
 	unsigned long is_dirty = 0;
 	mutex_lock(&kvm->slots_lock);
 	r = -EINVAL;
 	if (log->slot >= KVM_MEMORY_SLOTS)
 		goto out;
 	memslot = &kvm->memslots->memslots[log->slot];
 	r = -ENOENT;
 	if (!memslot->dirty_bitmap)
 		goto out;
 	n = kvm_dirty_bitmap_bytes(memslot);
 	for (i = 0; !is_dirty && i < n/sizeof(long); i++)
 		is_dirty = memslot->dirty_bitmap[i];
 	/* If nothing is dirty, don't bother messing with page tables. */
 	if (is_dirty) {
 		struct kvm_memslots *slots, *old_slots;
 		unsigned long *dirty_bitmap;
 		spin_lock(&kvm->mmu_lock);
 		kvm_mmu_slot_remove_write_access(kvm, log->slot);
 		spin_unlock(&kvm->mmu_lock);
 		r = -ENOMEM;
 		dirty_bitmap = vmalloc(n);
 		if (!dirty_bitmap)
 			goto out;
 		memset(dirty_bitmap, 0, n);
 		r = -ENOMEM;
 		slots = kzalloc(sizeof(struct kvm_memslots), GFP_KERNEL);
 		if (!slots) {
 			vfree(dirty_bitmap);
 			goto out;
 		}
 		memcpy(slots, kvm->memslots, sizeof(struct kvm_memslots));
 		slots->memslots[log->slot].dirty_bitmap = dirty_bitmap;
 		old_slots = kvm->memslots;
 		rcu_assign_pointer(kvm->memslots, slots);
 		synchronize_srcu_expedited(&kvm->srcu);
 		dirty_bitmap = old_slots->memslots[log->slot].dirty_bitmap;
 		kfree(old_slots);
 		r = -EFAULT;
 		if (copy_to_user(log->dirty_bitmap, dirty_bitmap, n)) {
 			vfree(dirty_bitmap);
 			goto out;
 		}
 		vfree(dirty_bitmap);
 	} else {
 		r = -EFAULT;
 		if (clear_user(log->dirty_bitmap, n))
 			goto out;
 	}
 	r = 0;
 out:
 	mutex_unlock(&kvm->slots_lock);
 	return r;
 }
 long kvm_arch_vm_ioctl(struct file *filp,
 		       unsigned int ioctl, unsigned long arg)
 {
 	struct kvm *kvm = filp->private_data;
 	void __user *argp = (void __user *)arg;
 	int r = -ENOTTY;
 	/*
 	 * This union makes it completely explicit to gcc-3.x
 	 * that these two variables' stack usage should be
 	 * combined, not added together.
 	 */
 	union {
 		struct kvm_pit_state ps;
 		struct kvm_pit_state2 ps2;
-		struct kvm_memory_alias alias;
 		struct kvm_pit_config pit_config;
 	} u;
 	switch (ioctl) {
 	case KVM_SET_TSS_ADDR:
 		r = kvm_vm_ioctl_set_tss_addr(kvm, arg);
 		if (r < 0)
 			goto out;
 		break;
 	case KVM_SET_IDENTITY_MAP_ADDR: {
 		u64 ident_addr;
 		r = -EFAULT;
 		if (copy_from_user(&ident_addr, argp, sizeof ident_addr))
 			goto out;
 		r = kvm_vm_ioctl_set_identity_map_addr(kvm, ident_addr);
 		if (r < 0)
 			goto out;
 		break;
 	}
 	case KVM_SET_MEMORY_REGION: {
 		struct kvm_memory_region kvm_mem;
 		struct kvm_userspace_memory_region kvm_userspace_mem;
 		r = -EFAULT;
 		if (copy_from_user(&kvm_mem, argp, sizeof kvm_mem))
 			goto out;
 		kvm_userspace_mem.slot = kvm_mem.slot;
 		kvm_userspace_mem.flags = kvm_mem.flags;
 		kvm_userspace_mem.guest_phys_addr = kvm_mem.guest_phys_addr;
 		kvm_userspace_mem.memory_size = kvm_mem.memory_size;
 		r = kvm_vm_ioctl_set_memory_region(kvm, &kvm_userspace_mem, 0);
 		if (r)
 			goto out;
 		break;
 	}
 	case KVM_SET_NR_MMU_PAGES:
 		r = kvm_vm_ioctl_set_nr_mmu_pages(kvm, arg);
 		if (r)
 			goto out;
 		break;
 	case KVM_GET_NR_MMU_PAGES:
 		r = kvm_vm_ioctl_get_nr_mmu_pages(kvm);
 		break;
-	case KVM_SET_MEMORY_ALIAS:
-		r = -EFAULT;
-		if (copy_from_user(&u.alias, argp, sizeof(struct kvm_memory_alias)))
-			goto out;
-		r = kvm_vm_ioctl_set_memory_alias(kvm, &u.alias);
-		if (r)
-			goto out;
-		break;
 	case KVM_CREATE_IRQCHIP: {
 		struct kvm_pic *vpic;
 		mutex_lock(&kvm->lock);
 		r = -EEXIST;
 		if (kvm->arch.vpic)
 			goto create_irqchip_unlock;
 		r = -ENOMEM;
 		vpic = kvm_create_pic(kvm);
 		if (vpic) {
 			r = kvm_ioapic_init(kvm);
 			if (r) {
 				kvm_io_bus_unregister_dev(kvm, KVM_PIO_BUS,
 							  &vpic->dev);
 				kfree(vpic);
 				goto create_irqchip_unlock;
 			}
 		} else
 			goto create_irqchip_unlock;
 		smp_wmb();
 		kvm->arch.vpic = vpic;
 		smp_wmb();
 		r = kvm_setup_default_irq_routing(kvm);
 		if (r) {
 			mutex_lock(&kvm->irq_lock);
 			kvm_ioapic_destroy(kvm);
 			kvm_destroy_pic(kvm);
 			mutex_unlock(&kvm->irq_lock);
 		}
 	create_irqchip_unlock:
 		mutex_unlock(&kvm->lock);
 		break;
 	}
 	case KVM_CREATE_PIT:
 		u.pit_config.flags = KVM_PIT_SPEAKER_DUMMY;
 		goto create_pit;
 	case KVM_CREATE_PIT2:
 		r = -EFAULT;
 		if (copy_from_user(&u.pit_config, argp,
 				   sizeof(struct kvm_pit_config)))
 			goto out;
 	create_pit:
 		mutex_lock(&kvm->slots_lock);
 		r = -EEXIST;
 		if (kvm->arch.vpit)
 			goto create_pit_unlock;
 		r = -ENOMEM;
 		kvm->arch.vpit = kvm_create_pit(kvm, u.pit_config.flags);
 		if (kvm->arch.vpit)
 			r = 0;
 	create_pit_unlock:
 		mutex_unlock(&kvm->slots_lock);
 		break;
 	case KVM_IRQ_LINE_STATUS:
 	case KVM_IRQ_LINE: {
 		struct kvm_irq_level irq_event;
 		r = -EFAULT;
 		if (copy_from_user(&irq_event, argp, sizeof irq_event))
 			goto out;
 		r = -ENXIO;
 		if (irqchip_in_kernel(kvm)) {
 			__s32 status;
 			status = kvm_set_irq(kvm, KVM_USERSPACE_IRQ_SOURCE_ID,
 					irq_event.irq, irq_event.level);
 			if (ioctl == KVM_IRQ_LINE_STATUS) {
 				r = -EFAULT;
 				irq_event.status = status;
 				if (copy_to_user(argp, &irq_event,
 							sizeof irq_event))
 					goto out;
 			}
 			r = 0;
 		}
 		break;
 	}
 	case KVM_GET_IRQCHIP: {
 		/* 0: PIC master, 1: PIC slave, 2: IOAPIC */
 		struct kvm_irqchip *chip = kmalloc(sizeof(*chip), GFP_KERNEL);
 		r = -ENOMEM;
 		if (!chip)
 			goto out;
 		r = -EFAULT;
 		if (copy_from_user(chip, argp, sizeof *chip))
 			goto get_irqchip_out;
 		r = -ENXIO;
 		if (!irqchip_in_kernel(kvm))
 			goto get_irqchip_out;
 		r = kvm_vm_ioctl_get_irqchip(kvm, chip);
 		if (r)
 			goto get_irqchip_out;
 		r = -EFAULT;
 		if (copy_to_user(argp, chip, sizeof *chip))
 			goto get_irqchip_out;
 		r = 0;
 	get_irqchip_out:
 		kfree(chip);
 		if (r)
 			goto out;
 		break;
 	}
 	case KVM_SET_IRQCHIP: {
 		/* 0: PIC master, 1: PIC slave, 2: IOAPIC */
 		struct kvm_irqchip *chip = kmalloc(sizeof(*chip), GFP_KERNEL);
 		r = -ENOMEM;
 		if (!chip)
 			goto out;
 		r = -EFAULT;
 		if (copy_from_user(chip, argp, sizeof *chip))
 			goto set_irqchip_out;
 		r = -ENXIO;
 		if (!irqchip_in_kernel(kvm))
 			goto set_irqchip_out;
 		r = kvm_vm_ioctl_set_irqchip(kvm, chip);
 		if (r)
 			goto set_irqchip_out;
 		r = 0;
 	set_irqchip_out:
 		kfree(chip);
 		if (r)
 			goto out;
 		break;
 	}
 	case KVM_GET_PIT: {
 		r = -EFAULT;
 		if (copy_from_user(&u.ps, argp, sizeof(struct kvm_pit_state)))
 			goto out;
 		r = -ENXIO;
 		if (!kvm->arch.vpit)
 			goto out;
 		r = kvm_vm_ioctl_get_pit(kvm, &u.ps);
 		if (r)
 			goto out;
 		r = -EFAULT;
 		if (copy_to_user(argp, &u.ps, sizeof(struct kvm_pit_state)))
 			goto out;
 		r = 0;
 		break;
 	}
 	case KVM_SET_PIT: {
 		r = -EFAULT;
 		if (copy_from_user(&u.ps, argp, sizeof u.ps))
 			goto out;
 		r = -ENXIO;
 		if (!kvm->arch.vpit)
 			goto out;
 		r = kvm_vm_ioctl_set_pit(kvm, &u.ps);
 		if (r)
 			goto out;
 		r = 0;
 		break;
 	}
 	case KVM_GET_PIT2: {
 		r = -ENXIO;
 		if (!kvm->arch.vpit)
 			goto out;
 		r = kvm_vm_ioctl_get_pit2(kvm, &u.ps2);
 		if (r)
 			goto out;
 		r = -EFAULT;
 		if (copy_to_user(argp, &u.ps2, sizeof(u.ps2)))
 			goto out;
 		r = 0;
 		break;
 	}
 	case KVM_SET_PIT2: {
 		r = -EFAULT;
 		if (copy_from_user(&u.ps2, argp, sizeof(u.ps2)))
 			goto out;
 		r = -ENXIO;
 		if (!kvm->arch.vpit)
 			goto out;
 		r = kvm_vm_ioctl_set_pit2(kvm, &u.ps2);
 		if (r)
 			goto out;
 		r = 0;
 		break;
 	}
 	case KVM_REINJECT_CONTROL: {
 		struct kvm_reinject_control control;
 		r =  -EFAULT;
 		if (copy_from_user(&control, argp, sizeof(control)))
 			goto out;
 		r = kvm_vm_ioctl_reinject(kvm, &control);
 		if (r)
 			goto out;
 		r = 0;
 		break;
 	}
 	case KVM_XEN_HVM_CONFIG: {
 		r = -EFAULT;
 		if (copy_from_user(&kvm->arch.xen_hvm_config, argp,
 				   sizeof(struct kvm_xen_hvm_config)))
 			goto out;
 		r = -EINVAL;
 		if (kvm->arch.xen_hvm_config.flags)
 			goto out;
 		r = 0;
 		break;
 	}
 	case KVM_SET_CLOCK: {
 		struct timespec now;
 		struct kvm_clock_data user_ns;
 		u64 now_ns;
 		s64 delta;
 		r = -EFAULT;
 		if (copy_from_user(&user_ns, argp, sizeof(user_ns)))
 			goto out;
 		r = -EINVAL;
 		if (user_ns.flags)
 			goto out;
 		r = 0;
 		ktime_get_ts(&now);
 		now_ns = timespec_to_ns(&now);
 		delta = user_ns.clock - now_ns;
 		kvm->arch.kvmclock_offset = delta;
 		break;
 	}
 	case KVM_GET_CLOCK: {
 		struct timespec now;
 		struct kvm_clock_data user_ns;
 		u64 now_ns;
 		ktime_get_ts(&now);
 		now_ns = timespec_to_ns(&now);
 		user_ns.clock = kvm->arch.kvmclock_offset + now_ns;
 		user_ns.flags = 0;
 		r = -EFAULT;
 		if (copy_to_user(argp, &user_ns, sizeof(user_ns)))
 			goto out;
 		r = 0;
 		break;
 	}
 	default:
 		;
 	}
 out:
 	return r;
 }
 static void kvm_init_msr_list(void)
 {
 	u32 dummy[2];
 	unsigned i, j;
 	/* skip the first msrs in the list. KVM-specific */
 	for (i = j = KVM_SAVE_MSRS_BEGIN; i < ARRAY_SIZE(msrs_to_save); i++) {
 		if (rdmsr_safe(msrs_to_save[i], &dummy[0], &dummy[1]) < 0)
 			continue;
 		if (j < i)
 			msrs_to_save[j] = msrs_to_save[i];
 		j++;
 	}
 	num_msrs_to_save = j;
 }
 static int vcpu_mmio_write(struct kvm_vcpu *vcpu, gpa_t addr, int len,
 			   const void *v)
 {
 	if (vcpu->arch.apic &&
 	    !kvm_iodevice_write(&vcpu->arch.apic->dev, addr, len, v))
 		return 0;
 	return kvm_io_bus_write(vcpu->kvm, KVM_MMIO_BUS, addr, len, v);
 }
 static int vcpu_mmio_read(struct kvm_vcpu *vcpu, gpa_t addr, int len, void *v)
 {
 	if (vcpu->arch.apic &&
 	    !kvm_iodevice_read(&vcpu->arch.apic->dev, addr, len, v))
 		return 0;
 	return kvm_io_bus_read(vcpu->kvm, KVM_MMIO_BUS, addr, len, v);
 }
 static void kvm_set_segment(struct kvm_vcpu *vcpu,
 			struct kvm_segment *var, int seg)
 {
 	kvm_x86_ops->set_segment(vcpu, var, seg);
 }
 void kvm_get_segment(struct kvm_vcpu *vcpu,
 		     struct kvm_segment *var, int seg)
 {
 	kvm_x86_ops->get_segment(vcpu, var, seg);
 }
 gpa_t kvm_mmu_gva_to_gpa_read(struct kvm_vcpu *vcpu, gva_t gva, u32 *error)
 {
 	u32 access = (kvm_x86_ops->get_cpl(vcpu) == 3) ? PFERR_USER_MASK : 0;
 	return vcpu->arch.mmu.gva_to_gpa(vcpu, gva, access, error);
 }
  gpa_t kvm_mmu_gva_to_gpa_fetch(struct kvm_vcpu *vcpu, gva_t gva, u32 *error)
 {
 	u32 access = (kvm_x86_ops->get_cpl(vcpu) == 3) ? PFERR_USER_MASK : 0;
 	access |= PFERR_FETCH_MASK;
 	return vcpu->arch.mmu.gva_to_gpa(vcpu, gva, access, error);
 }
 gpa_t kvm_mmu_gva_to_gpa_write(struct kvm_vcpu *vcpu, gva_t gva, u32 *error)
 {
 	u32 access = (kvm_x86_ops->get_cpl(vcpu) == 3) ? PFERR_USER_MASK : 0;
 	access |= PFERR_WRITE_MASK;
 	return vcpu->arch.mmu.gva_to_gpa(vcpu, gva, access, error);
 }
 /* uses this to access any guest's mapped memory without checking CPL */
 gpa_t kvm_mmu_gva_to_gpa_system(struct kvm_vcpu *vcpu, gva_t gva, u32 *error)
 {
 	return vcpu->arch.mmu.gva_to_gpa(vcpu, gva, 0, error);
 }
 static int kvm_read_guest_virt_helper(gva_t addr, void *val, unsigned int bytes,
 				      struct kvm_vcpu *vcpu, u32 access,
 				      u32 *error)
 {
 	void *data = val;
 	int r = X86EMUL_CONTINUE;
 	while (bytes) {
 		gpa_t gpa = vcpu->arch.mmu.gva_to_gpa(vcpu, addr, access, error);
 		unsigned offset = addr & (PAGE_SIZE-1);
 		unsigned toread = min(bytes, (unsigned)PAGE_SIZE - offset);
 		int ret;
 		if (gpa == UNMAPPED_GVA) {
 			r = X86EMUL_PROPAGATE_FAULT;
 			goto out;
 		}
 		ret = kvm_read_guest(vcpu->kvm, gpa, data, toread);
 		if (ret < 0) {
 			r = X86EMUL_IO_NEEDED;
 			goto out;
 		}
 		bytes -= toread;
 		data += toread;
 		addr += toread;
 	}
 out:
 	return r;
 }
 /* used for instruction fetching */
 static int kvm_fetch_guest_virt(gva_t addr, void *val, unsigned int bytes,
 				struct kvm_vcpu *vcpu, u32 *error)
 {
 	u32 access = (kvm_x86_ops->get_cpl(vcpu) == 3) ? PFERR_USER_MASK : 0;
 	return kvm_read_guest_virt_helper(addr, val, bytes, vcpu,
 					  access | PFERR_FETCH_MASK, error);
 }
 static int kvm_read_guest_virt(gva_t addr, void *val, unsigned int bytes,
 			       struct kvm_vcpu *vcpu, u32 *error)
 {
 	u32 access = (kvm_x86_ops->get_cpl(vcpu) == 3) ? PFERR_USER_MASK : 0;
 	return kvm_read_guest_virt_helper(addr, val, bytes, vcpu, access,
 					  error);
 }
 static int kvm_read_guest_virt_system(gva_t addr, void *val, unsigned int bytes,
 			       struct kvm_vcpu *vcpu, u32 *error)
 {
 	return kvm_read_guest_virt_helper(addr, val, bytes, vcpu, 0, error);
 }
 static int kvm_write_guest_virt_system(gva_t addr, void *val,
 				       unsigned int bytes,
 				       struct kvm_vcpu *vcpu,
 				       u32 *error)
 {
 	void *data = val;
 	int r = X86EMUL_CONTINUE;
 	while (bytes) {
 		gpa_t gpa =  vcpu->arch.mmu.gva_to_gpa(vcpu, addr,
 						       PFERR_WRITE_MASK, error);
 		unsigned offset = addr & (PAGE_SIZE-1);
 		unsigned towrite = min(bytes, (unsigned)PAGE_SIZE - offset);
 		int ret;
 		if (gpa == UNMAPPED_GVA) {
 			r = X86EMUL_PROPAGATE_FAULT;
 			goto out;
 		}
 		ret = kvm_write_guest(vcpu->kvm, gpa, data, towrite);
 		if (ret < 0) {
 			r = X86EMUL_IO_NEEDED;
 			goto out;
 		}
 		bytes -= towrite;
 		data += towrite;
 		addr += towrite;
 	}
 out:
 	return r;
 }
 static int emulator_read_emulated(unsigned long addr,
 				  void *val,
 				  unsigned int bytes,
 				  unsigned int *error_code,
 				  struct kvm_vcpu *vcpu)
 {
 	gpa_t                 gpa;
 	if (vcpu->mmio_read_completed) {
 		memcpy(val, vcpu->mmio_data, bytes);
 		trace_kvm_mmio(KVM_TRACE_MMIO_READ, bytes,
 			       vcpu->mmio_phys_addr, *(u64 *)val);
 		vcpu->mmio_read_completed = 0;
 		return X86EMUL_CONTINUE;
 	}
 	gpa = kvm_mmu_gva_to_gpa_read(vcpu, addr, error_code);
 	if (gpa == UNMAPPED_GVA)
 		return X86EMUL_PROPAGATE_FAULT;
 	/* For APIC access vmexit */
 	if ((gpa & PAGE_MASK) == APIC_DEFAULT_PHYS_BASE)
 		goto mmio;
 	if (kvm_read_guest_virt(addr, val, bytes, vcpu, NULL)
 				== X86EMUL_CONTINUE)
 		return X86EMUL_CONTINUE;
 mmio:
 	/*
 	 * Is this MMIO handled locally?
 	 */
 	if (!vcpu_mmio_read(vcpu, gpa, bytes, val)) {
 		trace_kvm_mmio(KVM_TRACE_MMIO_READ, bytes, gpa, *(u64 *)val);
 		return X86EMUL_CONTINUE;
 	}
 	trace_kvm_mmio(KVM_TRACE_MMIO_READ_UNSATISFIED, bytes, gpa, 0);
 	vcpu->mmio_needed = 1;
 	vcpu->run->exit_reason = KVM_EXIT_MMIO;
 	vcpu->run->mmio.phys_addr = vcpu->mmio_phys_addr = gpa;
 	vcpu->run->mmio.len = vcpu->mmio_size = bytes;
 	vcpu->run->mmio.is_write = vcpu->mmio_is_write = 0;
 	return X86EMUL_IO_NEEDED;
 }
 int emulator_write_phys(struct kvm_vcpu *vcpu, gpa_t gpa,
 			  const void *val, int bytes)
 {
 	int ret;
 	ret = kvm_write_guest(vcpu->kvm, gpa, val, bytes);
 	if (ret < 0)
 		return 0;
 	kvm_mmu_pte_write(vcpu, gpa, val, bytes, 1);
 	return 1;
 }
 static int emulator_write_emulated_onepage(unsigned long addr,
 					   const void *val,
 					   unsigned int bytes,
 					   unsigned int *error_code,
 					   struct kvm_vcpu *vcpu)
 {
 	gpa_t                 gpa;
 	gpa = kvm_mmu_gva_to_gpa_write(vcpu, addr, error_code);
 	if (gpa == UNMAPPED_GVA)
 		return X86EMUL_PROPAGATE_FAULT;
 	/* For APIC access vmexit */
 	if ((gpa & PAGE_MASK) == APIC_DEFAULT_PHYS_BASE)
 		goto mmio;
 	if (emulator_write_phys(vcpu, gpa, val, bytes))
 		return X86EMUL_CONTINUE;
 mmio:
 	trace_kvm_mmio(KVM_TRACE_MMIO_WRITE, bytes, gpa, *(u64 *)val);
 	/*
 	 * Is this MMIO handled locally?
 	 */
 	if (!vcpu_mmio_write(vcpu, gpa, bytes, val))
 		return X86EMUL_CONTINUE;
 	vcpu->mmio_needed = 1;
 	vcpu->run->exit_reason = KVM_EXIT_MMIO;
 	vcpu->run->mmio.phys_addr = vcpu->mmio_phys_addr = gpa;
 	vcpu->run->mmio.len = vcpu->mmio_size = bytes;
 	vcpu->run->mmio.is_write = vcpu->mmio_is_write = 1;
 	memcpy(vcpu->run->mmio.data, val, bytes);
 	return X86EMUL_CONTINUE;
 }
 int emulator_write_emulated(unsigned long addr,
 			    const void *val,
 			    unsigned int bytes,
 			    unsigned int *error_code,
 			    struct kvm_vcpu *vcpu)
 {
 	/* Crossing a page boundary? */
 	if (((addr + bytes - 1) ^ addr) & PAGE_MASK) {
 		int rc, now;
 		now = -addr & ~PAGE_MASK;
 		rc = emulator_write_emulated_onepage(addr, val, now, error_code,
 						     vcpu);
 		if (rc != X86EMUL_CONTINUE)
 			return rc;
 		addr += now;
 		val += now;
 		bytes -= now;
 	}
 	return emulator_write_emulated_onepage(addr, val, bytes, error_code,
 					       vcpu);
 }
 #define CMPXCHG_TYPE(t, ptr, old, new) \
 	(cmpxchg((t *)(ptr), *(t *)(old), *(t *)(new)) == *(t *)(old))
 #ifdef CONFIG_X86_64
 #  define CMPXCHG64(ptr, old, new) CMPXCHG_TYPE(u64, ptr, old, new)
 #else
 #  define CMPXCHG64(ptr, old, new) \
 	(cmpxchg64((u64 *)(ptr), *(u64 *)(old), *(u64 *)(new)) == *(u64 *)(old))
 #endif
 static int emulator_cmpxchg_emulated(unsigned long addr,
 				     const void *old,
 				     const void *new,
 				     unsigned int bytes,
 				     unsigned int *error_code,
 				     struct kvm_vcpu *vcpu)
 {
 	gpa_t gpa;
 	struct page *page;
 	char *kaddr;
 	bool exchanged;
 	/* guests cmpxchg8b have to be emulated atomically */
 	if (bytes > 8 || (bytes & (bytes - 1)))
 		goto emul_write;
 	gpa = kvm_mmu_gva_to_gpa_write(vcpu, addr, NULL);
 	if (gpa == UNMAPPED_GVA ||
 	    (gpa & PAGE_MASK) == APIC_DEFAULT_PHYS_BASE)
 		goto emul_write;
 	if (((gpa + bytes - 1) & PAGE_MASK) != (gpa & PAGE_MASK))
 		goto emul_write;
 	page = gfn_to_page(vcpu->kvm, gpa >> PAGE_SHIFT);
 	kaddr = kmap_atomic(page, KM_USER0);
 	kaddr += offset_in_page(gpa);
 	switch (bytes) {
 	case 1:
 		exchanged = CMPXCHG_TYPE(u8, kaddr, old, new);
 		break;
 	case 2:
 		exchanged = CMPXCHG_TYPE(u16, kaddr, old, new);
 		break;
 	case 4:
 		exchanged = CMPXCHG_TYPE(u32, kaddr, old, new);
 		break;
 	case 8:
 		exchanged = CMPXCHG64(kaddr, old, new);
 		break;
 	default:
 		BUG();
 	}
 	kunmap_atomic(kaddr, KM_USER0);
 	kvm_release_page_dirty(page);
 	if (!exchanged)
 		return X86EMUL_CMPXCHG_FAILED;
 	kvm_mmu_pte_write(vcpu, gpa, new, bytes, 1);
 	return X86EMUL_CONTINUE;
 emul_write:
 	printk_once(KERN_WARNING "kvm: emulating exchange as write\n");
 	return emulator_write_emulated(addr, new, bytes, error_code, vcpu);
 }
 static int kernel_pio(struct kvm_vcpu *vcpu, void *pd)
 {
 	/* TODO: String I/O for in kernel device */
 	int r;
 	if (vcpu->arch.pio.in)
 		r = kvm_io_bus_read(vcpu->kvm, KVM_PIO_BUS, vcpu->arch.pio.port,
 				    vcpu->arch.pio.size, pd);
 	else
 		r = kvm_io_bus_write(vcpu->kvm, KVM_PIO_BUS,
 				     vcpu->arch.pio.port, vcpu->arch.pio.size,
 				     pd);
 	return r;
 }
 static int emulator_pio_in_emulated(int size, unsigned short port, void *val,
 			     unsigned int count, struct kvm_vcpu *vcpu)
 {
 	if (vcpu->arch.pio.count)
 		goto data_avail;
 	trace_kvm_pio(1, port, size, 1);
 	vcpu->arch.pio.port = port;
 	vcpu->arch.pio.in = 1;
 	vcpu->arch.pio.count  = count;
 	vcpu->arch.pio.size = size;
 	if (!kernel_pio(vcpu, vcpu->arch.pio_data)) {
 	data_avail:
 		memcpy(val, vcpu->arch.pio_data, size * count);
 		vcpu->arch.pio.count = 0;
 		return 1;
 	}
 	vcpu->run->exit_reason = KVM_EXIT_IO;
 	vcpu->run->io.direction = KVM_EXIT_IO_IN;
 	vcpu->run->io.size = size;
 	vcpu->run->io.data_offset = KVM_PIO_PAGE_OFFSET * PAGE_SIZE;
 	vcpu->run->io.count = count;
 	vcpu->run->io.port = port;
 	return 0;
 }
 static int emulator_pio_out_emulated(int size, unsigned short port,
 			      const void *val, unsigned int count,
 			      struct kvm_vcpu *vcpu)
 {
 	trace_kvm_pio(0, port, size, 1);
 	vcpu->arch.pio.port = port;
 	vcpu->arch.pio.in = 0;
 	vcpu->arch.pio.count = count;
 	vcpu->arch.pio.size = size;
 	memcpy(vcpu->arch.pio_data, val, size * count);
 	if (!kernel_pio(vcpu, vcpu->arch.pio_data)) {
 		vcpu->arch.pio.count = 0;
 		return 1;
 	}
 	vcpu->run->exit_reason = KVM_EXIT_IO;
 	vcpu->run->io.direction = KVM_EXIT_IO_OUT;
 	vcpu->run->io.size = size;
 	vcpu->run->io.data_offset = KVM_PIO_PAGE_OFFSET * PAGE_SIZE;
 	vcpu->run->io.count = count;
 	vcpu->run->io.port = port;
 	return 0;
 }
 static unsigned long get_segment_base(struct kvm_vcpu *vcpu, int seg)
 {
 	return kvm_x86_ops->get_segment_base(vcpu, seg);
 }
 int emulate_invlpg(struct kvm_vcpu *vcpu, gva_t address)
 {
 	kvm_mmu_invlpg(vcpu, address);
 	return X86EMUL_CONTINUE;
 }
 int emulate_clts(struct kvm_vcpu *vcpu)
 {
 	kvm_x86_ops->set_cr0(vcpu, kvm_read_cr0_bits(vcpu, ~X86_CR0_TS));
 	kvm_x86_ops->fpu_activate(vcpu);
 	return X86EMUL_CONTINUE;
 }
 int emulator_get_dr(int dr, unsigned long *dest, struct kvm_vcpu *vcpu)
 {
 	return _kvm_get_dr(vcpu, dr, dest);
 }
 int emulator_set_dr(int dr, unsigned long value, struct kvm_vcpu *vcpu)
 {
 	return __kvm_set_dr(vcpu, dr, value);
 }
 static u64 mk_cr_64(u64 curr_cr, u32 new_val)
 {
 	return (curr_cr & ~((1ULL << 32) - 1)) | new_val;
 }
 static unsigned long emulator_get_cr(int cr, struct kvm_vcpu *vcpu)
 {
 	unsigned long value;
 	switch (cr) {
 	case 0:
 		value = kvm_read_cr0(vcpu);
 		break;
 	case 2:
 		value = vcpu->arch.cr2;
 		break;
 	case 3:
 		value = vcpu->arch.cr3;
 		break;
 	case 4:
 		value = kvm_read_cr4(vcpu);
 		break;
 	case 8:
 		value = kvm_get_cr8(vcpu);
 		break;
 	default:
 		vcpu_printf(vcpu, "%s: unexpected cr %u\n", __func__, cr);
 		return 0;
 	}
 	return value;
 }
 static int emulator_set_cr(int cr, unsigned long val, struct kvm_vcpu *vcpu)
 {
 	int res = 0;
 	switch (cr) {
 	case 0:
 		res = kvm_set_cr0(vcpu, mk_cr_64(kvm_read_cr0(vcpu), val));
 		break;
 	case 2:
 		vcpu->arch.cr2 = val;
 		break;
 	case 3:
 		res = kvm_set_cr3(vcpu, val);
 		break;
 	case 4:
 		res = kvm_set_cr4(vcpu, mk_cr_64(kvm_read_cr4(vcpu), val));
 		break;
 	case 8:
 		res = __kvm_set_cr8(vcpu, val & 0xfUL);
 		break;
 	default:
 		vcpu_printf(vcpu, "%s: unexpected cr %u\n", __func__, cr);
 		res = -1;
 	}
 	return res;
 }
 static int emulator_get_cpl(struct kvm_vcpu *vcpu)
 {
 	return kvm_x86_ops->get_cpl(vcpu);
 }
 static void emulator_get_gdt(struct desc_ptr *dt, struct kvm_vcpu *vcpu)
 {
 	kvm_x86_ops->get_gdt(vcpu, dt);
 }
 static unsigned long emulator_get_cached_segment_base(int seg,
 						      struct kvm_vcpu *vcpu)
 {
 	return get_segment_base(vcpu, seg);
 }
 static bool emulator_get_cached_descriptor(struct desc_struct *desc, int seg,
 					   struct kvm_vcpu *vcpu)
 {
 	struct kvm_segment var;
 	kvm_get_segment(vcpu, &var, seg);
 	if (var.unusable)
 		return false;
 	if (var.g)
 		var.limit >>= 12;
 	set_desc_limit(desc, var.limit);
 	set_desc_base(desc, (unsigned long)var.base);
 	desc->type = var.type;
 	desc->s = var.s;
 	desc->dpl = var.dpl;
 	desc->p = var.present;
 	desc->avl = var.avl;
 	desc->l = var.l;
 	desc->d = var.db;
 	desc->g = var.g;
 	return true;
 }
 static void emulator_set_cached_descriptor(struct desc_struct *desc, int seg,
 					   struct kvm_vcpu *vcpu)
 {
 	struct kvm_segment var;
 	/* needed to preserve selector */
 	kvm_get_segment(vcpu, &var, seg);
 	var.base = get_desc_base(desc);
 	var.limit = get_desc_limit(desc);
 	if (desc->g)
 		var.limit = (var.limit << 12) | 0xfff;
 	var.type = desc->type;
 	var.present = desc->p;
 	var.dpl = desc->dpl;
 	var.db = desc->d;
 	var.s = desc->s;
 	var.l = desc->l;
 	var.g = desc->g;
 	var.avl = desc->avl;
 	var.present = desc->p;
 	var.unusable = !var.present;
 	var.padding = 0;
 	kvm_set_segment(vcpu, &var, seg);
 	return;
 }
 static u16 emulator_get_segment_selector(int seg, struct kvm_vcpu *vcpu)
 {
 	struct kvm_segment kvm_seg;
 	kvm_get_segment(vcpu, &kvm_seg, seg);
 	return kvm_seg.selector;
 }
 static void emulator_set_segment_selector(u16 sel, int seg,
 					  struct kvm_vcpu *vcpu)
 {
 	struct kvm_segment kvm_seg;
 	kvm_get_segment(vcpu, &kvm_seg, seg);
 	kvm_seg.selector = sel;
 	kvm_set_segment(vcpu, &kvm_seg, seg);
 }
 static struct x86_emulate_ops emulate_ops = {
 	.read_std            = kvm_read_guest_virt_system,
 	.write_std           = kvm_write_guest_virt_system,
 	.fetch               = kvm_fetch_guest_virt,
 	.read_emulated       = emulator_read_emulated,
 	.write_emulated      = emulator_write_emulated,
 	.cmpxchg_emulated    = emulator_cmpxchg_emulated,
 	.pio_in_emulated     = emulator_pio_in_emulated,
 	.pio_out_emulated    = emulator_pio_out_emulated,
 	.get_cached_descriptor = emulator_get_cached_descriptor,
 	.set_cached_descriptor = emulator_set_cached_descriptor,
 	.get_segment_selector = emulator_get_segment_selector,
 	.set_segment_selector = emulator_set_segment_selector,
 	.get_cached_segment_base = emulator_get_cached_segment_base,
 	.get_gdt             = emulator_get_gdt,
 	.get_cr              = emulator_get_cr,
 	.set_cr              = emulator_set_cr,
 	.cpl                 = emulator_get_cpl,
 	.get_dr              = emulator_get_dr,
 	.set_dr              = emulator_set_dr,
 	.set_msr             = kvm_set_msr,
 	.get_msr             = kvm_get_msr,
 };
 static void cache_all_regs(struct kvm_vcpu *vcpu)
 {
 	kvm_register_read(vcpu, VCPU_REGS_RAX);
 	kvm_register_read(vcpu, VCPU_REGS_RSP);
 	kvm_register_read(vcpu, VCPU_REGS_RIP);
 	vcpu->arch.regs_dirty = ~0;
 }
 static void toggle_interruptibility(struct kvm_vcpu *vcpu, u32 mask)
 {
 	u32 int_shadow = kvm_x86_ops->get_interrupt_shadow(vcpu, mask);
 	/*
 	 * an sti; sti; sequence only disable interrupts for the first
 	 * instruction. So, if the last instruction, be it emulated or
 	 * not, left the system with the INT_STI flag enabled, it
 	 * means that the last instruction is an sti. We should not
 	 * leave the flag on in this case. The same goes for mov ss
 	 */
 	if (!(int_shadow & mask))
 		kvm_x86_ops->set_interrupt_shadow(vcpu, mask);
 }
 static void inject_emulated_exception(struct kvm_vcpu *vcpu)
 {
 	struct x86_emulate_ctxt *ctxt = &vcpu->arch.emulate_ctxt;
 	if (ctxt->exception == PF_VECTOR)
 		kvm_inject_page_fault(vcpu, ctxt->cr2, ctxt->error_code);
 	else if (ctxt->error_code_valid)
 		kvm_queue_exception_e(vcpu, ctxt->exception, ctxt->error_code);
 	else
 		kvm_queue_exception(vcpu, ctxt->exception);
 }
 static int handle_emulation_failure(struct kvm_vcpu *vcpu)
 {
 	++vcpu->stat.insn_emulation_fail;
 	trace_kvm_emulate_insn_failed(vcpu);
 	vcpu->run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
 	vcpu->run->internal.suberror = KVM_INTERNAL_ERROR_EMULATION;
 	vcpu->run->internal.ndata = 0;
 	kvm_queue_exception(vcpu, UD_VECTOR);
 	return EMULATE_FAIL;
 }
 int emulate_instruction(struct kvm_vcpu *vcpu,
 			unsigned long cr2,
 			u16 error_code,
 			int emulation_type)
 {
 	int r;
 	struct decode_cache *c = &vcpu->arch.emulate_ctxt.decode;
 	kvm_clear_exception_queue(vcpu);
 	vcpu->arch.mmio_fault_cr2 = cr2;
 	/*
 	 * TODO: fix emulate.c to use guest_read/write_register
 	 * instead of direct ->regs accesses, can save hundred cycles
 	 * on Intel for instructions that don't read/change RSP, for
 	 * for example.
 	 */
 	cache_all_regs(vcpu);
 	if (!(emulation_type & EMULTYPE_NO_DECODE)) {
 		int cs_db, cs_l;
 		kvm_x86_ops->get_cs_db_l_bits(vcpu, &cs_db, &cs_l);
 		vcpu->arch.emulate_ctxt.vcpu = vcpu;
 		vcpu->arch.emulate_ctxt.eflags = kvm_x86_ops->get_rflags(vcpu);
 		vcpu->arch.emulate_ctxt.eip = kvm_rip_read(vcpu);
 		vcpu->arch.emulate_ctxt.mode =
 			(!is_protmode(vcpu)) ? X86EMUL_MODE_REAL :
 			(vcpu->arch.emulate_ctxt.eflags & X86_EFLAGS_VM)
 			? X86EMUL_MODE_VM86 : cs_l
 			? X86EMUL_MODE_PROT64 :	cs_db
 			? X86EMUL_MODE_PROT32 : X86EMUL_MODE_PROT16;
 		memset(c, 0, sizeof(struct decode_cache));
 		memcpy(c->regs, vcpu->arch.regs, sizeof c->regs);
 		vcpu->arch.emulate_ctxt.interruptibility = 0;
 		vcpu->arch.emulate_ctxt.exception = -1;
 		r = x86_decode_insn(&vcpu->arch.emulate_ctxt, &emulate_ops);
 		trace_kvm_emulate_insn_start(vcpu);
 		/* Only allow emulation of specific instructions on #UD
 		 * (namely VMMCALL, sysenter, sysexit, syscall)*/
 		if (emulation_type & EMULTYPE_TRAP_UD) {
 			if (!c->twobyte)
 				return EMULATE_FAIL;
 			switch (c->b) {
 			case 0x01: /* VMMCALL */
 				if (c->modrm_mod != 3 || c->modrm_rm != 1)
 					return EMULATE_FAIL;
 				break;
 			case 0x34: /* sysenter */
 			case 0x35: /* sysexit */
 				if (c->modrm_mod != 0 || c->modrm_rm != 0)
 					return EMULATE_FAIL;
 				break;
 			case 0x05: /* syscall */
 				if (c->modrm_mod != 0 || c->modrm_rm != 0)
 					return EMULATE_FAIL;
 				break;
 			default:
 				return EMULATE_FAIL;
 			}
 			if (!(c->modrm_reg == 0 || c->modrm_reg == 3))
 				return EMULATE_FAIL;
 		}
 		++vcpu->stat.insn_emulation;
 		if (r)  {
 			if (kvm_mmu_unprotect_page_virt(vcpu, cr2))
 				return EMULATE_DONE;
 			if (emulation_type & EMULTYPE_SKIP)
 				return EMULATE_FAIL;
 			return handle_emulation_failure(vcpu);
 		}
 	}
 	if (emulation_type & EMULTYPE_SKIP) {
 		kvm_rip_write(vcpu, vcpu->arch.emulate_ctxt.decode.eip);
 		return EMULATE_DONE;
 	}
 	/* this is needed for vmware backdor interface to work since it
 	   changes registers values  during IO operation */
 	memcpy(c->regs, vcpu->arch.regs, sizeof c->regs);
 restart:
 	r = x86_emulate_insn(&vcpu->arch.emulate_ctxt, &emulate_ops);
 	if (r) { /* emulation failed */
 		/*
 		 * if emulation was due to access to shadowed page table
 		 * and it failed try to unshadow page and re-entetr the
 		 * guest to let CPU execute the instruction.
 		 */
 		if (kvm_mmu_unprotect_page_virt(vcpu, cr2))
 			return EMULATE_DONE;
 		return handle_emulation_failure(vcpu);
 	}
 	toggle_interruptibility(vcpu, vcpu->arch.emulate_ctxt.interruptibility);
 	kvm_x86_ops->set_rflags(vcpu, vcpu->arch.emulate_ctxt.eflags);
 	memcpy(vcpu->arch.regs, c->regs, sizeof c->regs);
 	kvm_rip_write(vcpu, vcpu->arch.emulate_ctxt.eip);
 	if (vcpu->arch.emulate_ctxt.exception >= 0) {
 		inject_emulated_exception(vcpu);
 		return EMULATE_DONE;
 	}
 	if (vcpu->arch.pio.count) {
 		if (!vcpu->arch.pio.in)
 			vcpu->arch.pio.count = 0;
 		return EMULATE_DO_MMIO;
 	}
 	if (vcpu->mmio_needed) {
 		if (vcpu->mmio_is_write)
 			vcpu->mmio_needed = 0;
 		return EMULATE_DO_MMIO;
 	}
 	if (vcpu->arch.emulate_ctxt.restart)
 		goto restart;
 	return EMULATE_DONE;
 }
 EXPORT_SYMBOL_GPL(emulate_instruction);
 int kvm_fast_pio_out(struct kvm_vcpu *vcpu, int size, unsigned short port)
 {
 	unsigned long val = kvm_register_read(vcpu, VCPU_REGS_RAX);
 	int ret = emulator_pio_out_emulated(size, port, &val, 1, vcpu);
 	/* do not return to emulator after return from userspace */
 	vcpu->arch.pio.count = 0;
 	return ret;
 }
 EXPORT_SYMBOL_GPL(kvm_fast_pio_out);
 static void bounce_off(void *info)
 {
 	/* nothing */
 }
 static int kvmclock_cpufreq_notifier(struct notifier_block *nb, unsigned long val,
 				     void *data)
 {
 	struct cpufreq_freqs *freq = data;
 	struct kvm *kvm;
 	struct kvm_vcpu *vcpu;
 	int i, send_ipi = 0;
 	if (val == CPUFREQ_PRECHANGE && freq->old > freq->new)
 		return 0;
 	if (val == CPUFREQ_POSTCHANGE && freq->old < freq->new)
 		return 0;
 	per_cpu(cpu_tsc_khz, freq->cpu) = freq->new;
 	spin_lock(&kvm_lock);
 	list_for_each_entry(kvm, &vm_list, vm_list) {
 		kvm_for_each_vcpu(i, vcpu, kvm) {
 			if (vcpu->cpu != freq->cpu)
 				continue;
 			if (!kvm_request_guest_time_update(vcpu))
 				continue;
 			if (vcpu->cpu != smp_processor_id())
 				send_ipi++;
 		}
 	}
 	spin_unlock(&kvm_lock);
 	if (freq->old < freq->new && send_ipi) {
 		/*
 		 * We upscale the frequency.  Must make the guest
 		 * doesn't see old kvmclock values while running with
 		 * the new frequency, otherwise we risk the guest sees
 		 * time go backwards.
 		 *
 		 * In case we update the frequency for another cpu
 		 * (which might be in guest context) send an interrupt
 		 * to kick the cpu out of guest context.  Next time
 		 * guest context is entered kvmclock will be updated,
 		 * so the guest will not see stale values.
 		 */
 		smp_call_function_single(freq->cpu, bounce_off, NULL, 1);
 	}
 	return 0;
 }
 static struct notifier_block kvmclock_cpufreq_notifier_block = {
         .notifier_call  = kvmclock_cpufreq_notifier
 };
 static void kvm_timer_init(void)
 {
 	int cpu;
 	if (!boot_cpu_has(X86_FEATURE_CONSTANT_TSC)) {
 		cpufreq_register_notifier(&kvmclock_cpufreq_notifier_block,
 					  CPUFREQ_TRANSITION_NOTIFIER);
 		for_each_online_cpu(cpu) {
 			unsigned long khz = cpufreq_get(cpu);
 			if (!khz)
 				khz = tsc_khz;
 			per_cpu(cpu_tsc_khz, cpu) = khz;
 		}
 	} else {
 		for_each_possible_cpu(cpu)
 			per_cpu(cpu_tsc_khz, cpu) = tsc_khz;
 	}
 }
 static DEFINE_PER_CPU(struct kvm_vcpu *, current_vcpu);
 static int kvm_is_in_guest(void)
 {
 	return percpu_read(current_vcpu) != NULL;
 }
 static int kvm_is_user_mode(void)
 {
 	int user_mode = 3;
 	if (percpu_read(current_vcpu))
 		user_mode = kvm_x86_ops->get_cpl(percpu_read(current_vcpu));
 	return user_mode != 0;
 }
 static unsigned long kvm_get_guest_ip(void)
 {
 	unsigned long ip = 0;
 	if (percpu_read(current_vcpu))
 		ip = kvm_rip_read(percpu_read(current_vcpu));
 	return ip;
 }
 static struct perf_guest_info_callbacks kvm_guest_cbs = {
 	.is_in_guest		= kvm_is_in_guest,
 	.is_user_mode		= kvm_is_user_mode,
 	.get_guest_ip		= kvm_get_guest_ip,
 };
 void kvm_before_handle_nmi(struct kvm_vcpu *vcpu)
 {
 	percpu_write(current_vcpu, vcpu);
 }
 EXPORT_SYMBOL_GPL(kvm_before_handle_nmi);
 void kvm_after_handle_nmi(struct kvm_vcpu *vcpu)
 {
 	percpu_write(current_vcpu, NULL);
 }
 EXPORT_SYMBOL_GPL(kvm_after_handle_nmi);
 int kvm_arch_init(void *opaque)
 {
 	int r;
 	struct kvm_x86_ops *ops = (struct kvm_x86_ops *)opaque;
 	if (kvm_x86_ops) {
 		printk(KERN_ERR "kvm: already loaded the other module\n");
 		r = -EEXIST;
 		goto out;
 	}
 	if (!ops->cpu_has_kvm_support()) {
 		printk(KERN_ERR "kvm: no hardware support\n");
 		r = -EOPNOTSUPP;
 		goto out;
 	}
 	if (ops->disabled_by_bios()) {
 		printk(KERN_ERR "kvm: disabled by bios\n");
 		r = -EOPNOTSUPP;
 		goto out;
 	}
 	r = kvm_mmu_module_init();
 	if (r)
 		goto out;
 	kvm_init_msr_list();
 	kvm_x86_ops = ops;
 	kvm_mmu_set_nonpresent_ptes(0ull, 0ull);
 	kvm_mmu_set_base_ptes(PT_PRESENT_MASK);
 	kvm_mmu_set_mask_ptes(PT_USER_MASK, PT_ACCESSED_MASK,
 			PT_DIRTY_MASK, PT64_NX_MASK, 0);
 	kvm_timer_init();
 	perf_register_guest_info_callbacks(&kvm_guest_cbs);
 	if (cpu_has_xsave)
 		host_xcr0 = xgetbv(XCR_XFEATURE_ENABLED_MASK);
 	return 0;
 out:
 	return r;
 }
 void kvm_arch_exit(void)
 {
 	perf_unregister_guest_info_callbacks(&kvm_guest_cbs);
 	if (!boot_cpu_has(X86_FEATURE_CONSTANT_TSC))
 		cpufreq_unregister_notifier(&kvmclock_cpufreq_notifier_block,
 					    CPUFREQ_TRANSITION_NOTIFIER);
 	kvm_x86_ops = NULL;
 	kvm_mmu_module_exit();
 }
 int kvm_emulate_halt(struct kvm_vcpu *vcpu)
 {
 	++vcpu->stat.halt_exits;
 	if (irqchip_in_kernel(vcpu->kvm)) {
 		vcpu->arch.mp_state = KVM_MP_STATE_HALTED;
 		return 1;
 	} else {
 		vcpu->run->exit_reason = KVM_EXIT_HLT;
 		return 0;
 	}
 }
 EXPORT_SYMBOL_GPL(kvm_emulate_halt);
 static inline gpa_t hc_gpa(struct kvm_vcpu *vcpu, unsigned long a0,
 			   unsigned long a1)
 {
 	if (is_long_mode(vcpu))
 		return a0;
 	else
 		return a0 | ((gpa_t)a1 << 32);
 }
 int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
 {
 	u64 param, ingpa, outgpa, ret;
 	uint16_t code, rep_idx, rep_cnt, res = HV_STATUS_SUCCESS, rep_done = 0;
 	bool fast, longmode;
 	int cs_db, cs_l;
 	/*
 	 * hypercall generates UD from non zero cpl and real mode
 	 * per HYPER-V spec
 	 */
 	if (kvm_x86_ops->get_cpl(vcpu) != 0 || !is_protmode(vcpu)) {
 		kvm_queue_exception(vcpu, UD_VECTOR);
 		return 0;
 	}
 	kvm_x86_ops->get_cs_db_l_bits(vcpu, &cs_db, &cs_l);
 	longmode = is_long_mode(vcpu) && cs_l == 1;
 	if (!longmode) {
 		param = ((u64)kvm_register_read(vcpu, VCPU_REGS_RDX) << 32) |
 			(kvm_register_read(vcpu, VCPU_REGS_RAX) & 0xffffffff);
 		ingpa = ((u64)kvm_register_read(vcpu, VCPU_REGS_RBX) << 32) |
 			(kvm_register_read(vcpu, VCPU_REGS_RCX) & 0xffffffff);
 		outgpa = ((u64)kvm_register_read(vcpu, VCPU_REGS_RDI) << 32) |
 			(kvm_register_read(vcpu, VCPU_REGS_RSI) & 0xffffffff);
 	}
 #ifdef CONFIG_X86_64
 	else {
 		param = kvm_register_read(vcpu, VCPU_REGS_RCX);
 		ingpa = kvm_register_read(vcpu, VCPU_REGS_RDX);
 		outgpa = kvm_register_read(vcpu, VCPU_REGS_R8);
 	}
 #endif
 	code = param & 0xffff;
 	fast = (param >> 16) & 0x1;
 	rep_cnt = (param >> 32) & 0xfff;
 	rep_idx = (param >> 48) & 0xfff;
 	trace_kvm_hv_hypercall(code, fast, rep_cnt, rep_idx, ingpa, outgpa);
 	switch (code) {
 	case HV_X64_HV_NOTIFY_LONG_SPIN_WAIT:
 		kvm_vcpu_on_spin(vcpu);
 		break;
 	default:
 		res = HV_STATUS_INVALID_HYPERCALL_CODE;
 		break;
 	}
 	ret = res | (((u64)rep_done & 0xfff) << 32);
 	if (longmode) {
 		kvm_register_write(vcpu, VCPU_REGS_RAX, ret);
 	} else {
 		kvm_register_write(vcpu, VCPU_REGS_RDX, ret >> 32);
 		kvm_register_write(vcpu, VCPU_REGS_RAX, ret & 0xffffffff);
 	}
 	return 1;
 }
 int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)
 {
 	unsigned long nr, a0, a1, a2, a3, ret;
 	int r = 1;
 	if (kvm_hv_hypercall_enabled(vcpu->kvm))
 		return kvm_hv_hypercall(vcpu);
 	nr = kvm_register_read(vcpu, VCPU_REGS_RAX);
 	a0 = kvm_register_read(vcpu, VCPU_REGS_RBX);
 	a1 = kvm_register_read(vcpu, VCPU_REGS_RCX);
 	a2 = kvm_register_read(vcpu, VCPU_REGS_RDX);
 	a3 = kvm_register_read(vcpu, VCPU_REGS_RSI);
 	trace_kvm_hypercall(nr, a0, a1, a2, a3);
 	if (!is_long_mode(vcpu)) {
 		nr &= 0xFFFFFFFF;
 		a0 &= 0xFFFFFFFF;
 		a1 &= 0xFFFFFFFF;
 		a2 &= 0xFFFFFFFF;
 		a3 &= 0xFFFFFFFF;
 	}
 	if (kvm_x86_ops->get_cpl(vcpu) != 0) {
 		ret = -KVM_EPERM;
 		goto out;
 	}
 	switch (nr) {
 	case KVM_HC_VAPIC_POLL_IRQ:
 		ret = 0;
 		break;
 	case KVM_HC_MMU_OP:
 		r = kvm_pv_mmu_op(vcpu, a0, hc_gpa(vcpu, a1, a2), &ret);
 		break;
 	default:
 		ret = -KVM_ENOSYS;
 		break;
 	}
 out:
 	kvm_register_write(vcpu, VCPU_REGS_RAX, ret);
 	++vcpu->stat.hypercalls;
 	return r;
 }
 EXPORT_SYMBOL_GPL(kvm_emulate_hypercall);
 int kvm_fix_hypercall(struct kvm_vcpu *vcpu)
 {
 	char instruction[3];
 	unsigned long rip = kvm_rip_read(vcpu);
 	/*
 	 * Blow out the MMU to ensure that no other VCPU has an active mapping
 	 * to ensure that the updated hypercall appears atomically across all
 	 * VCPUs.
 	 */
 	kvm_mmu_zap_all(vcpu->kvm);
 	kvm_x86_ops->patch_hypercall(vcpu, instruction);
 	return emulator_write_emulated(rip, instruction, 3, NULL, vcpu);
 }
 void realmode_lgdt(struct kvm_vcpu *vcpu, u16 limit, unsigned long base)
 {
 	struct desc_ptr dt = { limit, base };
 	kvm_x86_ops->set_gdt(vcpu, &dt);
 }
 void realmode_lidt(struct kvm_vcpu *vcpu, u16 limit, unsigned long base)
 {
 	struct desc_ptr dt = { limit, base };
 	kvm_x86_ops->set_idt(vcpu, &dt);
 }
 static int move_to_next_stateful_cpuid_entry(struct kvm_vcpu *vcpu, int i)
 {
 	struct kvm_cpuid_entry2 *e = &vcpu->arch.cpuid_entries[i];
 	int j, nent = vcpu->arch.cpuid_nent;
 	e->flags &= ~KVM_CPUID_FLAG_STATE_READ_NEXT;
 	/* when no next entry is found, the current entry[i] is reselected */
 	for (j = i + 1; ; j = (j + 1) % nent) {
 		struct kvm_cpuid_entry2 *ej = &vcpu->arch.cpuid_entries[j];
 		if (ej->function == e->function) {
 			ej->flags |= KVM_CPUID_FLAG_STATE_READ_NEXT;
 			return j;
 		}
 	}
 	return 0; /* silence gcc, even though control never reaches here */
 }
 /* find an entry with matching function, matching index (if needed), and that
  * should be read next (if it's stateful) */
 static int is_matching_cpuid_entry(struct kvm_cpuid_entry2 *e,
 	u32 function, u32 index)
 {
 	if (e->function != function)
 		return 0;
 	if ((e->flags & KVM_CPUID_FLAG_SIGNIFCANT_INDEX) && e->index != index)
 		return 0;
 	if ((e->flags & KVM_CPUID_FLAG_STATEFUL_FUNC) &&
 	    !(e->flags & KVM_CPUID_FLAG_STATE_READ_NEXT))
 		return 0;
 	return 1;
 }
 struct kvm_cpuid_entry2 *kvm_find_cpuid_entry(struct kvm_vcpu *vcpu,
 					      u32 function, u32 index)
 {
 	int i;
 	struct kvm_cpuid_entry2 *best = NULL;
 	for (i = 0; i < vcpu->arch.cpuid_nent; ++i) {
 		struct kvm_cpuid_entry2 *e;
 		e = &vcpu->arch.cpuid_entries[i];
 		if (is_matching_cpuid_entry(e, function, index)) {
 			if (e->flags & KVM_CPUID_FLAG_STATEFUL_FUNC)
 				move_to_next_stateful_cpuid_entry(vcpu, i);
 			best = e;
 			break;
 		}
 		/*
 		 * Both basic or both extended?
 		 */
 		if (((e->function ^ function) & 0x80000000) == 0)
 			if (!best || e->function > best->function)
 				best = e;
 	}
 	return best;
 }
 EXPORT_SYMBOL_GPL(kvm_find_cpuid_entry);
 int cpuid_maxphyaddr(struct kvm_vcpu *vcpu)
 {
 	struct kvm_cpuid_entry2 *best;
 	best = kvm_find_cpuid_entry(vcpu, 0x80000000, 0);
 	if (!best || best->eax < 0x80000008)
 		goto not_found;
 	best = kvm_find_cpuid_entry(vcpu, 0x80000008, 0);
 	if (best)
 		return best->eax & 0xff;
 not_found:
 	return 36;
 }
 void kvm_emulate_cpuid(struct kvm_vcpu *vcpu)
 {
 	u32 function, index;
 	struct kvm_cpuid_entry2 *best;
 	function = kvm_register_read(vcpu, VCPU_REGS_RAX);
 	index = kvm_register_read(vcpu, VCPU_REGS_RCX);
 	kvm_register_write(vcpu, VCPU_REGS_RAX, 0);
 	kvm_register_write(vcpu, VCPU_REGS_RBX, 0);
 	kvm_register_write(vcpu, VCPU_REGS_RCX, 0);
 	kvm_register_write(vcpu, VCPU_REGS_RDX, 0);
 	best = kvm_find_cpuid_entry(vcpu, function, index);
 	if (best) {
 		kvm_register_write(vcpu, VCPU_REGS_RAX, best->eax);
 		kvm_register_write(vcpu, VCPU_REGS_RBX, best->ebx);
 		kvm_register_write(vcpu, VCPU_REGS_RCX, best->ecx);
 		kvm_register_write(vcpu, VCPU_REGS_RDX, best->edx);
 	}
 	kvm_x86_ops->skip_emulated_instruction(vcpu);
 	trace_kvm_cpuid(function,
 			kvm_register_read(vcpu, VCPU_REGS_RAX),
 			kvm_register_read(vcpu, VCPU_REGS_RBX),
 			kvm_register_read(vcpu, VCPU_REGS_RCX),
 			kvm_register_read(vcpu, VCPU_REGS_RDX));
 }
 EXPORT_SYMBOL_GPL(kvm_emulate_cpuid);
 /*
  * Check if userspace requested an interrupt window, and that the
  * interrupt window is open.
  *
  * No need to exit to userspace if we already have an interrupt queued.
  */
 static int dm_request_for_irq_injection(struct kvm_vcpu *vcpu)
 {
 	return (!irqchip_in_kernel(vcpu->kvm) && !kvm_cpu_has_interrupt(vcpu) &&
 		vcpu->run->request_interrupt_window &&
 		kvm_arch_interrupt_allowed(vcpu));
 }
 static void post_kvm_run_save(struct kvm_vcpu *vcpu)
 {
 	struct kvm_run *kvm_run = vcpu->run;
 	kvm_run->if_flag = (kvm_get_rflags(vcpu) & X86_EFLAGS_IF) != 0;
 	kvm_run->cr8 = kvm_get_cr8(vcpu);
 	kvm_run->apic_base = kvm_get_apic_base(vcpu);
 	if (irqchip_in_kernel(vcpu->kvm))
 		kvm_run->ready_for_interrupt_injection = 1;
 	else
 		kvm_run->ready_for_interrupt_injection =
 			kvm_arch_interrupt_allowed(vcpu) &&
 			!kvm_cpu_has_interrupt(vcpu) &&
 			!kvm_event_needs_reinjection(vcpu);
 }
 static void vapic_enter(struct kvm_vcpu *vcpu)
 {
 	struct kvm_lapic *apic = vcpu->arch.apic;
 	struct page *page;
 	if (!apic || !apic->vapic_addr)
 		return;
 	page = gfn_to_page(vcpu->kvm, apic->vapic_addr >> PAGE_SHIFT);
 	vcpu->arch.apic->vapic_page = page;
 }
 static void vapic_exit(struct kvm_vcpu *vcpu)
 {
 	struct kvm_lapic *apic = vcpu->arch.apic;
 	int idx;
 	if (!apic || !apic->vapic_addr)
 		return;
 	idx = srcu_read_lock(&vcpu->kvm->srcu);
 	kvm_release_page_dirty(apic->vapic_page);
 	mark_page_dirty(vcpu->kvm, apic->vapic_addr >> PAGE_SHIFT);
 	srcu_read_unlock(&vcpu->kvm->srcu, idx);
 }
 static void update_cr8_intercept(struct kvm_vcpu *vcpu)
 {
 	int max_irr, tpr;
 	if (!kvm_x86_ops->update_cr8_intercept)
 		return;
 	if (!vcpu->arch.apic)
 		return;
 	if (!vcpu->arch.apic->vapic_addr)
 		max_irr = kvm_lapic_find_highest_irr(vcpu);
 	else
 		max_irr = -1;
 	if (max_irr != -1)
 		max_irr >>= 4;
 	tpr = kvm_lapic_get_cr8(vcpu);
 	kvm_x86_ops->update_cr8_intercept(vcpu, tpr, max_irr);
 }
 static void inject_pending_event(struct kvm_vcpu *vcpu)
 {
 	/* try to reinject previous events if any */
 	if (vcpu->arch.exception.pending) {
 		trace_kvm_inj_exception(vcpu->arch.exception.nr,
 					vcpu->arch.exception.has_error_code,
 					vcpu->arch.exception.error_code);
 		kvm_x86_ops->queue_exception(vcpu, vcpu->arch.exception.nr,
 					  vcpu->arch.exception.has_error_code,
 					  vcpu->arch.exception.error_code,
 					  vcpu->arch.exception.reinject);
 		return;
 	}
 	if (vcpu->arch.nmi_injected) {
 		kvm_x86_ops->set_nmi(vcpu);
 		return;
 	}
 	if (vcpu->arch.interrupt.pending) {
 		kvm_x86_ops->set_irq(vcpu);
 		return;
 	}
 	/* try to inject new event if pending */
 	if (vcpu->arch.nmi_pending) {
 		if (kvm_x86_ops->nmi_allowed(vcpu)) {
 			vcpu->arch.nmi_pending = false;
 			vcpu->arch.nmi_injected = true;
 			kvm_x86_ops->set_nmi(vcpu);
 		}
 	} else if (kvm_cpu_has_interrupt(vcpu)) {
 		if (kvm_x86_ops->interrupt_allowed(vcpu)) {
 			kvm_queue_interrupt(vcpu, kvm_cpu_get_interrupt(vcpu),
 					    false);
 			kvm_x86_ops->set_irq(vcpu);
 		}
 	}
 }
 static void kvm_load_guest_xcr0(struct kvm_vcpu *vcpu)
 {
 	if (kvm_read_cr4_bits(vcpu, X86_CR4_OSXSAVE) &&
 			!vcpu->guest_xcr0_loaded) {
 		/* kvm_set_xcr() also depends on this */
 		xsetbv(XCR_XFEATURE_ENABLED_MASK, vcpu->arch.xcr0);
 		vcpu->guest_xcr0_loaded = 1;
 	}
 }
 static void kvm_put_guest_xcr0(struct kvm_vcpu *vcpu)
 {
 	if (vcpu->guest_xcr0_loaded) {
 		if (vcpu->arch.xcr0 != host_xcr0)
 			xsetbv(XCR_XFEATURE_ENABLED_MASK, host_xcr0);
 		vcpu->guest_xcr0_loaded = 0;
 	}
 }
 static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 {
 	int r;
 	bool req_int_win = !irqchip_in_kernel(vcpu->kvm) &&
 		vcpu->run->request_interrupt_window;
 	if (vcpu->requests)
 		if (test_and_clear_bit(KVM_REQ_MMU_RELOAD, &vcpu->requests))
 			kvm_mmu_unload(vcpu);
 	r = kvm_mmu_reload(vcpu);
 	if (unlikely(r))
 		goto out;
 	if (vcpu->requests) {
 		if (test_and_clear_bit(KVM_REQ_MIGRATE_TIMER, &vcpu->requests))
 			__kvm_migrate_timers(vcpu);
 		if (test_and_clear_bit(KVM_REQ_KVMCLOCK_UPDATE, &vcpu->requests))
 			kvm_write_guest_time(vcpu);
 		if (test_and_clear_bit(KVM_REQ_MMU_SYNC, &vcpu->requests))
 			kvm_mmu_sync_roots(vcpu);
 		if (test_and_clear_bit(KVM_REQ_TLB_FLUSH, &vcpu->requests))
 			kvm_x86_ops->tlb_flush(vcpu);
 		if (test_and_clear_bit(KVM_REQ_REPORT_TPR_ACCESS,
 				       &vcpu->requests)) {
 			vcpu->run->exit_reason = KVM_EXIT_TPR_ACCESS;
 			r = 0;
 			goto out;
 		}
 		if (test_and_clear_bit(KVM_REQ_TRIPLE_FAULT, &vcpu->requests)) {
 			vcpu->run->exit_reason = KVM_EXIT_SHUTDOWN;
 			r = 0;
 			goto out;
 		}
 		if (test_and_clear_bit(KVM_REQ_DEACTIVATE_FPU, &vcpu->requests)) {
 			vcpu->fpu_active = 0;
 			kvm_x86_ops->fpu_deactivate(vcpu);
 		}
 	}
 	preempt_disable();
 	kvm_x86_ops->prepare_guest_switch(vcpu);
 	if (vcpu->fpu_active)
 		kvm_load_guest_fpu(vcpu);
 	kvm_load_guest_xcr0(vcpu);
 	atomic_set(&vcpu->guest_mode, 1);
 	smp_wmb();
 	local_irq_disable();
 	if (!atomic_read(&vcpu->guest_mode) || vcpu->requests
 	    || need_resched() || signal_pending(current)) {
 		atomic_set(&vcpu->guest_mode, 0);
 		smp_wmb();
 		local_irq_enable();
 		preempt_enable();
 		r = 1;
 		goto out;
 	}
 	inject_pending_event(vcpu);
 	/* enable NMI/IRQ window open exits if needed */
 	if (vcpu->arch.nmi_pending)
 		kvm_x86_ops->enable_nmi_window(vcpu);
 	else if (kvm_cpu_has_interrupt(vcpu) || req_int_win)
 		kvm_x86_ops->enable_irq_window(vcpu);
 	if (kvm_lapic_enabled(vcpu)) {
 		update_cr8_intercept(vcpu);
 		kvm_lapic_sync_to_vapic(vcpu);
 	}
 	srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx);
 	kvm_guest_enter();
 	if (unlikely(vcpu->arch.switch_db_regs)) {
 		set_debugreg(0, 7);
 		set_debugreg(vcpu->arch.eff_db[0], 0);
 		set_debugreg(vcpu->arch.eff_db[1], 1);
 		set_debugreg(vcpu->arch.eff_db[2], 2);
 		set_debugreg(vcpu->arch.eff_db[3], 3);
 	}
 	trace_kvm_entry(vcpu->vcpu_id);
 	kvm_x86_ops->run(vcpu);
 	/*
 	 * If the guest has used debug registers, at least dr7
 	 * will be disabled while returning to the host.
 	 * If we don't have active breakpoints in the host, we don't
 	 * care about the messed up debug address registers. But if
 	 * we have some of them active, restore the old state.
 	 */
 	if (hw_breakpoint_active())
 		hw_breakpoint_restore();
 	atomic_set(&vcpu->guest_mode, 0);
 	smp_wmb();
 	local_irq_enable();
 	++vcpu->stat.exits;
 	/*
 	 * We must have an instruction between local_irq_enable() and
 	 * kvm_guest_exit(), so the timer interrupt isn't delayed by
 	 * the interrupt shadow.  The stat.exits increment will do nicely.
 	 * But we need to prevent reordering, hence this barrier():
 	 */
 	barrier();
 	kvm_guest_exit();
 	preempt_enable();
 	vcpu->srcu_idx = srcu_read_lock(&vcpu->kvm->srcu);
 	/*
 	 * Profile KVM exit RIPs:
 	 */
 	if (unlikely(prof_on == KVM_PROFILING)) {
 		unsigned long rip = kvm_rip_read(vcpu);
 		profile_hit(KVM_PROFILING, (void *)rip);
 	}
 	kvm_lapic_sync_from_vapic(vcpu);
 	r = kvm_x86_ops->handle_exit(vcpu);
 out:
 	return r;
 }
 static int __vcpu_run(struct kvm_vcpu *vcpu)
 {
 	int r;
 	struct kvm *kvm = vcpu->kvm;
 	if (unlikely(vcpu->arch.mp_state == KVM_MP_STATE_SIPI_RECEIVED)) {
 		pr_debug("vcpu %d received sipi with vector # %x\n",
 			 vcpu->vcpu_id, vcpu->arch.sipi_vector);
 		kvm_lapic_reset(vcpu);
 		r = kvm_arch_vcpu_reset(vcpu);
 		if (r)
 			return r;
 		vcpu->arch.mp_state = KVM_MP_STATE_RUNNABLE;
 	}
 	vcpu->srcu_idx = srcu_read_lock(&kvm->srcu);
 	vapic_enter(vcpu);
 	r = 1;
 	while (r > 0) {
 		if (vcpu->arch.mp_state == KVM_MP_STATE_RUNNABLE)
 			r = vcpu_enter_guest(vcpu);
 		else {
 			srcu_read_unlock(&kvm->srcu, vcpu->srcu_idx);
 			kvm_vcpu_block(vcpu);
 			vcpu->srcu_idx = srcu_read_lock(&kvm->srcu);
 			if (test_and_clear_bit(KVM_REQ_UNHALT, &vcpu->requests))
 			{
 				switch(vcpu->arch.mp_state) {
 				case KVM_MP_STATE_HALTED:
 					vcpu->arch.mp_state =
 						KVM_MP_STATE_RUNNABLE;
 				case KVM_MP_STATE_RUNNABLE:
 					break;
 				case KVM_MP_STATE_SIPI_RECEIVED:
 				default:
 					r = -EINTR;
 					break;
 				}
 			}
 		}
 		if (r <= 0)
 			break;
 		clear_bit(KVM_REQ_PENDING_TIMER, &vcpu->requests);
 		if (kvm_cpu_has_pending_timer(vcpu))
 			kvm_inject_pending_timer_irqs(vcpu);
 		if (dm_request_for_irq_injection(vcpu)) {
 			r = -EINTR;
 			vcpu->run->exit_reason = KVM_EXIT_INTR;
 			++vcpu->stat.request_irq_exits;
 		}
 		if (signal_pending(current)) {
 			r = -EINTR;
 			vcpu->run->exit_reason = KVM_EXIT_INTR;
 			++vcpu->stat.signal_exits;
 		}
 		if (need_resched()) {
 			srcu_read_unlock(&kvm->srcu, vcpu->srcu_idx);
 			kvm_resched(vcpu);
 			vcpu->srcu_idx = srcu_read_lock(&kvm->srcu);
 		}
 	}
 	srcu_read_unlock(&kvm->srcu, vcpu->srcu_idx);
 	vapic_exit(vcpu);
 	return r;
 }
 int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
 {
 	int r;
 	sigset_t sigsaved;
 	if (vcpu->sigset_active)
 		sigprocmask(SIG_SETMASK, &vcpu->sigset, &sigsaved);
 	if (unlikely(vcpu->arch.mp_state == KVM_MP_STATE_UNINITIALIZED)) {
 		kvm_vcpu_block(vcpu);
 		clear_bit(KVM_REQ_UNHALT, &vcpu->requests);
 		r = -EAGAIN;
 		goto out;
 	}
 	/* re-sync apic's tpr */
 	if (!irqchip_in_kernel(vcpu->kvm))
 		kvm_set_cr8(vcpu, kvm_run->cr8);
 	if (vcpu->arch.pio.count || vcpu->mmio_needed ||
 	    vcpu->arch.emulate_ctxt.restart) {
 		if (vcpu->mmio_needed) {
 			memcpy(vcpu->mmio_data, kvm_run->mmio.data, 8);
 			vcpu->mmio_read_completed = 1;
 			vcpu->mmio_needed = 0;
 		}
 		vcpu->srcu_idx = srcu_read_lock(&vcpu->kvm->srcu);
 		r = emulate_instruction(vcpu, 0, 0, EMULTYPE_NO_DECODE);
 		srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx);
 		if (r != EMULATE_DONE) {
 			r = 0;
 			goto out;
 		}
 	}
 	if (kvm_run->exit_reason == KVM_EXIT_HYPERCALL)
 		kvm_register_write(vcpu, VCPU_REGS_RAX,
 				     kvm_run->hypercall.ret);
 	r = __vcpu_run(vcpu);
 out:
 	post_kvm_run_save(vcpu);
 	if (vcpu->sigset_active)
 		sigprocmask(SIG_SETMASK, &sigsaved, NULL);
 	return r;
 }
 int kvm_arch_vcpu_ioctl_get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs)
 {
 	regs->rax = kvm_register_read(vcpu, VCPU_REGS_RAX);
 	regs->rbx = kvm_register_read(vcpu, VCPU_REGS_RBX);
 	regs->rcx = kvm_register_read(vcpu, VCPU_REGS_RCX);
 	regs->rdx = kvm_register_read(vcpu, VCPU_REGS_RDX);
 	regs->rsi = kvm_register_read(vcpu, VCPU_REGS_RSI);
 	regs->rdi = kvm_register_read(vcpu, VCPU_REGS_RDI);
 	regs->rsp = kvm_register_read(vcpu, VCPU_REGS_RSP);
 	regs->rbp = kvm_register_read(vcpu, VCPU_REGS_RBP);
 #ifdef CONFIG_X86_64
 	regs->r8 = kvm_register_read(vcpu, VCPU_REGS_R8);
 	regs->r9 = kvm_register_read(vcpu, VCPU_REGS_R9);
 	regs->r10 = kvm_register_read(vcpu, VCPU_REGS_R10);
 	regs->r11 = kvm_register_read(vcpu, VCPU_REGS_R11);
 	regs->r12 = kvm_register_read(vcpu, VCPU_REGS_R12);
 	regs->r13 = kvm_register_read(vcpu, VCPU_REGS_R13);
 	regs->r14 = kvm_register_read(vcpu, VCPU_REGS_R14);
 	regs->r15 = kvm_register_read(vcpu, VCPU_REGS_R15);
 #endif
 	regs->rip = kvm_rip_read(vcpu);
 	regs->rflags = kvm_get_rflags(vcpu);
 	return 0;
 }
 int kvm_arch_vcpu_ioctl_set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs)
 {
 	kvm_register_write(vcpu, VCPU_REGS_RAX, regs->rax);
 	kvm_register_write(vcpu, VCPU_REGS_RBX, regs->rbx);
 	kvm_register_write(vcpu, VCPU_REGS_RCX, regs->rcx);
 	kvm_register_write(vcpu, VCPU_REGS_RDX, regs->rdx);
 	kvm_register_write(vcpu, VCPU_REGS_RSI, regs->rsi);
 	kvm_register_write(vcpu, VCPU_REGS_RDI, regs->rdi);
 	kvm_register_write(vcpu, VCPU_REGS_RSP, regs->rsp);
 	kvm_register_write(vcpu, VCPU_REGS_RBP, regs->rbp);
 #ifdef CONFIG_X86_64
 	kvm_register_write(vcpu, VCPU_REGS_R8, regs->r8);
 	kvm_register_write(vcpu, VCPU_REGS_R9, regs->r9);
 	kvm_register_write(vcpu, VCPU_REGS_R10, regs->r10);
 	kvm_register_write(vcpu, VCPU_REGS_R11, regs->r11);
 	kvm_register_write(vcpu, VCPU_REGS_R12, regs->r12);
 	kvm_register_write(vcpu, VCPU_REGS_R13, regs->r13);
 	kvm_register_write(vcpu, VCPU_REGS_R14, regs->r14);
 	kvm_register_write(vcpu, VCPU_REGS_R15, regs->r15);
 #endif
 	kvm_rip_write(vcpu, regs->rip);
 	kvm_set_rflags(vcpu, regs->rflags);
 	vcpu->arch.exception.pending = false;
 	return 0;
 }
 void kvm_get_cs_db_l_bits(struct kvm_vcpu *vcpu, int *db, int *l)
 {
 	struct kvm_segment cs;
 	kvm_get_segment(vcpu, &cs, VCPU_SREG_CS);
 	*db = cs.db;
 	*l = cs.l;
 }
 EXPORT_SYMBOL_GPL(kvm_get_cs_db_l_bits);
 int kvm_arch_vcpu_ioctl_get_sregs(struct kvm_vcpu *vcpu,
 				  struct kvm_sregs *sregs)
 {
 	struct desc_ptr dt;
 	kvm_get_segment(vcpu, &sregs->cs, VCPU_SREG_CS);
 	kvm_get_segment(vcpu, &sregs->ds, VCPU_SREG_DS);
 	kvm_get_segment(vcpu, &sregs->es, VCPU_SREG_ES);
 	kvm_get_segment(vcpu, &sregs->fs, VCPU_SREG_FS);
 	kvm_get_segment(vcpu, &sregs->gs, VCPU_SREG_GS);
 	kvm_get_segment(vcpu, &sregs->ss, VCPU_SREG_SS);
 	kvm_get_segment(vcpu, &sregs->tr, VCPU_SREG_TR);
 	kvm_get_segment(vcpu, &sregs->ldt, VCPU_SREG_LDTR);
 	kvm_x86_ops->get_idt(vcpu, &dt);
 	sregs->idt.limit = dt.size;
 	sregs->idt.base = dt.address;
 	kvm_x86_ops->get_gdt(vcpu, &dt);
 	sregs->gdt.limit = dt.size;
 	sregs->gdt.base = dt.address;
 	sregs->cr0 = kvm_read_cr0(vcpu);
 	sregs->cr2 = vcpu->arch.cr2;
 	sregs->cr3 = vcpu->arch.cr3;
 	sregs->cr4 = kvm_read_cr4(vcpu);
 	sregs->cr8 = kvm_get_cr8(vcpu);
 	sregs->efer = vcpu->arch.efer;
 	sregs->apic_base = kvm_get_apic_base(vcpu);
 	memset(sregs->interrupt_bitmap, 0, sizeof sregs->interrupt_bitmap);
 	if (vcpu->arch.interrupt.pending && !vcpu->arch.interrupt.soft)
 		set_bit(vcpu->arch.interrupt.nr,
 			(unsigned long *)sregs->interrupt_bitmap);
 	return 0;
 }
 int kvm_arch_vcpu_ioctl_get_mpstate(struct kvm_vcpu *vcpu,
 				    struct kvm_mp_state *mp_state)
 {
 	mp_state->mp_state = vcpu->arch.mp_state;
 	return 0;
 }
 int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu,
 				    struct kvm_mp_state *mp_state)
 {
 	vcpu->arch.mp_state = mp_state->mp_state;
 	return 0;
 }
 int kvm_task_switch(struct kvm_vcpu *vcpu, u16 tss_selector, int reason,
 		    bool has_error_code, u32 error_code)
 {
 	struct decode_cache *c = &vcpu->arch.emulate_ctxt.decode;
 	int cs_db, cs_l, ret;
 	cache_all_regs(vcpu);
 	kvm_x86_ops->get_cs_db_l_bits(vcpu, &cs_db, &cs_l);
 	vcpu->arch.emulate_ctxt.vcpu = vcpu;
 	vcpu->arch.emulate_ctxt.eflags = kvm_x86_ops->get_rflags(vcpu);
 	vcpu->arch.emulate_ctxt.eip = kvm_rip_read(vcpu);
 	vcpu->arch.emulate_ctxt.mode =
 		(!is_protmode(vcpu)) ? X86EMUL_MODE_REAL :
 		(vcpu->arch.emulate_ctxt.eflags & X86_EFLAGS_VM)
 		? X86EMUL_MODE_VM86 : cs_l
 		? X86EMUL_MODE_PROT64 :	cs_db
 		? X86EMUL_MODE_PROT32 : X86EMUL_MODE_PROT16;
 	memset(c, 0, sizeof(struct decode_cache));
 	memcpy(c->regs, vcpu->arch.regs, sizeof c->regs);
 	ret = emulator_task_switch(&vcpu->arch.emulate_ctxt, &emulate_ops,
 				   tss_selector, reason, has_error_code,
 				   error_code);
 	if (ret)
 		return EMULATE_FAIL;
 	memcpy(vcpu->arch.regs, c->regs, sizeof c->regs);
 	kvm_rip_write(vcpu, vcpu->arch.emulate_ctxt.eip);
 	kvm_x86_ops->set_rflags(vcpu, vcpu->arch.emulate_ctxt.eflags);
 	return EMULATE_DONE;
 }
 EXPORT_SYMBOL_GPL(kvm_task_switch);
 int kvm_arch_vcpu_ioctl_set_sregs(struct kvm_vcpu *vcpu,
 				  struct kvm_sregs *sregs)
 {
 	int mmu_reset_needed = 0;
 	int pending_vec, max_bits;
 	struct desc_ptr dt;
 	dt.size = sregs->idt.limit;
 	dt.address = sregs->idt.base;
 	kvm_x86_ops->set_idt(vcpu, &dt);
 	dt.size = sregs->gdt.limit;
 	dt.address = sregs->gdt.base;
 	kvm_x86_ops->set_gdt(vcpu, &dt);
 	vcpu->arch.cr2 = sregs->cr2;
 	mmu_reset_needed |= vcpu->arch.cr3 != sregs->cr3;
 	vcpu->arch.cr3 = sregs->cr3;
 	kvm_set_cr8(vcpu, sregs->cr8);
 	mmu_reset_needed |= vcpu->arch.efer != sregs->efer;
 	kvm_x86_ops->set_efer(vcpu, sregs->efer);
 	kvm_set_apic_base(vcpu, sregs->apic_base);
 	mmu_reset_needed |= kvm_read_cr0(vcpu) != sregs->cr0;
 	kvm_x86_ops->set_cr0(vcpu, sregs->cr0);
 	vcpu->arch.cr0 = sregs->cr0;
 	mmu_reset_needed |= kvm_read_cr4(vcpu) != sregs->cr4;
 	kvm_x86_ops->set_cr4(vcpu, sregs->cr4);
 	if (!is_long_mode(vcpu) && is_pae(vcpu)) {
 		load_pdptrs(vcpu, vcpu->arch.cr3);
 		mmu_reset_needed = 1;
 	}
 	if (mmu_reset_needed)
 		kvm_mmu_reset_context(vcpu);
 	max_bits = (sizeof sregs->interrupt_bitmap) << 3;
 	pending_vec = find_first_bit(
 		(const unsigned long *)sregs->interrupt_bitmap, max_bits);
 	if (pending_vec < max_bits) {
 		kvm_queue_interrupt(vcpu, pending_vec, false);
 		pr_debug("Set back pending irq %d\n", pending_vec);
 		if (irqchip_in_kernel(vcpu->kvm))
 			kvm_pic_clear_isr_ack(vcpu->kvm);
 	}
 	kvm_set_segment(vcpu, &sregs->cs, VCPU_SREG_CS);
 	kvm_set_segment(vcpu, &sregs->ds, VCPU_SREG_DS);
 	kvm_set_segment(vcpu, &sregs->es, VCPU_SREG_ES);
 	kvm_set_segment(vcpu, &sregs->fs, VCPU_SREG_FS);
 	kvm_set_segment(vcpu, &sregs->gs, VCPU_SREG_GS);
 	kvm_set_segment(vcpu, &sregs->ss, VCPU_SREG_SS);
 	kvm_set_segment(vcpu, &sregs->tr, VCPU_SREG_TR);
 	kvm_set_segment(vcpu, &sregs->ldt, VCPU_SREG_LDTR);
 	update_cr8_intercept(vcpu);
 	/* Older userspace won't unhalt the vcpu on reset. */
 	if (kvm_vcpu_is_bsp(vcpu) && kvm_rip_read(vcpu) == 0xfff0 &&
 	    sregs->cs.selector == 0xf000 && sregs->cs.base == 0xffff0000 &&
 	    !is_protmode(vcpu))
 		vcpu->arch.mp_state = KVM_MP_STATE_RUNNABLE;
 	return 0;
 }
 int kvm_arch_vcpu_ioctl_set_guest_debug(struct kvm_vcpu *vcpu,
 					struct kvm_guest_debug *dbg)
 {
 	unsigned long rflags;
 	int i, r;
 	if (dbg->control & (KVM_GUESTDBG_INJECT_DB | KVM_GUESTDBG_INJECT_BP)) {
 		r = -EBUSY;
 		if (vcpu->arch.exception.pending)
 			goto out;
 		if (dbg->control & KVM_GUESTDBG_INJECT_DB)
 			kvm_queue_exception(vcpu, DB_VECTOR);
 		else
 			kvm_queue_exception(vcpu, BP_VECTOR);
 	}
 	/*
 	 * Read rflags as long as potentially injected trace flags are still
 	 * filtered out.
 	 */
 	rflags = kvm_get_rflags(vcpu);
 	vcpu->guest_debug = dbg->control;
 	if (!(vcpu->guest_debug & KVM_GUESTDBG_ENABLE))
 		vcpu->guest_debug = 0;
 	if (vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP) {
 		for (i = 0; i < KVM_NR_DB_REGS; ++i)
 			vcpu->arch.eff_db[i] = dbg->arch.debugreg[i];
 		vcpu->arch.switch_db_regs =
 			(dbg->arch.debugreg[7] & DR7_BP_EN_MASK);
 	} else {
 		for (i = 0; i < KVM_NR_DB_REGS; i++)
 			vcpu->arch.eff_db[i] = vcpu->arch.db[i];
 		vcpu->arch.switch_db_regs = (vcpu->arch.dr7 & DR7_BP_EN_MASK);
 	}
 	if (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP)
 		vcpu->arch.singlestep_rip = kvm_rip_read(vcpu) +
 			get_segment_base(vcpu, VCPU_SREG_CS);
 	/*
 	 * Trigger an rflags update that will inject or remove the trace
 	 * flags.
 	 */
 	kvm_set_rflags(vcpu, rflags);
 	kvm_x86_ops->set_guest_debug(vcpu, dbg);
 	r = 0;
 out:
 	return r;
 }
 /*
  * Translate a guest virtual address to a guest physical address.
  */
 int kvm_arch_vcpu_ioctl_translate(struct kvm_vcpu *vcpu,
 				    struct kvm_translation *tr)
 {
 	unsigned long vaddr = tr->linear_address;
 	gpa_t gpa;
 	int idx;
 	idx = srcu_read_lock(&vcpu->kvm->srcu);
 	gpa = kvm_mmu_gva_to_gpa_system(vcpu, vaddr, NULL);
 	srcu_read_unlock(&vcpu->kvm->srcu, idx);
 	tr->physical_address = gpa;
 	tr->valid = gpa != UNMAPPED_GVA;
 	tr->writeable = 1;
 	tr->usermode = 0;
 	return 0;
 }
 int kvm_arch_vcpu_ioctl_get_fpu(struct kvm_vcpu *vcpu, struct kvm_fpu *fpu)
 {
 	struct i387_fxsave_struct *fxsave =
 			&vcpu->arch.guest_fpu.state->fxsave;
 	memcpy(fpu->fpr, fxsave->st_space, 128);
 	fpu->fcw = fxsave->cwd;
 	fpu->fsw = fxsave->swd;
 	fpu->ftwx = fxsave->twd;
 	fpu->last_opcode = fxsave->fop;
 	fpu->last_ip = fxsave->rip;
 	fpu->last_dp = fxsave->rdp;
 	memcpy(fpu->xmm, fxsave->xmm_space, sizeof fxsave->xmm_space);
 	return 0;
 }
 int kvm_arch_vcpu_ioctl_set_fpu(struct kvm_vcpu *vcpu, struct kvm_fpu *fpu)
 {
 	struct i387_fxsave_struct *fxsave =
 			&vcpu->arch.guest_fpu.state->fxsave;
 	memcpy(fxsave->st_space, fpu->fpr, 128);
 	fxsave->cwd = fpu->fcw;
 	fxsave->swd = fpu->fsw;
 	fxsave->twd = fpu->ftwx;
 	fxsave->fop = fpu->last_opcode;
 	fxsave->rip = fpu->last_ip;
 	fxsave->rdp = fpu->last_dp;
 	memcpy(fxsave->xmm_space, fpu->xmm, sizeof fxsave->xmm_space);
 	return 0;
 }
 int fx_init(struct kvm_vcpu *vcpu)
 {
 	int err;
 	err = fpu_alloc(&vcpu->arch.guest_fpu);
 	if (err)
 		return err;
 	fpu_finit(&vcpu->arch.guest_fpu);
 	/*
 	 * Ensure guest xcr0 is valid for loading
 	 */
 	vcpu->arch.xcr0 = XSTATE_FP;
 	vcpu->arch.cr0 |= X86_CR0_ET;
 	return 0;
 }
 EXPORT_SYMBOL_GPL(fx_init);
 static void fx_free(struct kvm_vcpu *vcpu)
 {
 	fpu_free(&vcpu->arch.guest_fpu);
 }
 void kvm_load_guest_fpu(struct kvm_vcpu *vcpu)
 {
 	if (vcpu->guest_fpu_loaded)
 		return;
 	/*
 	 * Restore all possible states in the guest,
 	 * and assume host would use all available bits.
 	 * Guest xcr0 would be loaded later.
 	 */
 	kvm_put_guest_xcr0(vcpu);
 	vcpu->guest_fpu_loaded = 1;
 	unlazy_fpu(current);
 	fpu_restore_checking(&vcpu->arch.guest_fpu);
 	trace_kvm_fpu(1);
 }
 void kvm_put_guest_fpu(struct kvm_vcpu *vcpu)
 {
 	kvm_put_guest_xcr0(vcpu);
 	if (!vcpu->guest_fpu_loaded)
 		return;
 	vcpu->guest_fpu_loaded = 0;
 	fpu_save_init(&vcpu->arch.guest_fpu);
 	++vcpu->stat.fpu_reload;
 	set_bit(KVM_REQ_DEACTIVATE_FPU, &vcpu->requests);
 	trace_kvm_fpu(0);
 }
 void kvm_arch_vcpu_free(struct kvm_vcpu *vcpu)
 {
 	if (vcpu->arch.time_page) {
 		kvm_release_page_dirty(vcpu->arch.time_page);
 		vcpu->arch.time_page = NULL;
 	}
 	fx_free(vcpu);
 	kvm_x86_ops->vcpu_free(vcpu);
 }
 struct kvm_vcpu *kvm_arch_vcpu_create(struct kvm *kvm,
 						unsigned int id)
 {
 	return kvm_x86_ops->vcpu_create(kvm, id);
 }
 int kvm_arch_vcpu_setup(struct kvm_vcpu *vcpu)
 {
 	int r;
 	vcpu->arch.mtrr_state.have_fixed = 1;
 	vcpu_load(vcpu);
 	r = kvm_arch_vcpu_reset(vcpu);
 	if (r == 0)
 		r = kvm_mmu_setup(vcpu);
 	vcpu_put(vcpu);
 	if (r < 0)
 		goto free_vcpu;
 	return 0;
 free_vcpu:
 	kvm_x86_ops->vcpu_free(vcpu);
 	return r;
 }
 void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu)
 {
 	vcpu_load(vcpu);
 	kvm_mmu_unload(vcpu);
 	vcpu_put(vcpu);
 	fx_free(vcpu);
 	kvm_x86_ops->vcpu_free(vcpu);
 }
 int kvm_arch_vcpu_reset(struct kvm_vcpu *vcpu)
 {
 	vcpu->arch.nmi_pending = false;
 	vcpu->arch.nmi_injected = false;
 	vcpu->arch.switch_db_regs = 0;
 	memset(vcpu->arch.db, 0, sizeof(vcpu->arch.db));
 	vcpu->arch.dr6 = DR6_FIXED_1;
 	vcpu->arch.dr7 = DR7_FIXED_1;
 	return kvm_x86_ops->vcpu_reset(vcpu);
 }
 int kvm_arch_hardware_enable(void *garbage)
 {
 	/*
 	 * Since this may be called from a hotplug notifcation,
 	 * we can't get the CPU frequency directly.
 	 */
 	if (!boot_cpu_has(X86_FEATURE_CONSTANT_TSC)) {
 		int cpu = raw_smp_processor_id();
 		per_cpu(cpu_tsc_khz, cpu) = 0;
 	}
 	kvm_shared_msr_cpu_online();
 	return kvm_x86_ops->hardware_enable(garbage);
 }
 void kvm_arch_hardware_disable(void *garbage)
 {
 	kvm_x86_ops->hardware_disable(garbage);
 	drop_user_return_notifiers(garbage);
 }
 int kvm_arch_hardware_setup(void)
 {
 	return kvm_x86_ops->hardware_setup();
 }
 void kvm_arch_hardware_unsetup(void)
 {
 	kvm_x86_ops->hardware_unsetup();
 }
 void kvm_arch_check_processor_compat(void *rtn)
 {
 	kvm_x86_ops->check_processor_compatibility(rtn);
 }
 int kvm_arch_vcpu_init(struct kvm_vcpu *vcpu)
 {
 	struct page *page;
 	struct kvm *kvm;
 	int r;
 	BUG_ON(vcpu->kvm == NULL);
 	kvm = vcpu->kvm;
 	vcpu->arch.mmu.root_hpa = INVALID_PAGE;
 	if (!irqchip_in_kernel(kvm) || kvm_vcpu_is_bsp(vcpu))
 		vcpu->arch.mp_state = KVM_MP_STATE_RUNNABLE;
 	else
 		vcpu->arch.mp_state = KVM_MP_STATE_UNINITIALIZED;
 	page = alloc_page(GFP_KERNEL | __GFP_ZERO);
 	if (!page) {
 		r = -ENOMEM;
 		goto fail;
 	}
 	vcpu->arch.pio_data = page_address(page);
 	r = kvm_mmu_create(vcpu);
 	if (r < 0)
 		goto fail_free_pio_data;
 	if (irqchip_in_kernel(kvm)) {
 		r = kvm_create_lapic(vcpu);
 		if (r < 0)
 			goto fail_mmu_destroy;
 	}
 	vcpu->arch.mce_banks = kzalloc(KVM_MAX_MCE_BANKS * sizeof(u64) * 4,
 				       GFP_KERNEL);
 	if (!vcpu->arch.mce_banks) {
 		r = -ENOMEM;
 		goto fail_free_lapic;
 	}
 	vcpu->arch.mcg_cap = KVM_MAX_MCE_BANKS;
 	return 0;
 fail_free_lapic:
 	kvm_free_lapic(vcpu);
 fail_mmu_destroy:
 	kvm_mmu_destroy(vcpu);
 fail_free_pio_data:
 	free_page((unsigned long)vcpu->arch.pio_data);
 fail:
 	return r;
 }
 void kvm_arch_vcpu_uninit(struct kvm_vcpu *vcpu)
 {
 	int idx;
 	kfree(vcpu->arch.mce_banks);
 	kvm_free_lapic(vcpu);
 	idx = srcu_read_lock(&vcpu->kvm->srcu);
 	kvm_mmu_destroy(vcpu);
 	srcu_read_unlock(&vcpu->kvm->srcu, idx);
 	free_page((unsigned long)vcpu->arch.pio_data);
 }
 struct  kvm *kvm_arch_create_vm(void)
 {
 	struct kvm *kvm = kzalloc(sizeof(struct kvm), GFP_KERNEL);
 	if (!kvm)
 		return ERR_PTR(-ENOMEM);
-	kvm->arch.aliases = kzalloc(sizeof(struct kvm_mem_aliases), GFP_KERNEL);
-	if (!kvm->arch.aliases) {
-		kfree(kvm);
-		return ERR_PTR(-ENOMEM);
-	}
 	INIT_LIST_HEAD(&kvm->arch.active_mmu_pages);
 	INIT_LIST_HEAD(&kvm->arch.assigned_dev_head);
 	/* Reserve bit 0 of irq_sources_bitmap for userspace irq source */
 	set_bit(KVM_USERSPACE_IRQ_SOURCE_ID, &kvm->arch.irq_sources_bitmap);
 	rdtscll(kvm->arch.vm_init_tsc);
 	return kvm;
 }
 static void kvm_unload_vcpu_mmu(struct kvm_vcpu *vcpu)
 {
 	vcpu_load(vcpu);
 	kvm_mmu_unload(vcpu);
 	vcpu_put(vcpu);
 }
 static void kvm_free_vcpus(struct kvm *kvm)
 {
 	unsigned int i;
 	struct kvm_vcpu *vcpu;
 	/*
 	 * Unpin any mmu pages first.
 	 */
 	kvm_for_each_vcpu(i, vcpu, kvm)
 		kvm_unload_vcpu_mmu(vcpu);
 	kvm_for_each_vcpu(i, vcpu, kvm)
 		kvm_arch_vcpu_free(vcpu);
 	mutex_lock(&kvm->lock);
 	for (i = 0; i < atomic_read(&kvm->online_vcpus); i++)
 		kvm->vcpus[i] = NULL;
 	atomic_set(&kvm->online_vcpus, 0);
 	mutex_unlock(&kvm->lock);
 }
 void kvm_arch_sync_events(struct kvm *kvm)
 {
 	kvm_free_all_assigned_devices(kvm);
 }
 void kvm_arch_destroy_vm(struct kvm *kvm)
 {
 	kvm_iommu_unmap_guest(kvm);
 	kvm_free_pit(kvm);
 	kfree(kvm->arch.vpic);
 	kfree(kvm->arch.vioapic);
 	kvm_free_vcpus(kvm);
 	kvm_free_physmem(kvm);
 	if (kvm->arch.apic_access_page)
 		put_page(kvm->arch.apic_access_page);
 	if (kvm->arch.ept_identity_pagetable)
 		put_page(kvm->arch.ept_identity_pagetable);
 	cleanup_srcu_struct(&kvm->srcu);
-	kfree(kvm->arch.aliases);
 	kfree(kvm);
 }
 int kvm_arch_prepare_memory_region(struct kvm *kvm,
 				struct kvm_memory_slot *memslot,
 				struct kvm_memory_slot old,
 				struct kvm_userspace_memory_region *mem,
 				int user_alloc)
 {
 	int npages = memslot->npages;
 	/*To keep backward compatibility with older userspace,
 	 *x86 needs to hanlde !user_alloc case.
 	 */
 	if (!user_alloc) {
 		if (npages && !old.rmap) {
 			unsigned long userspace_addr;
 			down_write(&current->mm->mmap_sem);
 			userspace_addr = do_mmap(NULL, 0,
 						 npages * PAGE_SIZE,
 						 PROT_READ | PROT_WRITE,
 						 MAP_PRIVATE | MAP_ANONYMOUS,
 						 0);
 			up_write(&current->mm->mmap_sem);
 			if (IS_ERR((void *)userspace_addr))
 				return PTR_ERR((void *)userspace_addr);
 			memslot->userspace_addr = userspace_addr;
 		}
 	}
 	return 0;
 }
 void kvm_arch_commit_memory_region(struct kvm *kvm,
 				struct kvm_userspace_memory_region *mem,
 				struct kvm_memory_slot old,
 				int user_alloc)
 {
 	int npages = mem->memory_size >> PAGE_SHIFT;
 	if (!user_alloc && !old.user_alloc && old.rmap && !npages) {
 		int ret;
 		down_write(&current->mm->mmap_sem);
 		ret = do_munmap(current->mm, old.userspace_addr,
 				old.npages * PAGE_SIZE);
 		up_write(&current->mm->mmap_sem);
 		if (ret < 0)
 			printk(KERN_WARNING
 			       "kvm_vm_ioctl_set_memory_region: "
 			       "failed to munmap memory\n");
 	}
 	spin_lock(&kvm->mmu_lock);
 	if (!kvm->arch.n_requested_mmu_pages) {
 		unsigned int nr_mmu_pages = kvm_mmu_calculate_mmu_pages(kvm);
 		kvm_mmu_change_mmu_pages(kvm, nr_mmu_pages);
 	}
 	kvm_mmu_slot_remove_write_access(kvm, mem->slot);
 	spin_unlock(&kvm->mmu_lock);
 }
 void kvm_arch_flush_shadow(struct kvm *kvm)
 {
 	kvm_mmu_zap_all(kvm);
 	kvm_reload_remote_mmus(kvm);
 }
 int kvm_arch_vcpu_runnable(struct kvm_vcpu *vcpu)
 {
 	return vcpu->arch.mp_state == KVM_MP_STATE_RUNNABLE
 		|| vcpu->arch.mp_state == KVM_MP_STATE_SIPI_RECEIVED
 		|| vcpu->arch.nmi_pending ||
 		(kvm_arch_interrupt_allowed(vcpu) &&
 		 kvm_cpu_has_interrupt(vcpu));
 }
 void kvm_vcpu_kick(struct kvm_vcpu *vcpu)
 {
 	int me;
 	int cpu = vcpu->cpu;
 	if (waitqueue_active(&vcpu->wq)) {
 		wake_up_interruptible(&vcpu->wq);
 		++vcpu->stat.halt_wakeup;
 	}
 	me = get_cpu();
 	if (cpu != me && (unsigned)cpu < nr_cpu_ids && cpu_online(cpu))
 		if (atomic_xchg(&vcpu->guest_mode, 0))
 			smp_send_reschedule(cpu);
 	put_cpu();
 }
 int kvm_arch_interrupt_allowed(struct kvm_vcpu *vcpu)
 {
 	return kvm_x86_ops->interrupt_allowed(vcpu);
 }
 bool kvm_is_linear_rip(struct kvm_vcpu *vcpu, unsigned long linear_rip)
 {
 	unsigned long current_rip = kvm_rip_read(vcpu) +
 		get_segment_base(vcpu, VCPU_SREG_CS);
 	return current_rip == linear_rip;
 }
 EXPORT_SYMBOL_GPL(kvm_is_linear_rip);
 unsigned long kvm_get_rflags(struct kvm_vcpu *vcpu)
 {
 	unsigned long rflags;
 	rflags = kvm_x86_ops->get_rflags(vcpu);
 	if (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP)
 		rflags &= ~X86_EFLAGS_TF;
 	return rflags;
 }
 EXPORT_SYMBOL_GPL(kvm_get_rflags);
 void kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags)
 {
 	if (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP &&
 	    kvm_is_linear_rip(vcpu, vcpu->arch.singlestep_rip))
 		rflags |= X86_EFLAGS_TF;
 	kvm_x86_ops->set_rflags(vcpu, rflags);
 }
 EXPORT_SYMBOL_GPL(kvm_set_rflags);
 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_exit);
 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_inj_virq);
 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_page_fault);
 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_msr);
 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_cr);
 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_nested_vmrun);
 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_nested_vmexit);
 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_nested_vmexit_inject);
 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_nested_intr_vmexit);
 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_invlpga);
 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_skinit);
 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_nested_intercepts);

arch/x86/kvm/x86.h

Diff comments View file @ a1f4d39

 #ifndef ARCH_X86_KVM_X86_H
 #define ARCH_X86_KVM_X86_H
 #include <linux/kvm_host.h>
 #include "kvm_cache_regs.h"
 static inline void kvm_clear_exception_queue(struct kvm_vcpu *vcpu)
 {
 	vcpu->arch.exception.pending = false;
 }
 static inline void kvm_queue_interrupt(struct kvm_vcpu *vcpu, u8 vector,
 	bool soft)
 {
 	vcpu->arch.interrupt.pending = true;
 	vcpu->arch.interrupt.soft = soft;
 	vcpu->arch.interrupt.nr = vector;
 }
 static inline void kvm_clear_interrupt_queue(struct kvm_vcpu *vcpu)
 {
 	vcpu->arch.interrupt.pending = false;
 }
 static inline bool kvm_event_needs_reinjection(struct kvm_vcpu *vcpu)
 {
 	return vcpu->arch.exception.pending || vcpu->arch.interrupt.pending ||
 		vcpu->arch.nmi_injected;
 }
 static inline bool kvm_exception_is_soft(unsigned int nr)
 {
 	return (nr == BP_VECTOR) || (nr == OF_VECTOR);
 }
 struct kvm_cpuid_entry2 *kvm_find_cpuid_entry(struct kvm_vcpu *vcpu,
                                              u32 function, u32 index);
 static inline bool is_protmode(struct kvm_vcpu *vcpu)
 {
 	return kvm_read_cr0_bits(vcpu, X86_CR0_PE);
 }
 static inline int is_long_mode(struct kvm_vcpu *vcpu)
 {
 #ifdef CONFIG_X86_64
 	return vcpu->arch.efer & EFER_LMA;
 #else
 	return 0;
 #endif
 }
 static inline int is_pae(struct kvm_vcpu *vcpu)
 {
 	return kvm_read_cr4_bits(vcpu, X86_CR4_PAE);
 }
 static inline int is_pse(struct kvm_vcpu *vcpu)
 {
 	return kvm_read_cr4_bits(vcpu, X86_CR4_PSE);
 }
 static inline int is_paging(struct kvm_vcpu *vcpu)
 {
 	return kvm_read_cr0_bits(vcpu, X86_CR0_PG);
 }
-static inline struct kvm_mem_aliases *kvm_aliases(struct kvm *kvm)
-{
-	return rcu_dereference_check(kvm->arch.aliases,
-			srcu_read_lock_held(&kvm->srcu)
-			|| lockdep_is_held(&kvm->slots_lock));
-}
 void kvm_before_handle_nmi(struct kvm_vcpu *vcpu);
 void kvm_after_handle_nmi(struct kvm_vcpu *vcpu);
 #endif

include/linux/kvm.h

Diff comments View file @ a1f4d39

 #ifndef __LINUX_KVM_H
 #define __LINUX_KVM_H
 /*
  * Userspace interface for /dev/kvm - kernel based virtual machine
  *
  * Note: you must update KVM_API_VERSION if you change this interface.
  */
 #include <linux/types.h>
 #include <linux/compiler.h>
 #include <linux/ioctl.h>
 #include <asm/kvm.h>
 #define KVM_API_VERSION 12
 /* *** Deprecated interfaces *** */
 #define KVM_TRC_SHIFT           16
 #define KVM_TRC_ENTRYEXIT       (1 << KVM_TRC_SHIFT)
 #define KVM_TRC_HANDLER         (1 << (KVM_TRC_SHIFT + 1))
 #define KVM_TRC_VMENTRY         (KVM_TRC_ENTRYEXIT + 0x01)
 #define KVM_TRC_VMEXIT          (KVM_TRC_ENTRYEXIT + 0x02)
 #define KVM_TRC_PAGE_FAULT      (KVM_TRC_HANDLER + 0x01)
 #define KVM_TRC_HEAD_SIZE       12
 #define KVM_TRC_CYCLE_SIZE      8
 #define KVM_TRC_EXTRA_MAX       7
 #define KVM_TRC_INJ_VIRQ         (KVM_TRC_HANDLER + 0x02)
 #define KVM_TRC_REDELIVER_EVT    (KVM_TRC_HANDLER + 0x03)
 #define KVM_TRC_PEND_INTR        (KVM_TRC_HANDLER + 0x04)
 #define KVM_TRC_IO_READ          (KVM_TRC_HANDLER + 0x05)
 #define KVM_TRC_IO_WRITE         (KVM_TRC_HANDLER + 0x06)
 #define KVM_TRC_CR_READ          (KVM_TRC_HANDLER + 0x07)
 #define KVM_TRC_CR_WRITE         (KVM_TRC_HANDLER + 0x08)
 #define KVM_TRC_DR_READ          (KVM_TRC_HANDLER + 0x09)
 #define KVM_TRC_DR_WRITE         (KVM_TRC_HANDLER + 0x0A)
 #define KVM_TRC_MSR_READ         (KVM_TRC_HANDLER + 0x0B)
 #define KVM_TRC_MSR_WRITE        (KVM_TRC_HANDLER + 0x0C)
 #define KVM_TRC_CPUID            (KVM_TRC_HANDLER + 0x0D)
 #define KVM_TRC_INTR             (KVM_TRC_HANDLER + 0x0E)
 #define KVM_TRC_NMI              (KVM_TRC_HANDLER + 0x0F)
 #define KVM_TRC_VMMCALL          (KVM_TRC_HANDLER + 0x10)
 #define KVM_TRC_HLT              (KVM_TRC_HANDLER + 0x11)
 #define KVM_TRC_CLTS             (KVM_TRC_HANDLER + 0x12)
 #define KVM_TRC_LMSW             (KVM_TRC_HANDLER + 0x13)
 #define KVM_TRC_APIC_ACCESS      (KVM_TRC_HANDLER + 0x14)
 #define KVM_TRC_TDP_FAULT        (KVM_TRC_HANDLER + 0x15)
 #define KVM_TRC_GTLB_WRITE       (KVM_TRC_HANDLER + 0x16)
 #define KVM_TRC_STLB_WRITE       (KVM_TRC_HANDLER + 0x17)
 #define KVM_TRC_STLB_INVAL       (KVM_TRC_HANDLER + 0x18)
 #define KVM_TRC_PPC_INSTR        (KVM_TRC_HANDLER + 0x19)
 struct kvm_user_trace_setup {
 	__u32 buf_size;
 	__u32 buf_nr;
 };
 #define __KVM_DEPRECATED_MAIN_W_0x06 \
 	_IOW(KVMIO, 0x06, struct kvm_user_trace_setup)
 #define __KVM_DEPRECATED_MAIN_0x07 _IO(KVMIO, 0x07)
 #define __KVM_DEPRECATED_MAIN_0x08 _IO(KVMIO, 0x08)
 #define __KVM_DEPRECATED_VM_R_0x70 _IOR(KVMIO, 0x70, struct kvm_assigned_irq)
 struct kvm_breakpoint {
 	__u32 enabled;
 	__u32 padding;
 	__u64 address;
 };
 struct kvm_debug_guest {
 	__u32 enabled;
 	__u32 pad;
 	struct kvm_breakpoint breakpoints[4];
 	__u32 singlestep;
 };
 #define __KVM_DEPRECATED_VCPU_W_0x87 _IOW(KVMIO, 0x87, struct kvm_debug_guest)
 /* *** End of deprecated interfaces *** */
 /* for KVM_CREATE_MEMORY_REGION */
 struct kvm_memory_region {
 	__u32 slot;
 	__u32 flags;
 	__u64 guest_phys_addr;
 	__u64 memory_size; /* bytes */
 };
 /* for KVM_SET_USER_MEMORY_REGION */
 struct kvm_userspace_memory_region {
 	__u32 slot;
 	__u32 flags;
 	__u64 guest_phys_addr;
 	__u64 memory_size; /* bytes */
 	__u64 userspace_addr; /* start of the userspace allocated memory */
 };
 /* for kvm_memory_region::flags */
 #define KVM_MEM_LOG_DIRTY_PAGES  1UL
 #define KVM_MEMSLOT_INVALID      (1UL << 1)
 /* for KVM_IRQ_LINE */
 struct kvm_irq_level {
 	/*
 	 * ACPI gsi notion of irq.
 	 * For IA-64 (APIC model) IOAPIC0: irq 0-23; IOAPIC1: irq 24-47..
 	 * For X86 (standard AT mode) PIC0/1: irq 0-15. IOAPIC0: 0-23..
 	 */
 	union {
 		__u32 irq;
 		__s32 status;
 	};
 	__u32 level;
 };
 struct kvm_irqchip {
 	__u32 chip_id;
 	__u32 pad;
         union {
 		char dummy[512];  /* reserving space */
 #ifdef __KVM_HAVE_PIT
 		struct kvm_pic_state pic;
 #endif
 #ifdef __KVM_HAVE_IOAPIC
 		struct kvm_ioapic_state ioapic;
 #endif
 	} chip;
 };
 /* for KVM_CREATE_PIT2 */
 struct kvm_pit_config {
 	__u32 flags;
 	__u32 pad[15];
 };
 #define KVM_PIT_SPEAKER_DUMMY     1
 #define KVM_EXIT_UNKNOWN          0
 #define KVM_EXIT_EXCEPTION        1
 #define KVM_EXIT_IO               2
 #define KVM_EXIT_HYPERCALL        3
 #define KVM_EXIT_DEBUG            4
 #define KVM_EXIT_HLT              5
 #define KVM_EXIT_MMIO             6
 #define KVM_EXIT_IRQ_WINDOW_OPEN  7
 #define KVM_EXIT_SHUTDOWN         8
 #define KVM_EXIT_FAIL_ENTRY       9
 #define KVM_EXIT_INTR             10
 #define KVM_EXIT_SET_TPR          11
 #define KVM_EXIT_TPR_ACCESS       12
 #define KVM_EXIT_S390_SIEIC       13
 #define KVM_EXIT_S390_RESET       14
 #define KVM_EXIT_DCR              15
 #define KVM_EXIT_NMI              16
 #define KVM_EXIT_INTERNAL_ERROR   17
 #define KVM_EXIT_OSI              18
 /* For KVM_EXIT_INTERNAL_ERROR */
 #define KVM_INTERNAL_ERROR_EMULATION 1
 #define KVM_INTERNAL_ERROR_SIMUL_EX 2
 /* for KVM_RUN, returned by mmap(vcpu_fd, offset=0) */
 struct kvm_run {
 	/* in */
 	__u8 request_interrupt_window;
 	__u8 padding1[7];
 	/* out */
 	__u32 exit_reason;
 	__u8 ready_for_interrupt_injection;
 	__u8 if_flag;
 	__u8 padding2[2];
 	/* in (pre_kvm_run), out (post_kvm_run) */
 	__u64 cr8;
 	__u64 apic_base;
 #ifdef __KVM_S390
 	/* the processor status word for s390 */
 	__u64 psw_mask; /* psw upper half */
 	__u64 psw_addr; /* psw lower half */
 #endif
 	union {
 		/* KVM_EXIT_UNKNOWN */
 		struct {
 			__u64 hardware_exit_reason;
 		} hw;
 		/* KVM_EXIT_FAIL_ENTRY */
 		struct {
 			__u64 hardware_entry_failure_reason;
 		} fail_entry;
 		/* KVM_EXIT_EXCEPTION */
 		struct {
 			__u32 exception;
 			__u32 error_code;
 		} ex;
 		/* KVM_EXIT_IO */
 		struct {
 #define KVM_EXIT_IO_IN  0
 #define KVM_EXIT_IO_OUT 1
 			__u8 direction;
 			__u8 size; /* bytes */
 			__u16 port;
 			__u32 count;
 			__u64 data_offset; /* relative to kvm_run start */
 		} io;
 		struct {
 			struct kvm_debug_exit_arch arch;
 		} debug;
 		/* KVM_EXIT_MMIO */
 		struct {
 			__u64 phys_addr;
 			__u8  data[8];
 			__u32 len;
 			__u8  is_write;
 		} mmio;
 		/* KVM_EXIT_HYPERCALL */
 		struct {
 			__u64 nr;
 			__u64 args[6];
 			__u64 ret;
 			__u32 longmode;
 			__u32 pad;
 		} hypercall;
 		/* KVM_EXIT_TPR_ACCESS */
 		struct {
 			__u64 rip;
 			__u32 is_write;
 			__u32 pad;
 		} tpr_access;
 		/* KVM_EXIT_S390_SIEIC */
 		struct {
 			__u8 icptcode;
 			__u16 ipa;
 			__u32 ipb;
 		} s390_sieic;
 		/* KVM_EXIT_S390_RESET */
 #define KVM_S390_RESET_POR       1
 #define KVM_S390_RESET_CLEAR     2
 #define KVM_S390_RESET_SUBSYSTEM 4
 #define KVM_S390_RESET_CPU_INIT  8
 #define KVM_S390_RESET_IPL       16
 		__u64 s390_reset_flags;
 		/* KVM_EXIT_DCR */
 		struct {
 			__u32 dcrn;
 			__u32 data;
 			__u8  is_write;
 		} dcr;
 		struct {
 			__u32 suberror;
 			/* Available with KVM_CAP_INTERNAL_ERROR_DATA: */
 			__u32 ndata;
 			__u64 data[16];
 		} internal;
 		/* KVM_EXIT_OSI */
 		struct {
 			__u64 gprs[32];
 		} osi;
 		/* Fix the size of the union. */
 		char padding[256];
 	};
 };
 /* for KVM_REGISTER_COALESCED_MMIO / KVM_UNREGISTER_COALESCED_MMIO */
 struct kvm_coalesced_mmio_zone {
 	__u64 addr;
 	__u32 size;
 	__u32 pad;
 };
 struct kvm_coalesced_mmio {
 	__u64 phys_addr;
 	__u32 len;
 	__u32 pad;
 	__u8  data[8];
 };
 struct kvm_coalesced_mmio_ring {
 	__u32 first, last;
 	struct kvm_coalesced_mmio coalesced_mmio[0];
 };
 #define KVM_COALESCED_MMIO_MAX \
 	((PAGE_SIZE - sizeof(struct kvm_coalesced_mmio_ring)) / \
 	 sizeof(struct kvm_coalesced_mmio))
 /* for KVM_TRANSLATE */
 struct kvm_translation {
 	/* in */
 	__u64 linear_address;
 	/* out */
 	__u64 physical_address;
 	__u8  valid;
 	__u8  writeable;
 	__u8  usermode;
 	__u8  pad[5];
 };
 /* for KVM_INTERRUPT */
 struct kvm_interrupt {
 	/* in */
 	__u32 irq;
 };
 /* for KVM_GET_DIRTY_LOG */
 struct kvm_dirty_log {
 	__u32 slot;
 	__u32 padding1;
 	union {
 		void __user *dirty_bitmap; /* one bit per page */
 		__u64 padding2;
 	};
 };
 /* for KVM_SET_SIGNAL_MASK */
 struct kvm_signal_mask {
 	__u32 len;
 	__u8  sigset[0];
 };
 /* for KVM_TPR_ACCESS_REPORTING */
 struct kvm_tpr_access_ctl {
 	__u32 enabled;
 	__u32 flags;
 	__u32 reserved[8];
 };
 /* for KVM_SET_VAPIC_ADDR */
 struct kvm_vapic_addr {
 	__u64 vapic_addr;
 };
 /* for KVM_SET_MPSTATE */
 #define KVM_MP_STATE_RUNNABLE          0
 #define KVM_MP_STATE_UNINITIALIZED     1
 #define KVM_MP_STATE_INIT_RECEIVED     2
 #define KVM_MP_STATE_HALTED            3
 #define KVM_MP_STATE_SIPI_RECEIVED     4
 struct kvm_mp_state {
 	__u32 mp_state;
 };
 struct kvm_s390_psw {
 	__u64 mask;
 	__u64 addr;
 };
 /* valid values for type in kvm_s390_interrupt */
 #define KVM_S390_SIGP_STOP		0xfffe0000u
 #define KVM_S390_PROGRAM_INT		0xfffe0001u
 #define KVM_S390_SIGP_SET_PREFIX	0xfffe0002u
 #define KVM_S390_RESTART		0xfffe0003u
 #define KVM_S390_INT_VIRTIO		0xffff2603u
 #define KVM_S390_INT_SERVICE		0xffff2401u
 #define KVM_S390_INT_EMERGENCY		0xffff1201u
 struct kvm_s390_interrupt {
 	__u32 type;
 	__u32 parm;
 	__u64 parm64;
 };
 /* for KVM_SET_GUEST_DEBUG */
 #define KVM_GUESTDBG_ENABLE		0x00000001
 #define KVM_GUESTDBG_SINGLESTEP		0x00000002
 struct kvm_guest_debug {
 	__u32 control;
 	__u32 pad;
 	struct kvm_guest_debug_arch arch;
 };
 enum {
 	kvm_ioeventfd_flag_nr_datamatch,
 	kvm_ioeventfd_flag_nr_pio,
 	kvm_ioeventfd_flag_nr_deassign,
 	kvm_ioeventfd_flag_nr_max,
 };
 #define KVM_IOEVENTFD_FLAG_DATAMATCH (1 << kvm_ioeventfd_flag_nr_datamatch)
 #define KVM_IOEVENTFD_FLAG_PIO       (1 << kvm_ioeventfd_flag_nr_pio)
 #define KVM_IOEVENTFD_FLAG_DEASSIGN  (1 << kvm_ioeventfd_flag_nr_deassign)
 #define KVM_IOEVENTFD_VALID_FLAG_MASK  ((1 << kvm_ioeventfd_flag_nr_max) - 1)
 struct kvm_ioeventfd {
 	__u64 datamatch;
 	__u64 addr;        /* legal pio/mmio address */
 	__u32 len;         /* 1, 2, 4, or 8 bytes    */
 	__s32 fd;
 	__u32 flags;
 	__u8  pad[36];
 };
 /* for KVM_ENABLE_CAP */
 struct kvm_enable_cap {
 	/* in */
 	__u32 cap;
 	__u32 flags;
 	__u64 args[4];
 	__u8  pad[64];
 };
 #define KVMIO 0xAE
 /*
  * ioctls for /dev/kvm fds:
  */
 #define KVM_GET_API_VERSION       _IO(KVMIO,   0x00)
 #define KVM_CREATE_VM             _IO(KVMIO,   0x01) /* returns a VM fd */
 #define KVM_GET_MSR_INDEX_LIST    _IOWR(KVMIO, 0x02, struct kvm_msr_list)
 #define KVM_S390_ENABLE_SIE       _IO(KVMIO,   0x06)
 /*
  * Check if a kvm extension is available.  Argument is extension number,
  * return is 1 (yes) or 0 (no, sorry).
  */
 #define KVM_CHECK_EXTENSION       _IO(KVMIO,   0x03)
 /*
  * Get size for mmap(vcpu_fd)
  */
 #define KVM_GET_VCPU_MMAP_SIZE    _IO(KVMIO,   0x04) /* in bytes */
 #define KVM_GET_SUPPORTED_CPUID   _IOWR(KVMIO, 0x05, struct kvm_cpuid2)
 #define KVM_TRACE_ENABLE          __KVM_DEPRECATED_MAIN_W_0x06
 #define KVM_TRACE_PAUSE           __KVM_DEPRECATED_MAIN_0x07
 #define KVM_TRACE_DISABLE         __KVM_DEPRECATED_MAIN_0x08
 /*
  * Extension capability list.
  */
 #define KVM_CAP_IRQCHIP	  0
 #define KVM_CAP_HLT	  1
 #define KVM_CAP_MMU_SHADOW_CACHE_CONTROL 2
 #define KVM_CAP_USER_MEMORY 3
 #define KVM_CAP_SET_TSS_ADDR 4
 #define KVM_CAP_VAPIC 6
 #define KVM_CAP_EXT_CPUID 7
 #define KVM_CAP_CLOCKSOURCE 8
 #define KVM_CAP_NR_VCPUS 9       /* returns max vcpus per vm */
 #define KVM_CAP_NR_MEMSLOTS 10   /* returns max memory slots per vm */
 #define KVM_CAP_PIT 11
 #define KVM_CAP_NOP_IO_DELAY 12
 #define KVM_CAP_PV_MMU 13
 #define KVM_CAP_MP_STATE 14
 #define KVM_CAP_COALESCED_MMIO 15
 #define KVM_CAP_SYNC_MMU 16  /* Changes to host mmap are reflected in guest */
 #ifdef __KVM_HAVE_DEVICE_ASSIGNMENT
 #define KVM_CAP_DEVICE_ASSIGNMENT 17
 #endif
 #define KVM_CAP_IOMMU 18
 #ifdef __KVM_HAVE_MSI
 #define KVM_CAP_DEVICE_MSI 20
 #endif
 /* Bug in KVM_SET_USER_MEMORY_REGION fixed: */
 #define KVM_CAP_DESTROY_MEMORY_REGION_WORKS 21
 #ifdef __KVM_HAVE_USER_NMI
 #define KVM_CAP_USER_NMI 22
 #endif
 #ifdef __KVM_HAVE_GUEST_DEBUG
 #define KVM_CAP_SET_GUEST_DEBUG 23
 #endif
 #ifdef __KVM_HAVE_PIT
 #define KVM_CAP_REINJECT_CONTROL 24
 #endif
 #ifdef __KVM_HAVE_IOAPIC
 #define KVM_CAP_IRQ_ROUTING 25
 #endif
 #define KVM_CAP_IRQ_INJECT_STATUS 26
 #ifdef __KVM_HAVE_DEVICE_ASSIGNMENT
 #define KVM_CAP_DEVICE_DEASSIGNMENT 27
 #endif
 #ifdef __KVM_HAVE_MSIX
 #define KVM_CAP_DEVICE_MSIX 28
 #endif
 #define KVM_CAP_ASSIGN_DEV_IRQ 29
 /* Another bug in KVM_SET_USER_MEMORY_REGION fixed: */
 #define KVM_CAP_JOIN_MEMORY_REGIONS_WORKS 30
 #ifdef __KVM_HAVE_MCE
 #define KVM_CAP_MCE 31
 #endif
 #define KVM_CAP_IRQFD 32
 #ifdef __KVM_HAVE_PIT
 #define KVM_CAP_PIT2 33
 #endif
 #define KVM_CAP_SET_BOOT_CPU_ID 34
 #ifdef __KVM_HAVE_PIT_STATE2
 #define KVM_CAP_PIT_STATE2 35
 #endif
 #define KVM_CAP_IOEVENTFD 36
 #define KVM_CAP_SET_IDENTITY_MAP_ADDR 37
 #ifdef __KVM_HAVE_XEN_HVM
 #define KVM_CAP_XEN_HVM 38
 #endif
 #define KVM_CAP_ADJUST_CLOCK 39
 #define KVM_CAP_INTERNAL_ERROR_DATA 40
 #ifdef __KVM_HAVE_VCPU_EVENTS
 #define KVM_CAP_VCPU_EVENTS 41
 #endif
 #define KVM_CAP_S390_PSW 42
 #define KVM_CAP_PPC_SEGSTATE 43
 #define KVM_CAP_HYPERV 44
 #define KVM_CAP_HYPERV_VAPIC 45
 #define KVM_CAP_HYPERV_SPIN 46
 #define KVM_CAP_PCI_SEGMENT 47
 #define KVM_CAP_PPC_PAIRED_SINGLES 48
 #define KVM_CAP_INTR_SHADOW 49
 #ifdef __KVM_HAVE_DEBUGREGS
 #define KVM_CAP_DEBUGREGS 50
 #endif
 #define KVM_CAP_X86_ROBUST_SINGLESTEP 51
 #define KVM_CAP_PPC_OSI 52
 #define KVM_CAP_PPC_UNSET_IRQ 53
 #define KVM_CAP_ENABLE_CAP 54
 #ifdef __KVM_HAVE_XSAVE
 #define KVM_CAP_XSAVE 55
 #endif
 #ifdef __KVM_HAVE_XCRS
 #define KVM_CAP_XCRS 56
 #endif
 #ifdef KVM_CAP_IRQ_ROUTING
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
 	__u32 pin;
 };
 struct kvm_irq_routing_msi {
 	__u32 address_lo;
 	__u32 address_hi;
 	__u32 data;
 	__u32 pad;
 };
 /* gsi routing entry types */
 #define KVM_IRQ_ROUTING_IRQCHIP 1
 #define KVM_IRQ_ROUTING_MSI 2
 struct kvm_irq_routing_entry {
 	__u32 gsi;
 	__u32 type;
 	__u32 flags;
 	__u32 pad;
 	union {
 		struct kvm_irq_routing_irqchip irqchip;
 		struct kvm_irq_routing_msi msi;
 		__u32 pad[8];
 	} u;
 };
 struct kvm_irq_routing {
 	__u32 nr;
 	__u32 flags;
 	struct kvm_irq_routing_entry entries[0];
 };
 #endif
 #ifdef KVM_CAP_MCE
 /* x86 MCE */
 struct kvm_x86_mce {
 	__u64 status;
 	__u64 addr;
 	__u64 misc;
 	__u64 mcg_status;
 	__u8 bank;
 	__u8 pad1[7];
 	__u64 pad2[3];
 };
 #endif
 #ifdef KVM_CAP_XEN_HVM
 struct kvm_xen_hvm_config {
 	__u32 flags;
 	__u32 msr;
 	__u64 blob_addr_32;
 	__u64 blob_addr_64;
 	__u8 blob_size_32;
 	__u8 blob_size_64;
 	__u8 pad2[30];
 };
 #endif
 #define KVM_IRQFD_FLAG_DEASSIGN (1 << 0)
 struct kvm_irqfd {
 	__u32 fd;
 	__u32 gsi;
 	__u32 flags;
 	__u8  pad[20];
 };
 struct kvm_clock_data {
 	__u64 clock;
 	__u32 flags;
 	__u32 pad[9];
 };
 /*
  * ioctls for VM fds
  */
 #define KVM_SET_MEMORY_REGION     _IOW(KVMIO,  0x40, struct kvm_memory_region)
 /*
  * KVM_CREATE_VCPU receives as a parameter the vcpu slot, and returns
  * a vcpu fd.
  */
 #define KVM_CREATE_VCPU           _IO(KVMIO,   0x41)
 #define KVM_GET_DIRTY_LOG         _IOW(KVMIO,  0x42, struct kvm_dirty_log)
+/* KVM_SET_MEMORY_ALIAS is obsolete: */
 #define KVM_SET_MEMORY_ALIAS      _IOW(KVMIO,  0x43, struct kvm_memory_alias)
 #define KVM_SET_NR_MMU_PAGES      _IO(KVMIO,   0x44)
 #define KVM_GET_NR_MMU_PAGES      _IO(KVMIO,   0x45)
 #define KVM_SET_USER_MEMORY_REGION _IOW(KVMIO, 0x46, \
 					struct kvm_userspace_memory_region)
 #define KVM_SET_TSS_ADDR          _IO(KVMIO,   0x47)
 #define KVM_SET_IDENTITY_MAP_ADDR _IOW(KVMIO,  0x48, __u64)
 /* Device model IOC */
 #define KVM_CREATE_IRQCHIP        _IO(KVMIO,   0x60)
 #define KVM_IRQ_LINE              _IOW(KVMIO,  0x61, struct kvm_irq_level)
 #define KVM_GET_IRQCHIP           _IOWR(KVMIO, 0x62, struct kvm_irqchip)
 #define KVM_SET_IRQCHIP           _IOR(KVMIO,  0x63, struct kvm_irqchip)
 #define KVM_CREATE_PIT            _IO(KVMIO,   0x64)
 #define KVM_GET_PIT               _IOWR(KVMIO, 0x65, struct kvm_pit_state)
 #define KVM_SET_PIT               _IOR(KVMIO,  0x66, struct kvm_pit_state)
 #define KVM_IRQ_LINE_STATUS       _IOWR(KVMIO, 0x67, struct kvm_irq_level)
 #define KVM_REGISTER_COALESCED_MMIO \
 			_IOW(KVMIO,  0x67, struct kvm_coalesced_mmio_zone)
 #define KVM_UNREGISTER_COALESCED_MMIO \
 			_IOW(KVMIO,  0x68, struct kvm_coalesced_mmio_zone)
 #define KVM_ASSIGN_PCI_DEVICE     _IOR(KVMIO,  0x69, \
 				       struct kvm_assigned_pci_dev)
 #define KVM_SET_GSI_ROUTING       _IOW(KVMIO,  0x6a, struct kvm_irq_routing)
 /* deprecated, replaced by KVM_ASSIGN_DEV_IRQ */
 #define KVM_ASSIGN_IRQ            __KVM_DEPRECATED_VM_R_0x70
 #define KVM_ASSIGN_DEV_IRQ        _IOW(KVMIO,  0x70, struct kvm_assigned_irq)
 #define KVM_REINJECT_CONTROL      _IO(KVMIO,   0x71)
 #define KVM_DEASSIGN_PCI_DEVICE   _IOW(KVMIO,  0x72, \
 				       struct kvm_assigned_pci_dev)
 #define KVM_ASSIGN_SET_MSIX_NR    _IOW(KVMIO,  0x73, \
 				       struct kvm_assigned_msix_nr)
 #define KVM_ASSIGN_SET_MSIX_ENTRY _IOW(KVMIO,  0x74, \
 				       struct kvm_assigned_msix_entry)
 #define KVM_DEASSIGN_DEV_IRQ      _IOW(KVMIO,  0x75, struct kvm_assigned_irq)
 #define KVM_IRQFD                 _IOW(KVMIO,  0x76, struct kvm_irqfd)
 #define KVM_CREATE_PIT2		  _IOW(KVMIO,  0x77, struct kvm_pit_config)
 #define KVM_SET_BOOT_CPU_ID       _IO(KVMIO,   0x78)
 #define KVM_IOEVENTFD             _IOW(KVMIO,  0x79, struct kvm_ioeventfd)
 #define KVM_XEN_HVM_CONFIG        _IOW(KVMIO,  0x7a, struct kvm_xen_hvm_config)
 #define KVM_SET_CLOCK             _IOW(KVMIO,  0x7b, struct kvm_clock_data)
 #define KVM_GET_CLOCK             _IOR(KVMIO,  0x7c, struct kvm_clock_data)
 /* Available with KVM_CAP_PIT_STATE2 */
 #define KVM_GET_PIT2              _IOR(KVMIO,  0x9f, struct kvm_pit_state2)
 #define KVM_SET_PIT2              _IOW(KVMIO,  0xa0, struct kvm_pit_state2)
 /*
  * ioctls for vcpu fds
  */
 #define KVM_RUN                   _IO(KVMIO,   0x80)
 #define KVM_GET_REGS              _IOR(KVMIO,  0x81, struct kvm_regs)
 #define KVM_SET_REGS              _IOW(KVMIO,  0x82, struct kvm_regs)
 #define KVM_GET_SREGS             _IOR(KVMIO,  0x83, struct kvm_sregs)
 #define KVM_SET_SREGS             _IOW(KVMIO,  0x84, struct kvm_sregs)
 #define KVM_TRANSLATE             _IOWR(KVMIO, 0x85, struct kvm_translation)
 #define KVM_INTERRUPT             _IOW(KVMIO,  0x86, struct kvm_interrupt)
 /* KVM_DEBUG_GUEST is no longer supported, use KVM_SET_GUEST_DEBUG instead */
 #define KVM_DEBUG_GUEST           __KVM_DEPRECATED_VCPU_W_0x87
 #define KVM_GET_MSRS              _IOWR(KVMIO, 0x88, struct kvm_msrs)
 #define KVM_SET_MSRS              _IOW(KVMIO,  0x89, struct kvm_msrs)
 #define KVM_SET_CPUID             _IOW(KVMIO,  0x8a, struct kvm_cpuid)
 #define KVM_SET_SIGNAL_MASK       _IOW(KVMIO,  0x8b, struct kvm_signal_mask)
 #define KVM_GET_FPU               _IOR(KVMIO,  0x8c, struct kvm_fpu)
 #define KVM_SET_FPU               _IOW(KVMIO,  0x8d, struct kvm_fpu)
 #define KVM_GET_LAPIC             _IOR(KVMIO,  0x8e, struct kvm_lapic_state)
 #define KVM_SET_LAPIC             _IOW(KVMIO,  0x8f, struct kvm_lapic_state)
 #define KVM_SET_CPUID2            _IOW(KVMIO,  0x90, struct kvm_cpuid2)
 #define KVM_GET_CPUID2            _IOWR(KVMIO, 0x91, struct kvm_cpuid2)
 /* Available with KVM_CAP_VAPIC */
 #define KVM_TPR_ACCESS_REPORTING  _IOWR(KVMIO, 0x92, struct kvm_tpr_access_ctl)
 /* Available with KVM_CAP_VAPIC */
 #define KVM_SET_VAPIC_ADDR        _IOW(KVMIO,  0x93, struct kvm_vapic_addr)
 /* valid for virtual machine (for floating interrupt)_and_ vcpu */
 #define KVM_S390_INTERRUPT        _IOW(KVMIO,  0x94, struct kvm_s390_interrupt)
 /* store status for s390 */
 #define KVM_S390_STORE_STATUS_NOADDR    (-1ul)
 #define KVM_S390_STORE_STATUS_PREFIXED  (-2ul)
 #define KVM_S390_STORE_STATUS	  _IOW(KVMIO,  0x95, unsigned long)
 /* initial ipl psw for s390 */
 #define KVM_S390_SET_INITIAL_PSW  _IOW(KVMIO,  0x96, struct kvm_s390_psw)
 /* initial reset for s390 */
 #define KVM_S390_INITIAL_RESET    _IO(KVMIO,   0x97)
 #define KVM_GET_MP_STATE          _IOR(KVMIO,  0x98, struct kvm_mp_state)
 #define KVM_SET_MP_STATE          _IOW(KVMIO,  0x99, struct kvm_mp_state)
 /* Available with KVM_CAP_NMI */
 #define KVM_NMI                   _IO(KVMIO,   0x9a)
 /* Available with KVM_CAP_SET_GUEST_DEBUG */
 #define KVM_SET_GUEST_DEBUG       _IOW(KVMIO,  0x9b, struct kvm_guest_debug)
 /* MCE for x86 */
 #define KVM_X86_SETUP_MCE         _IOW(KVMIO,  0x9c, __u64)
 #define KVM_X86_GET_MCE_CAP_SUPPORTED _IOR(KVMIO,  0x9d, __u64)
 #define KVM_X86_SET_MCE           _IOW(KVMIO,  0x9e, struct kvm_x86_mce)
 /* IA64 stack access */
 #define KVM_IA64_VCPU_GET_STACK   _IOR(KVMIO,  0x9a, void *)
 #define KVM_IA64_VCPU_SET_STACK   _IOW(KVMIO,  0x9b, void *)
 /* Available with KVM_CAP_VCPU_EVENTS */
 #define KVM_GET_VCPU_EVENTS       _IOR(KVMIO,  0x9f, struct kvm_vcpu_events)
 #define KVM_SET_VCPU_EVENTS       _IOW(KVMIO,  0xa0, struct kvm_vcpu_events)
 /* Available with KVM_CAP_DEBUGREGS */
 #define KVM_GET_DEBUGREGS         _IOR(KVMIO,  0xa1, struct kvm_debugregs)
 #define KVM_SET_DEBUGREGS         _IOW(KVMIO,  0xa2, struct kvm_debugregs)
 #define KVM_ENABLE_CAP            _IOW(KVMIO,  0xa3, struct kvm_enable_cap)
 /* Available with KVM_CAP_XSAVE */
 #define KVM_GET_XSAVE		  _IOR(KVMIO,  0xa4, struct kvm_xsave)
 #define KVM_SET_XSAVE		  _IOW(KVMIO,  0xa5, struct kvm_xsave)
 /* Available with KVM_CAP_XCRS */
 #define KVM_GET_XCRS		  _IOR(KVMIO,  0xa6, struct kvm_xcrs)
 #define KVM_SET_XCRS		  _IOW(KVMIO,  0xa7, struct kvm_xcrs)
 #define KVM_DEV_ASSIGN_ENABLE_IOMMU	(1 << 0)
 struct kvm_assigned_pci_dev {
 	__u32 assigned_dev_id;
 	__u32 busnr;
 	__u32 devfn;
 	__u32 flags;
 	__u32 segnr;
 	union {
 		__u32 reserved[11];
 	};
 };
 #define KVM_DEV_IRQ_HOST_INTX    (1 << 0)
 #define KVM_DEV_IRQ_HOST_MSI     (1 << 1)
 #define KVM_DEV_IRQ_HOST_MSIX    (1 << 2)
 #define KVM_DEV_IRQ_GUEST_INTX   (1 << 8)
 #define KVM_DEV_IRQ_GUEST_MSI    (1 << 9)
 #define KVM_DEV_IRQ_GUEST_MSIX   (1 << 10)
 #define KVM_DEV_IRQ_HOST_MASK	 0x00ff
 #define KVM_DEV_IRQ_GUEST_MASK   0xff00
 struct kvm_assigned_irq {
 	__u32 assigned_dev_id;
 	__u32 host_irq;
 	__u32 guest_irq;
 	__u32 flags;
 	union {
 		struct {
 			__u32 addr_lo;
 			__u32 addr_hi;
 			__u32 data;
 		} guest_msi;
 		__u32 reserved[12];
 	};
 };
 struct kvm_assigned_msix_nr {
 	__u32 assigned_dev_id;
 	__u16 entry_nr;
 	__u16 padding;
 };
 #define KVM_MAX_MSIX_PER_DEV		256
 struct kvm_assigned_msix_entry {
 	__u32 assigned_dev_id;
 	__u32 gsi;
 	__u16 entry; /* The index of entry in the MSI-X table */
 	__u16 padding[3];
 };
 #endif /* __LINUX_KVM_H */

include/linux/kvm_host.h

Diff comments View file @ a1f4d39

 #ifndef __KVM_HOST_H
 #define __KVM_HOST_H
 /*
  * This work is licensed under the terms of the GNU GPL, version 2.  See
  * the COPYING file in the top-level directory.
  */
 #include <linux/types.h>
 #include <linux/hardirq.h>
 #include <linux/list.h>
 #include <linux/mutex.h>
 #include <linux/spinlock.h>
 #include <linux/signal.h>
 #include <linux/sched.h>
 #include <linux/mm.h>
 #include <linux/preempt.h>
 #include <linux/msi.h>
 #include <asm/signal.h>
 #include <linux/kvm.h>
 #include <linux/kvm_para.h>
 #include <linux/kvm_types.h>
 #include <asm/kvm_host.h>
 /*
  * vcpu->requests bit members
  */
 #define KVM_REQ_TLB_FLUSH          0
 #define KVM_REQ_MIGRATE_TIMER      1
 #define KVM_REQ_REPORT_TPR_ACCESS  2
 #define KVM_REQ_MMU_RELOAD         3
 #define KVM_REQ_TRIPLE_FAULT       4
 #define KVM_REQ_PENDING_TIMER      5
 #define KVM_REQ_UNHALT             6
 #define KVM_REQ_MMU_SYNC           7
 #define KVM_REQ_KVMCLOCK_UPDATE    8
 #define KVM_REQ_KICK               9
 #define KVM_REQ_DEACTIVATE_FPU    10
 #define KVM_USERSPACE_IRQ_SOURCE_ID	0
 struct kvm;
 struct kvm_vcpu;
 extern struct kmem_cache *kvm_vcpu_cache;
 /*
  * It would be nice to use something smarter than a linear search, TBD...
  * Thankfully we dont expect many devices to register (famous last words :),
  * so until then it will suffice.  At least its abstracted so we can change
  * in one place.
  */
 struct kvm_io_bus {
 	int                   dev_count;
 #define NR_IOBUS_DEVS 200
 	struct kvm_io_device *devs[NR_IOBUS_DEVS];
 };
 enum kvm_bus {
 	KVM_MMIO_BUS,
 	KVM_PIO_BUS,
 	KVM_NR_BUSES
 };
 int kvm_io_bus_write(struct kvm *kvm, enum kvm_bus bus_idx, gpa_t addr,
 		     int len, const void *val);
 int kvm_io_bus_read(struct kvm *kvm, enum kvm_bus bus_idx, gpa_t addr, int len,
 		    void *val);
 int kvm_io_bus_register_dev(struct kvm *kvm, enum kvm_bus bus_idx,
 			    struct kvm_io_device *dev);
 int kvm_io_bus_unregister_dev(struct kvm *kvm, enum kvm_bus bus_idx,
 			      struct kvm_io_device *dev);
 struct kvm_vcpu {
 	struct kvm *kvm;
 #ifdef CONFIG_PREEMPT_NOTIFIERS
 	struct preempt_notifier preempt_notifier;
 #endif
 	int vcpu_id;
 	struct mutex mutex;
 	int   cpu;
 	atomic_t guest_mode;
 	struct kvm_run *run;
 	unsigned long requests;
 	unsigned long guest_debug;
 	int srcu_idx;
 	int fpu_active;
 	int guest_fpu_loaded, guest_xcr0_loaded;
 	wait_queue_head_t wq;
 	int sigset_active;
 	sigset_t sigset;
 	struct kvm_vcpu_stat stat;
 #ifdef CONFIG_HAS_IOMEM
 	int mmio_needed;
 	int mmio_read_completed;
 	int mmio_is_write;
 	int mmio_size;
 	unsigned char mmio_data[8];
 	gpa_t mmio_phys_addr;
 #endif
 	struct kvm_vcpu_arch arch;
 };
 /*
  * Some of the bitops functions do not support too long bitmaps.
  * This number must be determined not to exceed such limits.
  */
 #define KVM_MEM_MAX_NR_PAGES ((1UL << 31) - 1)
 struct kvm_memory_slot {
 	gfn_t base_gfn;
 	unsigned long npages;
 	unsigned long flags;
 	unsigned long *rmap;
 	unsigned long *dirty_bitmap;
 	struct {
 		unsigned long rmap_pde;
 		int write_count;
 	} *lpage_info[KVM_NR_PAGE_SIZES - 1];
 	unsigned long userspace_addr;
 	int user_alloc;
 };
 static inline unsigned long kvm_dirty_bitmap_bytes(struct kvm_memory_slot *memslot)
 {
 	return ALIGN(memslot->npages, BITS_PER_LONG) / 8;
 }
 struct kvm_kernel_irq_routing_entry {
 	u32 gsi;
 	u32 type;
 	int (*set)(struct kvm_kernel_irq_routing_entry *e,
 		   struct kvm *kvm, int irq_source_id, int level);
 	union {
 		struct {
 			unsigned irqchip;
 			unsigned pin;
 		} irqchip;
 		struct msi_msg msi;
 	};
 	struct hlist_node link;
 };
 #ifdef __KVM_HAVE_IOAPIC
 struct kvm_irq_routing_table {
 	int chip[KVM_NR_IRQCHIPS][KVM_IOAPIC_NUM_PINS];
 	struct kvm_kernel_irq_routing_entry *rt_entries;
 	u32 nr_rt_entries;
 	/*
 	 * Array indexed by gsi. Each entry contains list of irq chips
 	 * the gsi is connected to.
 	 */
 	struct hlist_head map[0];
 };
 #else
 struct kvm_irq_routing_table {};
 #endif
 struct kvm_memslots {
 	int nmemslots;
 	struct kvm_memory_slot memslots[KVM_MEMORY_SLOTS +
 					KVM_PRIVATE_MEM_SLOTS];
 };
 struct kvm {
 	spinlock_t mmu_lock;
 	raw_spinlock_t requests_lock;
 	struct mutex slots_lock;
 	struct mm_struct *mm; /* userspace tied to this vm */
 	struct kvm_memslots *memslots;
 	struct srcu_struct srcu;
 #ifdef CONFIG_KVM_APIC_ARCHITECTURE
 	u32 bsp_vcpu_id;
 	struct kvm_vcpu *bsp_vcpu;
 #endif
 	struct kvm_vcpu *vcpus[KVM_MAX_VCPUS];
 	atomic_t online_vcpus;
 	struct list_head vm_list;
 	struct mutex lock;
 	struct kvm_io_bus *buses[KVM_NR_BUSES];
 #ifdef CONFIG_HAVE_KVM_EVENTFD
 	struct {
 		spinlock_t        lock;
 		struct list_head  items;
 	} irqfds;
 	struct list_head ioeventfds;
 #endif
 	struct kvm_vm_stat stat;
 	struct kvm_arch arch;
 	atomic_t users_count;
 #ifdef KVM_COALESCED_MMIO_PAGE_OFFSET
 	struct kvm_coalesced_mmio_dev *coalesced_mmio_dev;
 	struct kvm_coalesced_mmio_ring *coalesced_mmio_ring;
 #endif
 	struct mutex irq_lock;
 #ifdef CONFIG_HAVE_KVM_IRQCHIP
 	struct kvm_irq_routing_table *irq_routing;
 	struct hlist_head mask_notifier_list;
 	struct hlist_head irq_ack_notifier_list;
 #endif
 #ifdef KVM_ARCH_WANT_MMU_NOTIFIER
 	struct mmu_notifier mmu_notifier;
 	unsigned long mmu_notifier_seq;
 	long mmu_notifier_count;
 #endif
 };
 /* The guest did something we don't support. */
 #define pr_unimpl(vcpu, fmt, ...)					\
  do {									\
 	if (printk_ratelimit())						\
 		printk(KERN_ERR "kvm: %i: cpu%i " fmt,			\
 		       current->tgid, (vcpu)->vcpu_id , ## __VA_ARGS__); \
  } while (0)
 #define kvm_printf(kvm, fmt ...) printk(KERN_DEBUG fmt)
 #define vcpu_printf(vcpu, fmt...) kvm_printf(vcpu->kvm, fmt)
 static inline struct kvm_vcpu *kvm_get_vcpu(struct kvm *kvm, int i)
 {
 	smp_rmb();
 	return kvm->vcpus[i];
 }
 #define kvm_for_each_vcpu(idx, vcpup, kvm) \
 	for (idx = 0, vcpup = kvm_get_vcpu(kvm, idx); \
 	     idx < atomic_read(&kvm->online_vcpus) && vcpup; \
 	     vcpup = kvm_get_vcpu(kvm, ++idx))
 int kvm_vcpu_init(struct kvm_vcpu *vcpu, struct kvm *kvm, unsigned id);
 void kvm_vcpu_uninit(struct kvm_vcpu *vcpu);
 void vcpu_load(struct kvm_vcpu *vcpu);
 void vcpu_put(struct kvm_vcpu *vcpu);
 int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align,
 		  struct module *module);
 void kvm_exit(void);
 void kvm_get_kvm(struct kvm *kvm);
 void kvm_put_kvm(struct kvm *kvm);
 static inline struct kvm_memslots *kvm_memslots(struct kvm *kvm)
 {
 	return rcu_dereference_check(kvm->memslots,
 			srcu_read_lock_held(&kvm->srcu)
 			|| lockdep_is_held(&kvm->slots_lock));
 }
 #define HPA_MSB ((sizeof(hpa_t) * 8) - 1)
 #define HPA_ERR_MASK ((hpa_t)1 << HPA_MSB)
 static inline int is_error_hpa(hpa_t hpa) { return hpa >> HPA_MSB; }
 extern struct page *bad_page;
 extern pfn_t bad_pfn;
 int is_error_page(struct page *page);
 int is_error_pfn(pfn_t pfn);
 int is_hwpoison_pfn(pfn_t pfn);
 int kvm_is_error_hva(unsigned long addr);
 int kvm_set_memory_region(struct kvm *kvm,
 			  struct kvm_userspace_memory_region *mem,
 			  int user_alloc);
 int __kvm_set_memory_region(struct kvm *kvm,
 			    struct kvm_userspace_memory_region *mem,
 			    int user_alloc);
 int kvm_arch_prepare_memory_region(struct kvm *kvm,
 				struct kvm_memory_slot *memslot,
 				struct kvm_memory_slot old,
 				struct kvm_userspace_memory_region *mem,
 				int user_alloc);
 void kvm_arch_commit_memory_region(struct kvm *kvm,
 				struct kvm_userspace_memory_region *mem,
 				struct kvm_memory_slot old,
 				int user_alloc);
 void kvm_disable_largepages(void);
 void kvm_arch_flush_shadow(struct kvm *kvm);
-gfn_t unalias_gfn(struct kvm *kvm, gfn_t gfn);
-gfn_t unalias_gfn_instantiation(struct kvm *kvm, gfn_t gfn);
 struct page *gfn_to_page(struct kvm *kvm, gfn_t gfn);
 unsigned long gfn_to_hva(struct kvm *kvm, gfn_t gfn);
 void kvm_release_page_clean(struct page *page);
 void kvm_release_page_dirty(struct page *page);
 void kvm_set_page_dirty(struct page *page);
 void kvm_set_page_accessed(struct page *page);
 pfn_t gfn_to_pfn(struct kvm *kvm, gfn_t gfn);
 pfn_t gfn_to_pfn_memslot(struct kvm *kvm,
 			 struct kvm_memory_slot *slot, gfn_t gfn);
 int memslot_id(struct kvm *kvm, gfn_t gfn);
 void kvm_release_pfn_dirty(pfn_t);
 void kvm_release_pfn_clean(pfn_t pfn);
 void kvm_set_pfn_dirty(pfn_t pfn);
 void kvm_set_pfn_accessed(pfn_t pfn);
 void kvm_get_pfn(pfn_t pfn);
 int kvm_read_guest_page(struct kvm *kvm, gfn_t gfn, void *data, int offset,
 			int len);
 int kvm_read_guest_atomic(struct kvm *kvm, gpa_t gpa, void *data,
 			  unsigned long len);
 int kvm_read_guest(struct kvm *kvm, gpa_t gpa, void *data, unsigned long len);
 int kvm_write_guest_page(struct kvm *kvm, gfn_t gfn, const void *data,
 			 int offset, int len);
 int kvm_write_guest(struct kvm *kvm, gpa_t gpa, const void *data,
 		    unsigned long len);
 int kvm_clear_guest_page(struct kvm *kvm, gfn_t gfn, int offset, int len);
 int kvm_clear_guest(struct kvm *kvm, gpa_t gpa, unsigned long len);
 struct kvm_memory_slot *gfn_to_memslot(struct kvm *kvm, gfn_t gfn);
 int kvm_is_visible_gfn(struct kvm *kvm, gfn_t gfn);
 unsigned long kvm_host_page_size(struct kvm *kvm, gfn_t gfn);
 void mark_page_dirty(struct kvm *kvm, gfn_t gfn);
 void kvm_vcpu_block(struct kvm_vcpu *vcpu);
 void kvm_vcpu_on_spin(struct kvm_vcpu *vcpu);
 void kvm_resched(struct kvm_vcpu *vcpu);
 void kvm_load_guest_fpu(struct kvm_vcpu *vcpu);
 void kvm_put_guest_fpu(struct kvm_vcpu *vcpu);
 void kvm_flush_remote_tlbs(struct kvm *kvm);
 void kvm_reload_remote_mmus(struct kvm *kvm);
 long kvm_arch_dev_ioctl(struct file *filp,
 			unsigned int ioctl, unsigned long arg);
 long kvm_arch_vcpu_ioctl(struct file *filp,
 			 unsigned int ioctl, unsigned long arg);
 int kvm_dev_ioctl_check_extension(long ext);
 int kvm_get_dirty_log(struct kvm *kvm,
 			struct kvm_dirty_log *log, int *is_dirty);
 int kvm_vm_ioctl_get_dirty_log(struct kvm *kvm,
 				struct kvm_dirty_log *log);
 int kvm_vm_ioctl_set_memory_region(struct kvm *kvm,
 				   struct
 				   kvm_userspace_memory_region *mem,
 				   int user_alloc);
 long kvm_arch_vm_ioctl(struct file *filp,
 		       unsigned int ioctl, unsigned long arg);
 int kvm_arch_vcpu_ioctl_get_fpu(struct kvm_vcpu *vcpu, struct kvm_fpu *fpu);
 int kvm_arch_vcpu_ioctl_set_fpu(struct kvm_vcpu *vcpu, struct kvm_fpu *fpu);
 int kvm_arch_vcpu_ioctl_translate(struct kvm_vcpu *vcpu,
 				    struct kvm_translation *tr);
 int kvm_arch_vcpu_ioctl_get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs);
 int kvm_arch_vcpu_ioctl_set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs);
 int kvm_arch_vcpu_ioctl_get_sregs(struct kvm_vcpu *vcpu,
 				  struct kvm_sregs *sregs);
 int kvm_arch_vcpu_ioctl_set_sregs(struct kvm_vcpu *vcpu,
 				  struct kvm_sregs *sregs);
 int kvm_arch_vcpu_ioctl_get_mpstate(struct kvm_vcpu *vcpu,
 				    struct kvm_mp_state *mp_state);
 int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu,
 				    struct kvm_mp_state *mp_state);
 int kvm_arch_vcpu_ioctl_set_guest_debug(struct kvm_vcpu *vcpu,
 					struct kvm_guest_debug *dbg);
 int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run);
 int kvm_arch_init(void *opaque);
 void kvm_arch_exit(void);
 int kvm_arch_vcpu_init(struct kvm_vcpu *vcpu);
 void kvm_arch_vcpu_uninit(struct kvm_vcpu *vcpu);
 void kvm_arch_vcpu_free(struct kvm_vcpu *vcpu);
 void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu);
 void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu);
 struct kvm_vcpu *kvm_arch_vcpu_create(struct kvm *kvm, unsigned int id);
 int kvm_arch_vcpu_setup(struct kvm_vcpu *vcpu);
 void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu);
 int kvm_arch_vcpu_reset(struct kvm_vcpu *vcpu);
 int kvm_arch_hardware_enable(void *garbage);
 void kvm_arch_hardware_disable(void *garbage);
 int kvm_arch_hardware_setup(void);
 void kvm_arch_hardware_unsetup(void);
 void kvm_arch_check_processor_compat(void *rtn);
 int kvm_arch_vcpu_runnable(struct kvm_vcpu *vcpu);
 void kvm_free_physmem(struct kvm *kvm);
 struct  kvm *kvm_arch_create_vm(void);
 void kvm_arch_destroy_vm(struct kvm *kvm);
 void kvm_free_all_assigned_devices(struct kvm *kvm);
 void kvm_arch_sync_events(struct kvm *kvm);
 int kvm_cpu_has_pending_timer(struct kvm_vcpu *vcpu);
 void kvm_vcpu_kick(struct kvm_vcpu *vcpu);
 int kvm_is_mmio_pfn(pfn_t pfn);
 struct kvm_irq_ack_notifier {
 	struct hlist_node link;
 	unsigned gsi;
 	void (*irq_acked)(struct kvm_irq_ack_notifier *kian);
 };
 #define KVM_ASSIGNED_MSIX_PENDING		0x1
 struct kvm_guest_msix_entry {
 	u32 vector;
 	u16 entry;
 	u16 flags;
 };
 struct kvm_assigned_dev_kernel {
 	struct kvm_irq_ack_notifier ack_notifier;
 	struct work_struct interrupt_work;
 	struct list_head list;
 	int assigned_dev_id;
 	int host_segnr;
 	int host_busnr;
 	int host_devfn;
 	unsigned int entries_nr;
 	int host_irq;
 	bool host_irq_disabled;
 	struct msix_entry *host_msix_entries;
 	int guest_irq;
 	struct kvm_guest_msix_entry *guest_msix_entries;
 	unsigned long irq_requested_type;
 	int irq_source_id;
 	int flags;
 	struct pci_dev *dev;
 	struct kvm *kvm;
 	spinlock_t assigned_dev_lock;
 };
 struct kvm_irq_mask_notifier {
 	void (*func)(struct kvm_irq_mask_notifier *kimn, bool masked);
 	int irq;
 	struct hlist_node link;
 };
 void kvm_register_irq_mask_notifier(struct kvm *kvm, int irq,
 				    struct kvm_irq_mask_notifier *kimn);
 void kvm_unregister_irq_mask_notifier(struct kvm *kvm, int irq,
 				      struct kvm_irq_mask_notifier *kimn);
 void kvm_fire_mask_notifiers(struct kvm *kvm, int irq, bool mask);
 #ifdef __KVM_HAVE_IOAPIC
 void kvm_get_intr_delivery_bitmask(struct kvm_ioapic *ioapic,
 				   union kvm_ioapic_redirect_entry *entry,
 				   unsigned long *deliver_bitmask);
 #endif
 int kvm_set_irq(struct kvm *kvm, int irq_source_id, u32 irq, int level);
 void kvm_notify_acked_irq(struct kvm *kvm, unsigned irqchip, unsigned pin);
 void kvm_register_irq_ack_notifier(struct kvm *kvm,
 				   struct kvm_irq_ack_notifier *kian);
 void kvm_unregister_irq_ack_notifier(struct kvm *kvm,
 				   struct kvm_irq_ack_notifier *kian);
 int kvm_request_irq_source_id(struct kvm *kvm);
 void kvm_free_irq_source_id(struct kvm *kvm, int irq_source_id);
 /* For vcpu->arch.iommu_flags */
 #define KVM_IOMMU_CACHE_COHERENCY	0x1
 #ifdef CONFIG_IOMMU_API
 int kvm_iommu_map_pages(struct kvm *kvm, struct kvm_memory_slot *slot);
 int kvm_iommu_map_guest(struct kvm *kvm);
 int kvm_iommu_unmap_guest(struct kvm *kvm);
 int kvm_assign_device(struct kvm *kvm,
 		      struct kvm_assigned_dev_kernel *assigned_dev);
 int kvm_deassign_device(struct kvm *kvm,
 			struct kvm_assigned_dev_kernel *assigned_dev);
 #else /* CONFIG_IOMMU_API */
 static inline int kvm_iommu_map_pages(struct kvm *kvm,
 				      gfn_t base_gfn,
 				      unsigned long npages)
 {
 	return 0;
 }
 static inline int kvm_iommu_map_guest(struct kvm *kvm)
 {
 	return -ENODEV;
 }
 static inline int kvm_iommu_unmap_guest(struct kvm *kvm)
 {
 	return 0;
 }
 static inline int kvm_assign_device(struct kvm *kvm,
 		struct kvm_assigned_dev_kernel *assigned_dev)
 {
 	return 0;
 }
 static inline int kvm_deassign_device(struct kvm *kvm,
 		struct kvm_assigned_dev_kernel *assigned_dev)
 {
 	return 0;
 }
 #endif /* CONFIG_IOMMU_API */
 static inline void kvm_guest_enter(void)
 {
 	account_system_vtime(current);
 	current->flags |= PF_VCPU;
 }
 static inline void kvm_guest_exit(void)
 {
 	account_system_vtime(current);
 	current->flags &= ~PF_VCPU;
 }
 static inline gpa_t gfn_to_gpa(gfn_t gfn)
 {
 	return (gpa_t)gfn << PAGE_SHIFT;
 }
 static inline hpa_t pfn_to_hpa(pfn_t pfn)
 {
 	return (hpa_t)pfn << PAGE_SHIFT;
 }
 static inline void kvm_migrate_timers(struct kvm_vcpu *vcpu)
 {
 	set_bit(KVM_REQ_MIGRATE_TIMER, &vcpu->requests);
 }
 enum kvm_stat_kind {
 	KVM_STAT_VM,
 	KVM_STAT_VCPU,
 };
 struct kvm_stats_debugfs_item {
 	const char *name;
 	int offset;
 	enum kvm_stat_kind kind;
 	struct dentry *dentry;
 };
 extern struct kvm_stats_debugfs_item debugfs_entries[];
 extern struct dentry *kvm_debugfs_dir;
 #ifdef KVM_ARCH_WANT_MMU_NOTIFIER
 static inline int mmu_notifier_retry(struct kvm_vcpu *vcpu, unsigned long mmu_seq)
 {
 	if (unlikely(vcpu->kvm->mmu_notifier_count))
 		return 1;
 	/*
 	 * Both reads happen under the mmu_lock and both values are
 	 * modified under mmu_lock, so there's no need of smb_rmb()
 	 * here in between, otherwise mmu_notifier_count should be
 	 * read before mmu_notifier_seq, see
 	 * mmu_notifier_invalidate_range_end write side.
 	 */
 	if (vcpu->kvm->mmu_notifier_seq != mmu_seq)
 		return 1;
 	return 0;
 }
-#endif
-#ifndef KVM_ARCH_HAS_UNALIAS_INSTANTIATION
-#define unalias_gfn_instantiation unalias_gfn
 #endif
 #ifdef CONFIG_HAVE_KVM_IRQCHIP
 #define KVM_MAX_IRQ_ROUTES 1024
 int kvm_setup_default_irq_routing(struct kvm *kvm);
 int kvm_set_irq_routing(struct kvm *kvm,
 			const struct kvm_irq_routing_entry *entries,
 			unsigned nr,
 			unsigned flags);
 void kvm_free_irq_routing(struct kvm *kvm);
 #else
 static inline void kvm_free_irq_routing(struct kvm *kvm) {}
 #endif
 #ifdef CONFIG_HAVE_KVM_EVENTFD
 void kvm_eventfd_init(struct kvm *kvm);
 int kvm_irqfd(struct kvm *kvm, int fd, int gsi, int flags);
 void kvm_irqfd_release(struct kvm *kvm);
 int kvm_ioeventfd(struct kvm *kvm, struct kvm_ioeventfd *args);
 #else
 static inline void kvm_eventfd_init(struct kvm *kvm) {}
 static inline int kvm_irqfd(struct kvm *kvm, int fd, int gsi, int flags)
 {
 	return -EINVAL;
 }
 static inline void kvm_irqfd_release(struct kvm *kvm) {}
 static inline int kvm_ioeventfd(struct kvm *kvm, struct kvm_ioeventfd *args)
 {
 	return -ENOSYS;
 }
 #endif /* CONFIG_HAVE_KVM_EVENTFD */
 #ifdef CONFIG_KVM_APIC_ARCHITECTURE
 static inline bool kvm_vcpu_is_bsp(struct kvm_vcpu *vcpu)
 {
 	return vcpu->kvm->bsp_vcpu_id == vcpu->vcpu_id;
 }
 #endif
 #ifdef __KVM_HAVE_DEVICE_ASSIGNMENT
 long kvm_vm_ioctl_assigned_device(struct kvm *kvm, unsigned ioctl,
 				  unsigned long arg);
 #else
 static inline long kvm_vm_ioctl_assigned_device(struct kvm *kvm, unsigned ioctl,
 						unsigned long arg)
 {
 	return -ENOTTY;
 }
 #endif
 #endif

virt/kvm/kvm_main.c

Diff comments View file @ a1f4d39

 /*
  * Kernel-based Virtual Machine driver for Linux
  *
  * This module enables machines with Intel VT-x extensions to run virtual
  * machines without emulation or binary translation.
  *
  * Copyright (C) 2006 Qumranet, Inc.
  * Copyright 2010 Red Hat, Inc. and/or its affilates.
  *
  * Authors:
  *   Avi Kivity   <avi@qumranet.com>
  *   Yaniv Kamay  <yaniv@qumranet.com>
  *
  * This work is licensed under the terms of the GNU GPL, version 2.  See
  * the COPYING file in the top-level directory.
  *
  */
 #include "iodev.h"
 #include <linux/kvm_host.h>
 #include <linux/kvm.h>
 #include <linux/module.h>
 #include <linux/errno.h>
 #include <linux/percpu.h>
 #include <linux/mm.h>
 #include <linux/miscdevice.h>
 #include <linux/vmalloc.h>
 #include <linux/reboot.h>
 #include <linux/debugfs.h>
 #include <linux/highmem.h>
 #include <linux/file.h>
 #include <linux/sysdev.h>
 #include <linux/cpu.h>
 #include <linux/sched.h>
 #include <linux/cpumask.h>
 #include <linux/smp.h>
 #include <linux/anon_inodes.h>
 #include <linux/profile.h>
 #include <linux/kvm_para.h>
 #include <linux/pagemap.h>
 #include <linux/mman.h>
 #include <linux/swap.h>
 #include <linux/bitops.h>
 #include <linux/spinlock.h>
 #include <linux/compat.h>
 #include <linux/srcu.h>
 #include <linux/hugetlb.h>
 #include <linux/slab.h>
 #include <asm/processor.h>
 #include <asm/io.h>
 #include <asm/uaccess.h>
 #include <asm/pgtable.h>
 #include <asm-generic/bitops/le.h>
 #include "coalesced_mmio.h"
 #define CREATE_TRACE_POINTS
 #include <trace/events/kvm.h>
 MODULE_AUTHOR("Qumranet");
 MODULE_LICENSE("GPL");
 /*
  * Ordering of locks:
  *
  * 		kvm->lock --> kvm->slots_lock --> kvm->irq_lock
  */
 DEFINE_SPINLOCK(kvm_lock);
 LIST_HEAD(vm_list);
 static cpumask_var_t cpus_hardware_enabled;
 static int kvm_usage_count = 0;
 static atomic_t hardware_enable_failed;
 struct kmem_cache *kvm_vcpu_cache;
 EXPORT_SYMBOL_GPL(kvm_vcpu_cache);
 static __read_mostly struct preempt_ops kvm_preempt_ops;
 struct dentry *kvm_debugfs_dir;
 static long kvm_vcpu_ioctl(struct file *file, unsigned int ioctl,
 			   unsigned long arg);
 static int hardware_enable_all(void);
 static void hardware_disable_all(void);
 static void kvm_io_bus_destroy(struct kvm_io_bus *bus);
 static bool kvm_rebooting;
 static bool largepages_enabled = true;
 struct page *hwpoison_page;
 pfn_t hwpoison_pfn;
 inline int kvm_is_mmio_pfn(pfn_t pfn)
 {
 	if (pfn_valid(pfn)) {
 		struct page *page = compound_head(pfn_to_page(pfn));
 		return PageReserved(page);
 	}
 	return true;
 }
 /*
  * Switches to specified vcpu, until a matching vcpu_put()
  */
 void vcpu_load(struct kvm_vcpu *vcpu)
 {
 	int cpu;
 	mutex_lock(&vcpu->mutex);
 	cpu = get_cpu();
 	preempt_notifier_register(&vcpu->preempt_notifier);
 	kvm_arch_vcpu_load(vcpu, cpu);
 	put_cpu();
 }
 void vcpu_put(struct kvm_vcpu *vcpu)
 {
 	preempt_disable();
 	kvm_arch_vcpu_put(vcpu);
 	preempt_notifier_unregister(&vcpu->preempt_notifier);
 	preempt_enable();
 	mutex_unlock(&vcpu->mutex);
 }
 static void ack_flush(void *_completed)
 {
 }
 static bool make_all_cpus_request(struct kvm *kvm, unsigned int req)
 {
 	int i, cpu, me;
 	cpumask_var_t cpus;
 	bool called = true;
 	struct kvm_vcpu *vcpu;
 	zalloc_cpumask_var(&cpus, GFP_ATOMIC);
 	raw_spin_lock(&kvm->requests_lock);
 	me = smp_processor_id();
 	kvm_for_each_vcpu(i, vcpu, kvm) {
 		if (test_and_set_bit(req, &vcpu->requests))
 			continue;
 		cpu = vcpu->cpu;
 		if (cpus != NULL && cpu != -1 && cpu != me)
 			cpumask_set_cpu(cpu, cpus);
 	}
 	if (unlikely(cpus == NULL))
 		smp_call_function_many(cpu_online_mask, ack_flush, NULL, 1);
 	else if (!cpumask_empty(cpus))
 		smp_call_function_many(cpus, ack_flush, NULL, 1);
 	else
 		called = false;
 	raw_spin_unlock(&kvm->requests_lock);
 	free_cpumask_var(cpus);
 	return called;
 }
 void kvm_flush_remote_tlbs(struct kvm *kvm)
 {
 	if (make_all_cpus_request(kvm, KVM_REQ_TLB_FLUSH))
 		++kvm->stat.remote_tlb_flush;
 }
 void kvm_reload_remote_mmus(struct kvm *kvm)
 {
 	make_all_cpus_request(kvm, KVM_REQ_MMU_RELOAD);
 }
 int kvm_vcpu_init(struct kvm_vcpu *vcpu, struct kvm *kvm, unsigned id)
 {
 	struct page *page;
 	int r;
 	mutex_init(&vcpu->mutex);
 	vcpu->cpu = -1;
 	vcpu->kvm = kvm;
 	vcpu->vcpu_id = id;
 	init_waitqueue_head(&vcpu->wq);
 	page = alloc_page(GFP_KERNEL | __GFP_ZERO);
 	if (!page) {
 		r = -ENOMEM;
 		goto fail;
 	}
 	vcpu->run = page_address(page);
 	r = kvm_arch_vcpu_init(vcpu);
 	if (r < 0)
 		goto fail_free_run;
 	return 0;
 fail_free_run:
 	free_page((unsigned long)vcpu->run);
 fail:
 	return r;
 }
 EXPORT_SYMBOL_GPL(kvm_vcpu_init);
 void kvm_vcpu_uninit(struct kvm_vcpu *vcpu)
 {
 	kvm_arch_vcpu_uninit(vcpu);
 	free_page((unsigned long)vcpu->run);
 }
 EXPORT_SYMBOL_GPL(kvm_vcpu_uninit);
 #if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER)
 static inline struct kvm *mmu_notifier_to_kvm(struct mmu_notifier *mn)
 {
 	return container_of(mn, struct kvm, mmu_notifier);
 }
 static void kvm_mmu_notifier_invalidate_page(struct mmu_notifier *mn,
 					     struct mm_struct *mm,
 					     unsigned long address)
 {
 	struct kvm *kvm = mmu_notifier_to_kvm(mn);
 	int need_tlb_flush, idx;
 	/*
 	 * When ->invalidate_page runs, the linux pte has been zapped
 	 * already but the page is still allocated until
 	 * ->invalidate_page returns. So if we increase the sequence
 	 * here the kvm page fault will notice if the spte can't be
 	 * established because the page is going to be freed. If
 	 * instead the kvm page fault establishes the spte before
 	 * ->invalidate_page runs, kvm_unmap_hva will release it
 	 * before returning.
 	 *
 	 * The sequence increase only need to be seen at spin_unlock
 	 * time, and not at spin_lock time.
 	 *
 	 * Increasing the sequence after the spin_unlock would be
 	 * unsafe because the kvm page fault could then establish the
 	 * pte after kvm_unmap_hva returned, without noticing the page
 	 * is going to be freed.
 	 */
 	idx = srcu_read_lock(&kvm->srcu);
 	spin_lock(&kvm->mmu_lock);
 	kvm->mmu_notifier_seq++;
 	need_tlb_flush = kvm_unmap_hva(kvm, address);
 	spin_unlock(&kvm->mmu_lock);
 	srcu_read_unlock(&kvm->srcu, idx);
 	/* we've to flush the tlb before the pages can be freed */
 	if (need_tlb_flush)
 		kvm_flush_remote_tlbs(kvm);
 }
 static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn,
 					struct mm_struct *mm,
 					unsigned long address,
 					pte_t pte)
 {
 	struct kvm *kvm = mmu_notifier_to_kvm(mn);
 	int idx;
 	idx = srcu_read_lock(&kvm->srcu);
 	spin_lock(&kvm->mmu_lock);
 	kvm->mmu_notifier_seq++;
 	kvm_set_spte_hva(kvm, address, pte);
 	spin_unlock(&kvm->mmu_lock);
 	srcu_read_unlock(&kvm->srcu, idx);
 }
 static void kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn,
 						    struct mm_struct *mm,
 						    unsigned long start,
 						    unsigned long end)
 {
 	struct kvm *kvm = mmu_notifier_to_kvm(mn);
 	int need_tlb_flush = 0, idx;
 	idx = srcu_read_lock(&kvm->srcu);
 	spin_lock(&kvm->mmu_lock);
 	/*
 	 * The count increase must become visible at unlock time as no
 	 * spte can be established without taking the mmu_lock and
 	 * count is also read inside the mmu_lock critical section.
 	 */
 	kvm->mmu_notifier_count++;
 	for (; start < end; start += PAGE_SIZE)
 		need_tlb_flush |= kvm_unmap_hva(kvm, start);
 	spin_unlock(&kvm->mmu_lock);
 	srcu_read_unlock(&kvm->srcu, idx);
 	/* we've to flush the tlb before the pages can be freed */
 	if (need_tlb_flush)
 		kvm_flush_remote_tlbs(kvm);
 }
 static void kvm_mmu_notifier_invalidate_range_end(struct mmu_notifier *mn,
 						  struct mm_struct *mm,
 						  unsigned long start,
 						  unsigned long end)
 {
 	struct kvm *kvm = mmu_notifier_to_kvm(mn);
 	spin_lock(&kvm->mmu_lock);
 	/*
 	 * This sequence increase will notify the kvm page fault that
 	 * the page that is going to be mapped in the spte could have
 	 * been freed.
 	 */
 	kvm->mmu_notifier_seq++;
 	/*
 	 * The above sequence increase must be visible before the
 	 * below count decrease but both values are read by the kvm
 	 * page fault under mmu_lock spinlock so we don't need to add
 	 * a smb_wmb() here in between the two.
 	 */
 	kvm->mmu_notifier_count--;
 	spin_unlock(&kvm->mmu_lock);
 	BUG_ON(kvm->mmu_notifier_count < 0);
 }
 static int kvm_mmu_notifier_clear_flush_young(struct mmu_notifier *mn,
 					      struct mm_struct *mm,
 					      unsigned long address)
 {
 	struct kvm *kvm = mmu_notifier_to_kvm(mn);
 	int young, idx;
 	idx = srcu_read_lock(&kvm->srcu);
 	spin_lock(&kvm->mmu_lock);
 	young = kvm_age_hva(kvm, address);
 	spin_unlock(&kvm->mmu_lock);
 	srcu_read_unlock(&kvm->srcu, idx);
 	if (young)
 		kvm_flush_remote_tlbs(kvm);
 	return young;
 }
 static void kvm_mmu_notifier_release(struct mmu_notifier *mn,
 				     struct mm_struct *mm)
 {
 	struct kvm *kvm = mmu_notifier_to_kvm(mn);
 	int idx;
 	idx = srcu_read_lock(&kvm->srcu);
 	kvm_arch_flush_shadow(kvm);
 	srcu_read_unlock(&kvm->srcu, idx);
 }
 static const struct mmu_notifier_ops kvm_mmu_notifier_ops = {
 	.invalidate_page	= kvm_mmu_notifier_invalidate_page,
 	.invalidate_range_start	= kvm_mmu_notifier_invalidate_range_start,
 	.invalidate_range_end	= kvm_mmu_notifier_invalidate_range_end,
 	.clear_flush_young	= kvm_mmu_notifier_clear_flush_young,
 	.change_pte		= kvm_mmu_notifier_change_pte,
 	.release		= kvm_mmu_notifier_release,
 };
 static int kvm_init_mmu_notifier(struct kvm *kvm)
 {
 	kvm->mmu_notifier.ops = &kvm_mmu_notifier_ops;
 	return mmu_notifier_register(&kvm->mmu_notifier, current->mm);
 }
 #else  /* !(CONFIG_MMU_NOTIFIER && KVM_ARCH_WANT_MMU_NOTIFIER) */
 static int kvm_init_mmu_notifier(struct kvm *kvm)
 {
 	return 0;
 }
 #endif /* CONFIG_MMU_NOTIFIER && KVM_ARCH_WANT_MMU_NOTIFIER */
 static struct kvm *kvm_create_vm(void)
 {
 	int r = 0, i;
 	struct kvm *kvm = kvm_arch_create_vm();
 	if (IS_ERR(kvm))
 		goto out;
 	r = hardware_enable_all();
 	if (r)
 		goto out_err_nodisable;
 #ifdef CONFIG_HAVE_KVM_IRQCHIP
 	INIT_HLIST_HEAD(&kvm->mask_notifier_list);
 	INIT_HLIST_HEAD(&kvm->irq_ack_notifier_list);
 #endif
 	r = -ENOMEM;
 	kvm->memslots = kzalloc(sizeof(struct kvm_memslots), GFP_KERNEL);
 	if (!kvm->memslots)
 		goto out_err;
 	if (init_srcu_struct(&kvm->srcu))
 		goto out_err;
 	for (i = 0; i < KVM_NR_BUSES; i++) {
 		kvm->buses[i] = kzalloc(sizeof(struct kvm_io_bus),
 					GFP_KERNEL);
 		if (!kvm->buses[i]) {
 			cleanup_srcu_struct(&kvm->srcu);
 			goto out_err;
 		}
 	}
 	r = kvm_init_mmu_notifier(kvm);
 	if (r) {
 		cleanup_srcu_struct(&kvm->srcu);
 		goto out_err;
 	}
 	kvm->mm = current->mm;
 	atomic_inc(&kvm->mm->mm_count);
 	spin_lock_init(&kvm->mmu_lock);
 	raw_spin_lock_init(&kvm->requests_lock);
 	kvm_eventfd_init(kvm);
 	mutex_init(&kvm->lock);
 	mutex_init(&kvm->irq_lock);
 	mutex_init(&kvm->slots_lock);
 	atomic_set(&kvm->users_count, 1);
 	spin_lock(&kvm_lock);
 	list_add(&kvm->vm_list, &vm_list);
 	spin_unlock(&kvm_lock);
 out:
 	return kvm;
 out_err:
 	hardware_disable_all();
 out_err_nodisable:
 	for (i = 0; i < KVM_NR_BUSES; i++)
 		kfree(kvm->buses[i]);
 	kfree(kvm->memslots);
 	kfree(kvm);
 	return ERR_PTR(r);
 }
 /*
  * Free any memory in @free but not in @dont.
  */
 static void kvm_free_physmem_slot(struct kvm_memory_slot *free,
 				  struct kvm_memory_slot *dont)
 {
 	int i;
 	if (!dont || free->rmap != dont->rmap)
 		vfree(free->rmap);
 	if (!dont || free->dirty_bitmap != dont->dirty_bitmap)
 		vfree(free->dirty_bitmap);
 	for (i = 0; i < KVM_NR_PAGE_SIZES - 1; ++i) {
 		if (!dont || free->lpage_info[i] != dont->lpage_info[i]) {
 			vfree(free->lpage_info[i]);
 			free->lpage_info[i] = NULL;
 		}
 	}
 	free->npages = 0;
 	free->dirty_bitmap = NULL;
 	free->rmap = NULL;
 }
 void kvm_free_physmem(struct kvm *kvm)
 {
 	int i;
 	struct kvm_memslots *slots = kvm->memslots;
 	for (i = 0; i < slots->nmemslots; ++i)
 		kvm_free_physmem_slot(&slots->memslots[i], NULL);
 	kfree(kvm->memslots);
 }
 static void kvm_destroy_vm(struct kvm *kvm)
 {
 	int i;
 	struct mm_struct *mm = kvm->mm;
 	kvm_arch_sync_events(kvm);
 	spin_lock(&kvm_lock);
 	list_del(&kvm->vm_list);
 	spin_unlock(&kvm_lock);
 	kvm_free_irq_routing(kvm);
 	for (i = 0; i < KVM_NR_BUSES; i++)
 		kvm_io_bus_destroy(kvm->buses[i]);
 	kvm_coalesced_mmio_free(kvm);
 #if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER)
 	mmu_notifier_unregister(&kvm->mmu_notifier, kvm->mm);
 #else
 	kvm_arch_flush_shadow(kvm);
 #endif
 	kvm_arch_destroy_vm(kvm);
 	hardware_disable_all();
 	mmdrop(mm);
 }
 void kvm_get_kvm(struct kvm *kvm)
 {
 	atomic_inc(&kvm->users_count);
 }
 EXPORT_SYMBOL_GPL(kvm_get_kvm);
 void kvm_put_kvm(struct kvm *kvm)
 {
 	if (atomic_dec_and_test(&kvm->users_count))
 		kvm_destroy_vm(kvm);
 }
 EXPORT_SYMBOL_GPL(kvm_put_kvm);
 static int kvm_vm_release(struct inode *inode, struct file *filp)
 {
 	struct kvm *kvm = filp->private_data;
 	kvm_irqfd_release(kvm);
 	kvm_put_kvm(kvm);
 	return 0;
 }
 /*
  * Allocate some memory and give it an address in the guest physical address
  * space.
  *
  * Discontiguous memory is allowed, mostly for framebuffers.
  *
  * Must be called holding mmap_sem for write.
  */
 int __kvm_set_memory_region(struct kvm *kvm,
 			    struct kvm_userspace_memory_region *mem,
 			    int user_alloc)
 {
 	int r, flush_shadow = 0;
 	gfn_t base_gfn;
 	unsigned long npages;
 	unsigned long i;
 	struct kvm_memory_slot *memslot;
 	struct kvm_memory_slot old, new;
 	struct kvm_memslots *slots, *old_memslots;
 	r = -EINVAL;
 	/* General sanity checks */
 	if (mem->memory_size & (PAGE_SIZE - 1))
 		goto out;
 	if (mem->guest_phys_addr & (PAGE_SIZE - 1))
 		goto out;
 	if (user_alloc && (mem->userspace_addr & (PAGE_SIZE - 1)))
 		goto out;
 	if (mem->slot >= KVM_MEMORY_SLOTS + KVM_PRIVATE_MEM_SLOTS)
 		goto out;
 	if (mem->guest_phys_addr + mem->memory_size < mem->guest_phys_addr)
 		goto out;
 	memslot = &kvm->memslots->memslots[mem->slot];
 	base_gfn = mem->guest_phys_addr >> PAGE_SHIFT;
 	npages = mem->memory_size >> PAGE_SHIFT;
 	r = -EINVAL;
 	if (npages > KVM_MEM_MAX_NR_PAGES)
 		goto out;
 	if (!npages)
 		mem->flags &= ~KVM_MEM_LOG_DIRTY_PAGES;
 	new = old = *memslot;
 	new.base_gfn = base_gfn;
 	new.npages = npages;
 	new.flags = mem->flags;
 	/* Disallow changing a memory slot's size. */
 	r = -EINVAL;
 	if (npages && old.npages && npages != old.npages)
 		goto out_free;
 	/* Check for overlaps */
 	r = -EEXIST;
 	for (i = 0; i < KVM_MEMORY_SLOTS; ++i) {
 		struct kvm_memory_slot *s = &kvm->memslots->memslots[i];
 		if (s == memslot || !s->npages)
 			continue;
 		if (!((base_gfn + npages <= s->base_gfn) ||
 		      (base_gfn >= s->base_gfn + s->npages)))
 			goto out_free;
 	}
 	/* Free page dirty bitmap if unneeded */
 	if (!(new.flags & KVM_MEM_LOG_DIRTY_PAGES))
 		new.dirty_bitmap = NULL;
 	r = -ENOMEM;
 	/* Allocate if a slot is being created */
 #ifndef CONFIG_S390
 	if (npages && !new.rmap) {
 		new.rmap = vmalloc(npages * sizeof(*new.rmap));
 		if (!new.rmap)
 			goto out_free;
 		memset(new.rmap, 0, npages * sizeof(*new.rmap));
 		new.user_alloc = user_alloc;
 		new.userspace_addr = mem->userspace_addr;
 	}
 	if (!npages)
 		goto skip_lpage;
 	for (i = 0; i < KVM_NR_PAGE_SIZES - 1; ++i) {
 		unsigned long ugfn;
 		unsigned long j;
 		int lpages;
 		int level = i + 2;
 		/* Avoid unused variable warning if no large pages */
 		(void)level;
 		if (new.lpage_info[i])
 			continue;
 		lpages = 1 + (base_gfn + npages - 1) /
 			     KVM_PAGES_PER_HPAGE(level);
 		lpages -= base_gfn / KVM_PAGES_PER_HPAGE(level);
 		new.lpage_info[i] = vmalloc(lpages * sizeof(*new.lpage_info[i]));
 		if (!new.lpage_info[i])
 			goto out_free;
 		memset(new.lpage_info[i], 0,
 		       lpages * sizeof(*new.lpage_info[i]));
 		if (base_gfn % KVM_PAGES_PER_HPAGE(level))
 			new.lpage_info[i][0].write_count = 1;
 		if ((base_gfn+npages) % KVM_PAGES_PER_HPAGE(level))
 			new.lpage_info[i][lpages - 1].write_count = 1;
 		ugfn = new.userspace_addr >> PAGE_SHIFT;
 		/*
 		 * If the gfn and userspace address are not aligned wrt each
 		 * other, or if explicitly asked to, disable large page
 		 * support for this slot
 		 */
 		if ((base_gfn ^ ugfn) & (KVM_PAGES_PER_HPAGE(level) - 1) ||
 		    !largepages_enabled)
 			for (j = 0; j < lpages; ++j)
 				new.lpage_info[i][j].write_count = 1;
 	}
 skip_lpage:
 	/* Allocate page dirty bitmap if needed */
 	if ((new.flags & KVM_MEM_LOG_DIRTY_PAGES) && !new.dirty_bitmap) {
 		unsigned long dirty_bytes = kvm_dirty_bitmap_bytes(&new);
 		new.dirty_bitmap = vmalloc(dirty_bytes);
 		if (!new.dirty_bitmap)
 			goto out_free;
 		memset(new.dirty_bitmap, 0, dirty_bytes);
 		/* destroy any largepage mappings for dirty tracking */
 		if (old.npages)
 			flush_shadow = 1;
 	}
 #else  /* not defined CONFIG_S390 */
 	new.user_alloc = user_alloc;
 	if (user_alloc)
 		new.userspace_addr = mem->userspace_addr;
 #endif /* not defined CONFIG_S390 */
 	if (!npages) {
 		r = -ENOMEM;
 		slots = kzalloc(sizeof(struct kvm_memslots), GFP_KERNEL);
 		if (!slots)
 			goto out_free;
 		memcpy(slots, kvm->memslots, sizeof(struct kvm_memslots));
 		if (mem->slot >= slots->nmemslots)
 			slots->nmemslots = mem->slot + 1;
 		slots->memslots[mem->slot].flags |= KVM_MEMSLOT_INVALID;
 		old_memslots = kvm->memslots;
 		rcu_assign_pointer(kvm->memslots, slots);
 		synchronize_srcu_expedited(&kvm->srcu);
 		/* From this point no new shadow pages pointing to a deleted
 		 * memslot will be created.
 		 *
 		 * validation of sp->gfn happens in:
 		 * 	- gfn_to_hva (kvm_read_guest, gfn_to_pfn)
 		 * 	- kvm_is_visible_gfn (mmu_check_roots)
 		 */
 		kvm_arch_flush_shadow(kvm);
 		kfree(old_memslots);
 	}
 	r = kvm_arch_prepare_memory_region(kvm, &new, old, mem, user_alloc);
 	if (r)
 		goto out_free;
 #ifdef CONFIG_DMAR
 	/* map the pages in iommu page table */
 	if (npages) {
 		r = kvm_iommu_map_pages(kvm, &new);
 		if (r)
 			goto out_free;
 	}
 #endif
 	r = -ENOMEM;
 	slots = kzalloc(sizeof(struct kvm_memslots), GFP_KERNEL);
 	if (!slots)
 		goto out_free;
 	memcpy(slots, kvm->memslots, sizeof(struct kvm_memslots));
 	if (mem->slot >= slots->nmemslots)
 		slots->nmemslots = mem->slot + 1;
 	/* actual memory is freed via old in kvm_free_physmem_slot below */
 	if (!npages) {
 		new.rmap = NULL;
 		new.dirty_bitmap = NULL;
 		for (i = 0; i < KVM_NR_PAGE_SIZES - 1; ++i)
 			new.lpage_info[i] = NULL;
 	}
 	slots->memslots[mem->slot] = new;
 	old_memslots = kvm->memslots;
 	rcu_assign_pointer(kvm->memslots, slots);
 	synchronize_srcu_expedited(&kvm->srcu);
 	kvm_arch_commit_memory_region(kvm, mem, old, user_alloc);
 	kvm_free_physmem_slot(&old, &new);
 	kfree(old_memslots);
 	if (flush_shadow)
 		kvm_arch_flush_shadow(kvm);
 	return 0;
 out_free:
 	kvm_free_physmem_slot(&new, &old);
 out:
 	return r;
 }
 EXPORT_SYMBOL_GPL(__kvm_set_memory_region);
 int kvm_set_memory_region(struct kvm *kvm,
 			  struct kvm_userspace_memory_region *mem,
 			  int user_alloc)
 {
 	int r;
 	mutex_lock(&kvm->slots_lock);
 	r = __kvm_set_memory_region(kvm, mem, user_alloc);
 	mutex_unlock(&kvm->slots_lock);
 	return r;
 }
 EXPORT_SYMBOL_GPL(kvm_set_memory_region);
 int kvm_vm_ioctl_set_memory_region(struct kvm *kvm,
 				   struct
 				   kvm_userspace_memory_region *mem,
 				   int user_alloc)
 {
 	if (mem->slot >= KVM_MEMORY_SLOTS)
 		return -EINVAL;
 	return kvm_set_memory_region(kvm, mem, user_alloc);
 }
 int kvm_get_dirty_log(struct kvm *kvm,
 			struct kvm_dirty_log *log, int *is_dirty)
 {
 	struct kvm_memory_slot *memslot;
 	int r, i;
 	unsigned long n;
 	unsigned long any = 0;
 	r = -EINVAL;
 	if (log->slot >= KVM_MEMORY_SLOTS)
 		goto out;
 	memslot = &kvm->memslots->memslots[log->slot];
 	r = -ENOENT;
 	if (!memslot->dirty_bitmap)
 		goto out;
 	n = kvm_dirty_bitmap_bytes(memslot);
 	for (i = 0; !any && i < n/sizeof(long); ++i)
 		any = memslot->dirty_bitmap[i];
 	r = -EFAULT;
 	if (copy_to_user(log->dirty_bitmap, memslot->dirty_bitmap, n))
 		goto out;
 	if (any)
 		*is_dirty = 1;
 	r = 0;
 out:
 	return r;
 }
 void kvm_disable_largepages(void)
 {
 	largepages_enabled = false;
 }
 EXPORT_SYMBOL_GPL(kvm_disable_largepages);
 int is_error_page(struct page *page)
 {
 	return page == bad_page || page == hwpoison_page;
 }
 EXPORT_SYMBOL_GPL(is_error_page);
 int is_error_pfn(pfn_t pfn)
 {
 	return pfn == bad_pfn || pfn == hwpoison_pfn;
 }
 EXPORT_SYMBOL_GPL(is_error_pfn);
 int is_hwpoison_pfn(pfn_t pfn)
 {
 	return pfn == hwpoison_pfn;
 }
 EXPORT_SYMBOL_GPL(is_hwpoison_pfn);
 static inline unsigned long bad_hva(void)
 {
 	return PAGE_OFFSET;
 }
 int kvm_is_error_hva(unsigned long addr)
 {
 	return addr == bad_hva();
 }
 EXPORT_SYMBOL_GPL(kvm_is_error_hva);
-struct kvm_memory_slot *gfn_to_memslot_unaliased(struct kvm *kvm, gfn_t gfn)
+struct kvm_memory_slot *gfn_to_memslot(struct kvm *kvm, gfn_t gfn)
 {
 	int i;
 	struct kvm_memslots *slots = kvm_memslots(kvm);
 	for (i = 0; i < slots->nmemslots; ++i) {
 		struct kvm_memory_slot *memslot = &slots->memslots[i];
 		if (gfn >= memslot->base_gfn
 		    && gfn < memslot->base_gfn + memslot->npages)
 			return memslot;
 	}
 	return NULL;
 }
-EXPORT_SYMBOL_GPL(gfn_to_memslot_unaliased);
+EXPORT_SYMBOL_GPL(gfn_to_memslot);
-struct kvm_memory_slot *gfn_to_memslot(struct kvm *kvm, gfn_t gfn)
-{
-	gfn = unalias_gfn(kvm, gfn);
-	return gfn_to_memslot_unaliased(kvm, gfn);
-}
 int kvm_is_visible_gfn(struct kvm *kvm, gfn_t gfn)
 {
 	int i;
 	struct kvm_memslots *slots = kvm_memslots(kvm);
-	gfn = unalias_gfn_instantiation(kvm, gfn);
 	for (i = 0; i < KVM_MEMORY_SLOTS; ++i) {
 		struct kvm_memory_slot *memslot = &slots->memslots[i];
 		if (memslot->flags & KVM_MEMSLOT_INVALID)
 			continue;
 		if (gfn >= memslot->base_gfn
 		    && gfn < memslot->base_gfn + memslot->npages)
 			return 1;
 	}
 	return 0;
 }
 EXPORT_SYMBOL_GPL(kvm_is_visible_gfn);
 unsigned long kvm_host_page_size(struct kvm *kvm, gfn_t gfn)
 {
 	struct vm_area_struct *vma;
 	unsigned long addr, size;
 	size = PAGE_SIZE;
 	addr = gfn_to_hva(kvm, gfn);
 	if (kvm_is_error_hva(addr))
 		return PAGE_SIZE;
 	down_read(&current->mm->mmap_sem);
 	vma = find_vma(current->mm, addr);
 	if (!vma)
 		goto out;
 	size = vma_kernel_pagesize(vma);
 out:
 	up_read(&current->mm->mmap_sem);
 	return size;
 }
 int memslot_id(struct kvm *kvm, gfn_t gfn)
 {
 	int i;
 	struct kvm_memslots *slots = kvm_memslots(kvm);
 	struct kvm_memory_slot *memslot = NULL;
-	gfn = unalias_gfn(kvm, gfn);
 	for (i = 0; i < slots->nmemslots; ++i) {
 		memslot = &slots->memslots[i];
 		if (gfn >= memslot->base_gfn
 		    && gfn < memslot->base_gfn + memslot->npages)
 			break;
 	}
 	return memslot - slots->memslots;
 }
 static unsigned long gfn_to_hva_memslot(struct kvm_memory_slot *slot, gfn_t gfn)
 {
 	return slot->userspace_addr + (gfn - slot->base_gfn) * PAGE_SIZE;
 }
 unsigned long gfn_to_hva(struct kvm *kvm, gfn_t gfn)
 {
 	struct kvm_memory_slot *slot;
-	gfn = unalias_gfn_instantiation(kvm, gfn);
+	slot = gfn_to_memslot(kvm, gfn);
-	slot = gfn_to_memslot_unaliased(kvm, gfn);
 	if (!slot || slot->flags & KVM_MEMSLOT_INVALID)
 		return bad_hva();
 	return gfn_to_hva_memslot(slot, gfn);
 }
 EXPORT_SYMBOL_GPL(gfn_to_hva);
 static pfn_t hva_to_pfn(struct kvm *kvm, unsigned long addr)
 {
 	struct page *page[1];
 	int npages;
 	pfn_t pfn;
 	might_sleep();
 	npages = get_user_pages_fast(addr, 1, 1, page);
 	if (unlikely(npages != 1)) {
 		struct vm_area_struct *vma;
 		if (is_hwpoison_address(addr)) {
 			get_page(hwpoison_page);
 			return page_to_pfn(hwpoison_page);
 		}
 		down_read(&current->mm->mmap_sem);
 		vma = find_vma(current->mm, addr);
 		if (vma == NULL || addr < vma->vm_start ||
 		    !(vma->vm_flags & VM_PFNMAP)) {
 			up_read(&current->mm->mmap_sem);
 			get_page(bad_page);
 			return page_to_pfn(bad_page);
 		}
 		pfn = ((addr - vma->vm_start) >> PAGE_SHIFT) + vma->vm_pgoff;
 		up_read(&current->mm->mmap_sem);
 		BUG_ON(!kvm_is_mmio_pfn(pfn));
 	} else
 		pfn = page_to_pfn(page[0]);
 	return pfn;
 }
 pfn_t gfn_to_pfn(struct kvm *kvm, gfn_t gfn)
 {
 	unsigned long addr;
 	addr = gfn_to_hva(kvm, gfn);
 	if (kvm_is_error_hva(addr)) {
 		get_page(bad_page);
 		return page_to_pfn(bad_page);
 	}
 	return hva_to_pfn(kvm, addr);
 }
 EXPORT_SYMBOL_GPL(gfn_to_pfn);
 pfn_t gfn_to_pfn_memslot(struct kvm *kvm,
 			 struct kvm_memory_slot *slot, gfn_t gfn)
 {
 	unsigned long addr = gfn_to_hva_memslot(slot, gfn);
 	return hva_to_pfn(kvm, addr);
 }
 struct page *gfn_to_page(struct kvm *kvm, gfn_t gfn)
 {
 	pfn_t pfn;
 	pfn = gfn_to_pfn(kvm, gfn);
 	if (!kvm_is_mmio_pfn(pfn))
 		return pfn_to_page(pfn);
 	WARN_ON(kvm_is_mmio_pfn(pfn));
 	get_page(bad_page);
 	return bad_page;
 }
 EXPORT_SYMBOL_GPL(gfn_to_page);
 void kvm_release_page_clean(struct page *page)
 {
 	kvm_release_pfn_clean(page_to_pfn(page));
 }
 EXPORT_SYMBOL_GPL(kvm_release_page_clean);
 void kvm_release_pfn_clean(pfn_t pfn)
 {
 	if (!kvm_is_mmio_pfn(pfn))
 		put_page(pfn_to_page(pfn));
 }
 EXPORT_SYMBOL_GPL(kvm_release_pfn_clean);
 void kvm_release_page_dirty(struct page *page)
 {
 	kvm_release_pfn_dirty(page_to_pfn(page));
 }
 EXPORT_SYMBOL_GPL(kvm_release_page_dirty);
 void kvm_release_pfn_dirty(pfn_t pfn)
 {
 	kvm_set_pfn_dirty(pfn);
 	kvm_release_pfn_clean(pfn);
 }
 EXPORT_SYMBOL_GPL(kvm_release_pfn_dirty);
 void kvm_set_page_dirty(struct page *page)
 {
 	kvm_set_pfn_dirty(page_to_pfn(page));
 }
 EXPORT_SYMBOL_GPL(kvm_set_page_dirty);
 void kvm_set_pfn_dirty(pfn_t pfn)
 {
 	if (!kvm_is_mmio_pfn(pfn)) {
 		struct page *page = pfn_to_page(pfn);
 		if (!PageReserved(page))
 			SetPageDirty(page);
 	}
 }
 EXPORT_SYMBOL_GPL(kvm_set_pfn_dirty);
 void kvm_set_pfn_accessed(pfn_t pfn)
 {
 	if (!kvm_is_mmio_pfn(pfn))
 		mark_page_accessed(pfn_to_page(pfn));
 }
 EXPORT_SYMBOL_GPL(kvm_set_pfn_accessed);
 void kvm_get_pfn(pfn_t pfn)
 {
 	if (!kvm_is_mmio_pfn(pfn))
 		get_page(pfn_to_page(pfn));
 }
 EXPORT_SYMBOL_GPL(kvm_get_pfn);
 static int next_segment(unsigned long len, int offset)
 {
 	if (len > PAGE_SIZE - offset)
 		return PAGE_SIZE - offset;
 	else
 		return len;
 }
 int kvm_read_guest_page(struct kvm *kvm, gfn_t gfn, void *data, int offset,
 			int len)
 {
 	int r;
 	unsigned long addr;
 	addr = gfn_to_hva(kvm, gfn);
 	if (kvm_is_error_hva(addr))
 		return -EFAULT;
 	r = copy_from_user(data, (void __user *)addr + offset, len);
 	if (r)
 		return -EFAULT;
 	return 0;
 }
 EXPORT_SYMBOL_GPL(kvm_read_guest_page);
 int kvm_read_guest(struct kvm *kvm, gpa_t gpa, void *data, unsigned long len)
 {
 	gfn_t gfn = gpa >> PAGE_SHIFT;
 	int seg;
 	int offset = offset_in_page(gpa);
 	int ret;
 	while ((seg = next_segment(len, offset)) != 0) {
 		ret = kvm_read_guest_page(kvm, gfn, data, offset, seg);
 		if (ret < 0)
 			return ret;
 		offset = 0;
 		len -= seg;
 		data += seg;
 		++gfn;
 	}
 	return 0;
 }
 EXPORT_SYMBOL_GPL(kvm_read_guest);
 int kvm_read_guest_atomic(struct kvm *kvm, gpa_t gpa, void *data,
 			  unsigned long len)
 {
 	int r;
 	unsigned long addr;
 	gfn_t gfn = gpa >> PAGE_SHIFT;
 	int offset = offset_in_page(gpa);
 	addr = gfn_to_hva(kvm, gfn);
 	if (kvm_is_error_hva(addr))
 		return -EFAULT;
 	pagefault_disable();
 	r = __copy_from_user_inatomic(data, (void __user *)addr + offset, len);
 	pagefault_enable();
 	if (r)
 		return -EFAULT;
 	return 0;
 }
 EXPORT_SYMBOL(kvm_read_guest_atomic);
 int kvm_write_guest_page(struct kvm *kvm, gfn_t gfn, const void *data,
 			 int offset, int len)
 {
 	int r;
 	unsigned long addr;
 	addr = gfn_to_hva(kvm, gfn);
 	if (kvm_is_error_hva(addr))
 		return -EFAULT;
 	r = copy_to_user((void __user *)addr + offset, data, len);
 	if (r)
 		return -EFAULT;
 	mark_page_dirty(kvm, gfn);
 	return 0;
 }
 EXPORT_SYMBOL_GPL(kvm_write_guest_page);
 int kvm_write_guest(struct kvm *kvm, gpa_t gpa, const void *data,
 		    unsigned long len)
 {
 	gfn_t gfn = gpa >> PAGE_SHIFT;
 	int seg;
 	int offset = offset_in_page(gpa);
 	int ret;
 	while ((seg = next_segment(len, offset)) != 0) {
 		ret = kvm_write_guest_page(kvm, gfn, data, offset, seg);
 		if (ret < 0)
 			return ret;
 		offset = 0;
 		len -= seg;
 		data += seg;
 		++gfn;
 	}
 	return 0;
 }
 int kvm_clear_guest_page(struct kvm *kvm, gfn_t gfn, int offset, int len)
 {
 	return kvm_write_guest_page(kvm, gfn, empty_zero_page, offset, len);
 }
 EXPORT_SYMBOL_GPL(kvm_clear_guest_page);
 int kvm_clear_guest(struct kvm *kvm, gpa_t gpa, unsigned long len)
 {
 	gfn_t gfn = gpa >> PAGE_SHIFT;
 	int seg;
 	int offset = offset_in_page(gpa);
 	int ret;
         while ((seg = next_segment(len, offset)) != 0) {
 		ret = kvm_clear_guest_page(kvm, gfn, offset, seg);
 		if (ret < 0)
 			return ret;
 		offset = 0;
 		len -= seg;
 		++gfn;
 	}
 	return 0;
 }
 EXPORT_SYMBOL_GPL(kvm_clear_guest);
 void mark_page_dirty(struct kvm *kvm, gfn_t gfn)
 {
 	struct kvm_memory_slot *memslot;
-	gfn = unalias_gfn(kvm, gfn);
+	memslot = gfn_to_memslot(kvm, gfn);
-	memslot = gfn_to_memslot_unaliased(kvm, gfn);
 	if (memslot && memslot->dirty_bitmap) {
 		unsigned long rel_gfn = gfn - memslot->base_gfn;
 		generic___set_le_bit(rel_gfn, memslot->dirty_bitmap);
 	}
 }
 /*
  * The vCPU has executed a HLT instruction with in-kernel mode enabled.
  */
 void kvm_vcpu_block(struct kvm_vcpu *vcpu)
 {
 	DEFINE_WAIT(wait);
 	for (;;) {
 		prepare_to_wait(&vcpu->wq, &wait, TASK_INTERRUPTIBLE);
 		if (kvm_arch_vcpu_runnable(vcpu)) {
 			set_bit(KVM_REQ_UNHALT, &vcpu->requests);
 			break;
 		}
 		if (kvm_cpu_has_pending_timer(vcpu))
 			break;
 		if (signal_pending(current))
 			break;
 		schedule();
 	}
 	finish_wait(&vcpu->wq, &wait);
 }
 void kvm_resched(struct kvm_vcpu *vcpu)
 {
 	if (!need_resched())
 		return;
 	cond_resched();
 }
 EXPORT_SYMBOL_GPL(kvm_resched);
 void kvm_vcpu_on_spin(struct kvm_vcpu *vcpu)
 {
 	ktime_t expires;
 	DEFINE_WAIT(wait);
 	prepare_to_wait(&vcpu->wq, &wait, TASK_INTERRUPTIBLE);
 	/* Sleep for 100 us, and hope lock-holder got scheduled */
 	expires = ktime_add_ns(ktime_get(), 100000UL);
 	schedule_hrtimeout(&expires, HRTIMER_MODE_ABS);
 	finish_wait(&vcpu->wq, &wait);
 }
 EXPORT_SYMBOL_GPL(kvm_vcpu_on_spin);
 static int kvm_vcpu_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
 {
 	struct kvm_vcpu *vcpu = vma->vm_file->private_data;
 	struct page *page;
 	if (vmf->pgoff == 0)
 		page = virt_to_page(vcpu->run);
 #ifdef CONFIG_X86
 	else if (vmf->pgoff == KVM_PIO_PAGE_OFFSET)
 		page = virt_to_page(vcpu->arch.pio_data);
 #endif
 #ifdef KVM_COALESCED_MMIO_PAGE_OFFSET
 	else if (vmf->pgoff == KVM_COALESCED_MMIO_PAGE_OFFSET)
 		page = virt_to_page(vcpu->kvm->coalesced_mmio_ring);
 #endif
 	else
 		return VM_FAULT_SIGBUS;
 	get_page(page);
 	vmf->page = page;
 	return 0;
 }
 static const struct vm_operations_struct kvm_vcpu_vm_ops = {
 	.fault = kvm_vcpu_fault,
 };
 static int kvm_vcpu_mmap(struct file *file, struct vm_area_struct *vma)
 {
 	vma->vm_ops = &kvm_vcpu_vm_ops;
 	return 0;
 }
 static int kvm_vcpu_release(struct inode *inode, struct file *filp)
 {
 	struct kvm_vcpu *vcpu = filp->private_data;
 	kvm_put_kvm(vcpu->kvm);
 	return 0;
 }
 static struct file_operations kvm_vcpu_fops = {
 	.release        = kvm_vcpu_release,
 	.unlocked_ioctl = kvm_vcpu_ioctl,
 	.compat_ioctl   = kvm_vcpu_ioctl,
 	.mmap           = kvm_vcpu_mmap,
 };
 /*
  * Allocates an inode for the vcpu.
  */
 static int create_vcpu_fd(struct kvm_vcpu *vcpu)
 {
 	return anon_inode_getfd("kvm-vcpu", &kvm_vcpu_fops, vcpu, O_RDWR);
 }
 /*
  * Creates some virtual cpus.  Good luck creating more than one.
  */
 static int kvm_vm_ioctl_create_vcpu(struct kvm *kvm, u32 id)
 {
 	int r;
 	struct kvm_vcpu *vcpu, *v;
 	vcpu = kvm_arch_vcpu_create(kvm, id);
 	if (IS_ERR(vcpu))
 		return PTR_ERR(vcpu);
 	preempt_notifier_init(&vcpu->preempt_notifier, &kvm_preempt_ops);
 	r = kvm_arch_vcpu_setup(vcpu);
 	if (r)
 		return r;
 	mutex_lock(&kvm->lock);
 	if (atomic_read(&kvm->online_vcpus) == KVM_MAX_VCPUS) {
 		r = -EINVAL;
 		goto vcpu_destroy;
 	}
 	kvm_for_each_vcpu(r, v, kvm)
 		if (v->vcpu_id == id) {
 			r = -EEXIST;
 			goto vcpu_destroy;
 		}
 	BUG_ON(kvm->vcpus[atomic_read(&kvm->online_vcpus)]);
 	/* Now it's all set up, let userspace reach it */
 	kvm_get_kvm(kvm);
 	r = create_vcpu_fd(vcpu);
 	if (r < 0) {
 		kvm_put_kvm(kvm);
 		goto vcpu_destroy;
 	}
 	kvm->vcpus[atomic_read(&kvm->online_vcpus)] = vcpu;
 	smp_wmb();
 	atomic_inc(&kvm->online_vcpus);
 #ifdef CONFIG_KVM_APIC_ARCHITECTURE
 	if (kvm->bsp_vcpu_id == id)
 		kvm->bsp_vcpu = vcpu;
 #endif
 	mutex_unlock(&kvm->lock);
 	return r;
 vcpu_destroy:
 	mutex_unlock(&kvm->lock);
 	kvm_arch_vcpu_destroy(vcpu);
 	return r;
 }
 static int kvm_vcpu_ioctl_set_sigmask(struct kvm_vcpu *vcpu, sigset_t *sigset)
 {
 	if (sigset) {
 		sigdelsetmask(sigset, sigmask(SIGKILL)|sigmask(SIGSTOP));
 		vcpu->sigset_active = 1;
 		vcpu->sigset = *sigset;
 	} else
 		vcpu->sigset_active = 0;
 	return 0;
 }
 static long kvm_vcpu_ioctl(struct file *filp,
 			   unsigned int ioctl, unsigned long arg)
 {
 	struct kvm_vcpu *vcpu = filp->private_data;
 	void __user *argp = (void __user *)arg;
 	int r;
 	struct kvm_fpu *fpu = NULL;
 	struct kvm_sregs *kvm_sregs = NULL;
 	if (vcpu->kvm->mm != current->mm)
 		return -EIO;
 #if defined(CONFIG_S390) || defined(CONFIG_PPC)
 	/*
 	 * Special cases: vcpu ioctls that are asynchronous to vcpu execution,
 	 * so vcpu_load() would break it.
 	 */
 	if (ioctl == KVM_S390_INTERRUPT || ioctl == KVM_INTERRUPT)
 		return kvm_arch_vcpu_ioctl(filp, ioctl, arg);
 #endif
 	vcpu_load(vcpu);
 	switch (ioctl) {
 	case KVM_RUN:
 		r = -EINVAL;
 		if (arg)
 			goto out;
 		r = kvm_arch_vcpu_ioctl_run(vcpu, vcpu->run);
 		break;
 	case KVM_GET_REGS: {
 		struct kvm_regs *kvm_regs;
 		r = -ENOMEM;
 		kvm_regs = kzalloc(sizeof(struct kvm_regs), GFP_KERNEL);
 		if (!kvm_regs)
 			goto out;
 		r = kvm_arch_vcpu_ioctl_get_regs(vcpu, kvm_regs);
 		if (r)
 			goto out_free1;
 		r = -EFAULT;
 		if (copy_to_user(argp, kvm_regs, sizeof(struct kvm_regs)))
 			goto out_free1;
 		r = 0;
 out_free1:
 		kfree(kvm_regs);
 		break;
 	}
 	case KVM_SET_REGS: {
 		struct kvm_regs *kvm_regs;
 		r = -ENOMEM;
 		kvm_regs = kzalloc(sizeof(struct kvm_regs), GFP_KERNEL);
 		if (!kvm_regs)
 			goto out;
 		r = -EFAULT;
 		if (copy_from_user(kvm_regs, argp, sizeof(struct kvm_regs)))
 			goto out_free2;
 		r = kvm_arch_vcpu_ioctl_set_regs(vcpu, kvm_regs);
 		if (r)
 			goto out_free2;
 		r = 0;
 out_free2:
 		kfree(kvm_regs);
 		break;
 	}
 	case KVM_GET_SREGS: {
 		kvm_sregs = kzalloc(sizeof(struct kvm_sregs), GFP_KERNEL);
 		r = -ENOMEM;
 		if (!kvm_sregs)
 			goto out;
 		r = kvm_arch_vcpu_ioctl_get_sregs(vcpu, kvm_sregs);
 		if (r)
 			goto out;
 		r = -EFAULT;
 		if (copy_to_user(argp, kvm_sregs, sizeof(struct kvm_sregs)))
 			goto out;
 		r = 0;
 		break;
 	}
 	case KVM_SET_SREGS: {
 		kvm_sregs = kmalloc(sizeof(struct kvm_sregs), GFP_KERNEL);
 		r = -ENOMEM;
 		if (!kvm_sregs)
 			goto out;
 		r = -EFAULT;
 		if (copy_from_user(kvm_sregs, argp, sizeof(struct kvm_sregs)))
 			goto out;
 		r = kvm_arch_vcpu_ioctl_set_sregs(vcpu, kvm_sregs);
 		if (r)
 			goto out;
 		r = 0;
 		break;
 	}
 	case KVM_GET_MP_STATE: {
 		struct kvm_mp_state mp_state;
 		r = kvm_arch_vcpu_ioctl_get_mpstate(vcpu, &mp_state);
 		if (r)
 			goto out;
 		r = -EFAULT;
 		if (copy_to_user(argp, &mp_state, sizeof mp_state))
 			goto out;
 		r = 0;
 		break;
 	}
 	case KVM_SET_MP_STATE: {
 		struct kvm_mp_state mp_state;
 		r = -EFAULT;
 		if (copy_from_user(&mp_state, argp, sizeof mp_state))
 			goto out;
 		r = kvm_arch_vcpu_ioctl_set_mpstate(vcpu, &mp_state);
 		if (r)
 			goto out;
 		r = 0;
 		break;
 	}
 	case KVM_TRANSLATE: {
 		struct kvm_translation tr;
 		r = -EFAULT;
 		if (copy_from_user(&tr, argp, sizeof tr))
 			goto out;
 		r = kvm_arch_vcpu_ioctl_translate(vcpu, &tr);
 		if (r)
 			goto out;
 		r = -EFAULT;
 		if (copy_to_user(argp, &tr, sizeof tr))
 			goto out;
 		r = 0;
 		break;
 	}
 	case KVM_SET_GUEST_DEBUG: {
 		struct kvm_guest_debug dbg;
 		r = -EFAULT;
 		if (copy_from_user(&dbg, argp, sizeof dbg))
 			goto out;
 		r = kvm_arch_vcpu_ioctl_set_guest_debug(vcpu, &dbg);
 		if (r)
 			goto out;
 		r = 0;
 		break;
 	}
 	case KVM_SET_SIGNAL_MASK: {
 		struct kvm_signal_mask __user *sigmask_arg = argp;
 		struct kvm_signal_mask kvm_sigmask;
 		sigset_t sigset, *p;
 		p = NULL;
 		if (argp) {
 			r = -EFAULT;
 			if (copy_from_user(&kvm_sigmask, argp,
 					   sizeof kvm_sigmask))
 				goto out;
 			r = -EINVAL;
 			if (kvm_sigmask.len != sizeof sigset)
 				goto out;
 			r = -EFAULT;
 			if (copy_from_user(&sigset, sigmask_arg->sigset,
 					   sizeof sigset))
 				goto out;
 			p = &sigset;
 		}
 		r = kvm_vcpu_ioctl_set_sigmask(vcpu, p);
 		break;
 	}
 	case KVM_GET_FPU: {
 		fpu = kzalloc(sizeof(struct kvm_fpu), GFP_KERNEL);
 		r = -ENOMEM;
 		if (!fpu)
 			goto out;
 		r = kvm_arch_vcpu_ioctl_get_fpu(vcpu, fpu);
 		if (r)
 			goto out;
 		r = -EFAULT;
 		if (copy_to_user(argp, fpu, sizeof(struct kvm_fpu)))
 			goto out;
 		r = 0;
 		break;
 	}
 	case KVM_SET_FPU: {
 		fpu = kmalloc(sizeof(struct kvm_fpu), GFP_KERNEL);
 		r = -ENOMEM;
 		if (!fpu)
 			goto out;
 		r = -EFAULT;
 		if (copy_from_user(fpu, argp, sizeof(struct kvm_fpu)))
 			goto out;
 		r = kvm_arch_vcpu_ioctl_set_fpu(vcpu, fpu);
 		if (r)
 			goto out;
 		r = 0;
 		break;
 	}
 	default:
 		r = kvm_arch_vcpu_ioctl(filp, ioctl, arg);
 	}
 out:
 	vcpu_put(vcpu);
 	kfree(fpu);
 	kfree(kvm_sregs);
 	return r;
 }
 static long kvm_vm_ioctl(struct file *filp,
 			   unsigned int ioctl, unsigned long arg)
 {
 	struct kvm *kvm = filp->private_data;
 	void __user *argp = (void __user *)arg;
 	int r;
 	if (kvm->mm != current->mm)
 		return -EIO;
 	switch (ioctl) {
 	case KVM_CREATE_VCPU:
 		r = kvm_vm_ioctl_create_vcpu(kvm, arg);
 		if (r < 0)
 			goto out;
 		break;
 	case KVM_SET_USER_MEMORY_REGION: {
 		struct kvm_userspace_memory_region kvm_userspace_mem;
 		r = -EFAULT;
 		if (copy_from_user(&kvm_userspace_mem, argp,
 						sizeof kvm_userspace_mem))
 			goto out;
 		r = kvm_vm_ioctl_set_memory_region(kvm, &kvm_userspace_mem, 1);
 		if (r)
 			goto out;
 		break;
 	}
 	case KVM_GET_DIRTY_LOG: {
 		struct kvm_dirty_log log;
 		r = -EFAULT;
 		if (copy_from_user(&log, argp, sizeof log))
 			goto out;
 		r = kvm_vm_ioctl_get_dirty_log(kvm, &log);
 		if (r)
 			goto out;
 		break;
 	}
 #ifdef KVM_COALESCED_MMIO_PAGE_OFFSET
 	case KVM_REGISTER_COALESCED_MMIO: {
 		struct kvm_coalesced_mmio_zone zone;
 		r = -EFAULT;
 		if (copy_from_user(&zone, argp, sizeof zone))
 			goto out;
 		r = kvm_vm_ioctl_register_coalesced_mmio(kvm, &zone);
 		if (r)
 			goto out;
 		r = 0;
 		break;
 	}
 	case KVM_UNREGISTER_COALESCED_MMIO: {
 		struct kvm_coalesced_mmio_zone zone;
 		r = -EFAULT;
 		if (copy_from_user(&zone, argp, sizeof zone))
 			goto out;
 		r = kvm_vm_ioctl_unregister_coalesced_mmio(kvm, &zone);
 		if (r)
 			goto out;
 		r = 0;
 		break;
 	}
 #endif
 	case KVM_IRQFD: {
 		struct kvm_irqfd data;
 		r = -EFAULT;
 		if (copy_from_user(&data, argp, sizeof data))
 			goto out;
 		r = kvm_irqfd(kvm, data.fd, data.gsi, data.flags);
 		break;
 	}
 	case KVM_IOEVENTFD: {
 		struct kvm_ioeventfd data;
 		r = -EFAULT;
 		if (copy_from_user(&data, argp, sizeof data))
 			goto out;
 		r = kvm_ioeventfd(kvm, &data);
 		break;
 	}
 #ifdef CONFIG_KVM_APIC_ARCHITECTURE
 	case KVM_SET_BOOT_CPU_ID:
 		r = 0;
 		mutex_lock(&kvm->lock);
 		if (atomic_read(&kvm->online_vcpus) != 0)
 			r = -EBUSY;
 		else
 			kvm->bsp_vcpu_id = arg;
 		mutex_unlock(&kvm->lock);
 		break;
 #endif
 	default:
 		r = kvm_arch_vm_ioctl(filp, ioctl, arg);
 		if (r == -ENOTTY)
 			r = kvm_vm_ioctl_assigned_device(kvm, ioctl, arg);
 	}
 out:
 	return r;
 }
 #ifdef CONFIG_COMPAT
 struct compat_kvm_dirty_log {
 	__u32 slot;
 	__u32 padding1;
 	union {
 		compat_uptr_t dirty_bitmap; /* one bit per page */
 		__u64 padding2;
 	};
 };
 static long kvm_vm_compat_ioctl(struct file *filp,
 			   unsigned int ioctl, unsigned long arg)
 {
 	struct kvm *kvm = filp->private_data;
 	int r;
 	if (kvm->mm != current->mm)
 		return -EIO;
 	switch (ioctl) {
 	case KVM_GET_DIRTY_LOG: {
 		struct compat_kvm_dirty_log compat_log;
 		struct kvm_dirty_log log;
 		r = -EFAULT;
 		if (copy_from_user(&compat_log, (void __user *)arg,
 				   sizeof(compat_log)))
 			goto out;
 		log.slot	 = compat_log.slot;
 		log.padding1	 = compat_log.padding1;
 		log.padding2	 = compat_log.padding2;
 		log.dirty_bitmap = compat_ptr(compat_log.dirty_bitmap);
 		r = kvm_vm_ioctl_get_dirty_log(kvm, &log);
 		if (r)
 			goto out;
 		break;
 	}
 	default:
 		r = kvm_vm_ioctl(filp, ioctl, arg);
 	}
 out:
 	return r;
 }
 #endif
 static int kvm_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
 {
 	struct page *page[1];
 	unsigned long addr;
 	int npages;
 	gfn_t gfn = vmf->pgoff;
 	struct kvm *kvm = vma->vm_file->private_data;
 	addr = gfn_to_hva(kvm, gfn);
 	if (kvm_is_error_hva(addr))
 		return VM_FAULT_SIGBUS;
 	npages = get_user_pages(current, current->mm, addr, 1, 1, 0, page,
 				NULL);
 	if (unlikely(npages != 1))
 		return VM_FAULT_SIGBUS;
 	vmf->page = page[0];
 	return 0;
 }
 static const struct vm_operations_struct kvm_vm_vm_ops = {
 	.fault = kvm_vm_fault,
 };
 static int kvm_vm_mmap(struct file *file, struct vm_area_struct *vma)
 {
 	vma->vm_ops = &kvm_vm_vm_ops;
 	return 0;
 }
 static struct file_operations kvm_vm_fops = {
 	.release        = kvm_vm_release,
 	.unlocked_ioctl = kvm_vm_ioctl,
 #ifdef CONFIG_COMPAT
 	.compat_ioctl   = kvm_vm_compat_ioctl,
 #endif
 	.mmap           = kvm_vm_mmap,
 };
 static int kvm_dev_ioctl_create_vm(void)
 {
 	int fd, r;
 	struct kvm *kvm;
 	kvm = kvm_create_vm();
 	if (IS_ERR(kvm))
 		return PTR_ERR(kvm);
 #ifdef KVM_COALESCED_MMIO_PAGE_OFFSET
 	r = kvm_coalesced_mmio_init(kvm);
 	if (r < 0) {
 		kvm_put_kvm(kvm);
 		return r;
 	}
 #endif
 	fd = anon_inode_getfd("kvm-vm", &kvm_vm_fops, kvm, O_RDWR);
 	if (fd < 0)
 		kvm_put_kvm(kvm);
 	return fd;
 }
 static long kvm_dev_ioctl_check_extension_generic(long arg)
 {
 	switch (arg) {
 	case KVM_CAP_USER_MEMORY:
 	case KVM_CAP_DESTROY_MEMORY_REGION_WORKS:
 	case KVM_CAP_JOIN_MEMORY_REGIONS_WORKS:
 #ifdef CONFIG_KVM_APIC_ARCHITECTURE
 	case KVM_CAP_SET_BOOT_CPU_ID:
 #endif
 	case KVM_CAP_INTERNAL_ERROR_DATA:
 		return 1;
 #ifdef CONFIG_HAVE_KVM_IRQCHIP
 	case KVM_CAP_IRQ_ROUTING:
 		return KVM_MAX_IRQ_ROUTES;
 #endif
 	default:
 		break;
 	}
 	return kvm_dev_ioctl_check_extension(arg);
 }
 static long kvm_dev_ioctl(struct file *filp,
 			  unsigned int ioctl, unsigned long arg)
 {
 	long r = -EINVAL;
 	switch (ioctl) {
 	case KVM_GET_API_VERSION:
 		r = -EINVAL;
 		if (arg)
 			goto out;
 		r = KVM_API_VERSION;
 		break;
 	case KVM_CREATE_VM:
 		r = -EINVAL;
 		if (arg)
 			goto out;
 		r = kvm_dev_ioctl_create_vm();
 		break;
 	case KVM_CHECK_EXTENSION:
 		r = kvm_dev_ioctl_check_extension_generic(arg);
 		break;
 	case KVM_GET_VCPU_MMAP_SIZE:
 		r = -EINVAL;
 		if (arg)
 			goto out;
 		r = PAGE_SIZE;     /* struct kvm_run */
 #ifdef CONFIG_X86
 		r += PAGE_SIZE;    /* pio data page */
 #endif
 #ifdef KVM_COALESCED_MMIO_PAGE_OFFSET
 		r += PAGE_SIZE;    /* coalesced mmio ring page */
 #endif
 		break;
 	case KVM_TRACE_ENABLE:
 	case KVM_TRACE_PAUSE:
 	case KVM_TRACE_DISABLE:
 		r = -EOPNOTSUPP;
 		break;
 	default:
 		return kvm_arch_dev_ioctl(filp, ioctl, arg);
 	}
 out:
 	return r;
 }
 static struct file_operations kvm_chardev_ops = {
 	.unlocked_ioctl = kvm_dev_ioctl,
 	.compat_ioctl   = kvm_dev_ioctl,
 };
 static struct miscdevice kvm_dev = {
 	KVM_MINOR,
 	"kvm",
 	&kvm_chardev_ops,
 };
 static void hardware_enable(void *junk)
 {
 	int cpu = raw_smp_processor_id();
 	int r;
 	if (cpumask_test_cpu(cpu, cpus_hardware_enabled))
 		return;
 	cpumask_set_cpu(cpu, cpus_hardware_enabled);
 	r = kvm_arch_hardware_enable(NULL);
 	if (r) {
 		cpumask_clear_cpu(cpu, cpus_hardware_enabled);
 		atomic_inc(&hardware_enable_failed);
 		printk(KERN_INFO "kvm: enabling virtualization on "
 				 "CPU%d failed\n", cpu);
 	}
 }
 static void hardware_disable(void *junk)
 {
 	int cpu = raw_smp_processor_id();
 	if (!cpumask_test_cpu(cpu, cpus_hardware_enabled))
 		return;
 	cpumask_clear_cpu(cpu, cpus_hardware_enabled);
 	kvm_arch_hardware_disable(NULL);
 }
 static void hardware_disable_all_nolock(void)
 {
 	BUG_ON(!kvm_usage_count);
 	kvm_usage_count--;
 	if (!kvm_usage_count)
 		on_each_cpu(hardware_disable, NULL, 1);
 }
 static void hardware_disable_all(void)
 {
 	spin_lock(&kvm_lock);
 	hardware_disable_all_nolock();
 	spin_unlock(&kvm_lock);
 }
 static int hardware_enable_all(void)
 {
 	int r = 0;
 	spin_lock(&kvm_lock);
 	kvm_usage_count++;
 	if (kvm_usage_count == 1) {
 		atomic_set(&hardware_enable_failed, 0);
 		on_each_cpu(hardware_enable, NULL, 1);
 		if (atomic_read(&hardware_enable_failed)) {
 			hardware_disable_all_nolock();
 			r = -EBUSY;
 		}
 	}
 	spin_unlock(&kvm_lock);
 	return r;
 }
 static int kvm_cpu_hotplug(struct notifier_block *notifier, unsigned long val,
 			   void *v)
 {
 	int cpu = (long)v;
 	if (!kvm_usage_count)
 		return NOTIFY_OK;
 	val &= ~CPU_TASKS_FROZEN;
 	switch (val) {
 	case CPU_DYING:
 		printk(KERN_INFO "kvm: disabling virtualization on CPU%d\n",
 		       cpu);
 		hardware_disable(NULL);
 		break;
 	case CPU_ONLINE:
 		printk(KERN_INFO "kvm: enabling virtualization on CPU%d\n",
 		       cpu);
 		smp_call_function_single(cpu, hardware_enable, NULL, 1);
 		break;
 	}
 	return NOTIFY_OK;
 }
 asmlinkage void kvm_handle_fault_on_reboot(void)
 {
 	if (kvm_rebooting)
 		/* spin while reset goes on */
 		while (true)
 			;
 	/* Fault while not rebooting.  We want the trace. */
 	BUG();
 }
 EXPORT_SYMBOL_GPL(kvm_handle_fault_on_reboot);
 static int kvm_reboot(struct notifier_block *notifier, unsigned long val,
 		      void *v)
 {
 	/*
 	 * Some (well, at least mine) BIOSes hang on reboot if
 	 * in vmx root mode.
 	 *
 	 * And Intel TXT required VMX off for all cpu when system shutdown.
 	 */
 	printk(KERN_INFO "kvm: exiting hardware virtualization\n");
 	kvm_rebooting = true;
 	on_each_cpu(hardware_disable, NULL, 1);
 	return NOTIFY_OK;
 }
 static struct notifier_block kvm_reboot_notifier = {
 	.notifier_call = kvm_reboot,
 	.priority = 0,
 };
 static void kvm_io_bus_destroy(struct kvm_io_bus *bus)
 {
 	int i;
 	for (i = 0; i < bus->dev_count; i++) {
 		struct kvm_io_device *pos = bus->devs[i];
 		kvm_iodevice_destructor(pos);
 	}
 	kfree(bus);
 }
 /* kvm_io_bus_write - called under kvm->slots_lock */
 int kvm_io_bus_write(struct kvm *kvm, enum kvm_bus bus_idx, gpa_t addr,
 		     int len, const void *val)
 {
 	int i;
 	struct kvm_io_bus *bus;
 	bus = srcu_dereference(kvm->buses[bus_idx], &kvm->srcu);
 	for (i = 0; i < bus->dev_count; i++)
 		if (!kvm_iodevice_write(bus->devs[i], addr, len, val))
 			return 0;
 	return -EOPNOTSUPP;
 }
 /* kvm_io_bus_read - called under kvm->slots_lock */
 int kvm_io_bus_read(struct kvm *kvm, enum kvm_bus bus_idx, gpa_t addr,
 		    int len, void *val)
 {
 	int i;
 	struct kvm_io_bus *bus;
 	bus = srcu_dereference(kvm->buses[bus_idx], &kvm->srcu);
 	for (i = 0; i < bus->dev_count; i++)
 		if (!kvm_iodevice_read(bus->devs[i], addr, len, val))
 			return 0;
 	return -EOPNOTSUPP;
 }
 /* Caller must hold slots_lock. */
 int kvm_io_bus_register_dev(struct kvm *kvm, enum kvm_bus bus_idx,
 			    struct kvm_io_device *dev)
 {
 	struct kvm_io_bus *new_bus, *bus;
 	bus = kvm->buses[bus_idx];
 	if (bus->dev_count > NR_IOBUS_DEVS-1)
 		return -ENOSPC;
 	new_bus = kzalloc(sizeof(struct kvm_io_bus), GFP_KERNEL);
 	if (!new_bus)
 		return -ENOMEM;
 	memcpy(new_bus, bus, sizeof(struct kvm_io_bus));
 	new_bus->devs[new_bus->dev_count++] = dev;
 	rcu_assign_pointer(kvm->buses[bus_idx], new_bus);
 	synchronize_srcu_expedited(&kvm->srcu);
 	kfree(bus);
 	return 0;
 }
 /* Caller must hold slots_lock. */
 int kvm_io_bus_unregister_dev(struct kvm *kvm, enum kvm_bus bus_idx,
 			      struct kvm_io_device *dev)
 {
 	int i, r;
 	struct kvm_io_bus *new_bus, *bus;
 	new_bus = kzalloc(sizeof(struct kvm_io_bus), GFP_KERNEL);
 	if (!new_bus)
 		return -ENOMEM;
 	bus = kvm->buses[bus_idx];
 	memcpy(new_bus, bus, sizeof(struct kvm_io_bus));
 	r = -ENOENT;
 	for (i = 0; i < new_bus->dev_count; i++)
 		if (new_bus->devs[i] == dev) {
 			r = 0;
 			new_bus->devs[i] = new_bus->devs[--new_bus->dev_count];
 			break;
 		}
 	if (r) {
 		kfree(new_bus);
 		return r;
 	}
 	rcu_assign_pointer(kvm->buses[bus_idx], new_bus);
 	synchronize_srcu_expedited(&kvm->srcu);
 	kfree(bus);
 	return r;
 }
 static struct notifier_block kvm_cpu_notifier = {
 	.notifier_call = kvm_cpu_hotplug,
 	.priority = 20, /* must be > scheduler priority */
 };
 static int vm_stat_get(void *_offset, u64 *val)
 {
 	unsigned offset = (long)_offset;
 	struct kvm *kvm;
 	*val = 0;
 	spin_lock(&kvm_lock);
 	list_for_each_entry(kvm, &vm_list, vm_list)
 		*val += *(u32 *)((void *)kvm + offset);
 	spin_unlock(&kvm_lock);
 	return 0;
 }
 DEFINE_SIMPLE_ATTRIBUTE(vm_stat_fops, vm_stat_get, NULL, "%llu\n");
 static int vcpu_stat_get(void *_offset, u64 *val)
 {
 	unsigned offset = (long)_offset;
 	struct kvm *kvm;
 	struct kvm_vcpu *vcpu;
 	int i;
 	*val = 0;
 	spin_lock(&kvm_lock);
 	list_for_each_entry(kvm, &vm_list, vm_list)
 		kvm_for_each_vcpu(i, vcpu, kvm)
 			*val += *(u32 *)((void *)vcpu + offset);
 	spin_unlock(&kvm_lock);
 	return 0;
 }
 DEFINE_SIMPLE_ATTRIBUTE(vcpu_stat_fops, vcpu_stat_get, NULL, "%llu\n");
 static const struct file_operations *stat_fops[] = {
 	[KVM_STAT_VCPU] = &vcpu_stat_fops,
 	[KVM_STAT_VM]   = &vm_stat_fops,
 };
 static void kvm_init_debug(void)
 {
 	struct kvm_stats_debugfs_item *p;
 	kvm_debugfs_dir = debugfs_create_dir("kvm", NULL);
 	for (p = debugfs_entries; p->name; ++p)
 		p->dentry = debugfs_create_file(p->name, 0444, kvm_debugfs_dir,
 						(void *)(long)p->offset,
 						stat_fops[p->kind]);
 }
 static void kvm_exit_debug(void)
 {
 	struct kvm_stats_debugfs_item *p;
 	for (p = debugfs_entries; p->name; ++p)
 		debugfs_remove(p->dentry);
 	debugfs_remove(kvm_debugfs_dir);
 }
 static int kvm_suspend(struct sys_device *dev, pm_message_t state)
 {
 	if (kvm_usage_count)
 		hardware_disable(NULL);
 	return 0;
 }
 static int kvm_resume(struct sys_device *dev)
 {
 	if (kvm_usage_count)
 		hardware_enable(NULL);
 	return 0;
 }
 static struct sysdev_class kvm_sysdev_class = {
 	.name = "kvm",
 	.suspend = kvm_suspend,
 	.resume = kvm_resume,
 };
 static struct sys_device kvm_sysdev = {
 	.id = 0,
 	.cls = &kvm_sysdev_class,
 };
 struct page *bad_page;
 pfn_t bad_pfn;
 static inline
 struct kvm_vcpu *preempt_notifier_to_vcpu(struct preempt_notifier *pn)
 {
 	return container_of(pn, struct kvm_vcpu, preempt_notifier);
 }
 static void kvm_sched_in(struct preempt_notifier *pn, int cpu)
 {
 	struct kvm_vcpu *vcpu = preempt_notifier_to_vcpu(pn);
 	kvm_arch_vcpu_load(vcpu, cpu);
 }
 static void kvm_sched_out(struct preempt_notifier *pn,
 			  struct task_struct *next)
 {
 	struct kvm_vcpu *vcpu = preempt_notifier_to_vcpu(pn);
 	kvm_arch_vcpu_put(vcpu);
 }
 int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align,
 		  struct module *module)
 {
 	int r;
 	int cpu;
 	r = kvm_arch_init(opaque);
 	if (r)
 		goto out_fail;
 	bad_page = alloc_page(GFP_KERNEL | __GFP_ZERO);
 	if (bad_page == NULL) {
 		r = -ENOMEM;
 		goto out;
 	}
 	bad_pfn = page_to_pfn(bad_page);
 	hwpoison_page = alloc_page(GFP_KERNEL | __GFP_ZERO);
 	if (hwpoison_page == NULL) {
 		r = -ENOMEM;
 		goto out_free_0;
 	}
 	hwpoison_pfn = page_to_pfn(hwpoison_page);
 	if (!zalloc_cpumask_var(&cpus_hardware_enabled, GFP_KERNEL)) {
 		r = -ENOMEM;
 		goto out_free_0;
 	}
 	r = kvm_arch_hardware_setup();
 	if (r < 0)
 		goto out_free_0a;
 	for_each_online_cpu(cpu) {
 		smp_call_function_single(cpu,
 				kvm_arch_check_processor_compat,
 				&r, 1);
 		if (r < 0)
 			goto out_free_1;
 	}
 	r = register_cpu_notifier(&kvm_cpu_notifier);
 	if (r)
 		goto out_free_2;
 	register_reboot_notifier(&kvm_reboot_notifier);
 	r = sysdev_class_register(&kvm_sysdev_class);
 	if (r)
 		goto out_free_3;
 	r = sysdev_register(&kvm_sysdev);
 	if (r)
 		goto out_free_4;
 	/* A kmem cache lets us meet the alignment requirements of fx_save. */
 	if (!vcpu_align)
 		vcpu_align = __alignof__(struct kvm_vcpu);
 	kvm_vcpu_cache = kmem_cache_create("kvm_vcpu", vcpu_size, vcpu_align,
 					   0, NULL);
 	if (!kvm_vcpu_cache) {
 		r = -ENOMEM;
 		goto out_free_5;
 	}
 	kvm_chardev_ops.owner = module;
 	kvm_vm_fops.owner = module;
 	kvm_vcpu_fops.owner = module;
 	r = misc_register(&kvm_dev);
 	if (r) {
 		printk(KERN_ERR "kvm: misc device register failed\n");
 		goto out_free;
 	}
 	kvm_preempt_ops.sched_in = kvm_sched_in;
 	kvm_preempt_ops.sched_out = kvm_sched_out;
 	kvm_init_debug();
 	return 0;
 out_free:
 	kmem_cache_destroy(kvm_vcpu_cache);
 out_free_5:
 	sysdev_unregister(&kvm_sysdev);
 out_free_4:
 	sysdev_class_unregister(&kvm_sysdev_class);
 out_free_3:
 	unregister_reboot_notifier(&kvm_reboot_notifier);
 	unregister_cpu_notifier(&kvm_cpu_notifier);
 out_free_2:
 out_free_1:
 	kvm_arch_hardware_unsetup();
 out_free_0a:
 	free_cpumask_var(cpus_hardware_enabled);
 out_free_0:
 	if (hwpoison_page)
 		__free_page(hwpoison_page);
 	__free_page(bad_page);
 out:
 	kvm_arch_exit();
 out_fail:
 	return r;
 }
 EXPORT_SYMBOL_GPL(kvm_init);
 void kvm_exit(void)
 {
 	kvm_exit_debug();
 	misc_deregister(&kvm_dev);
 	kmem_cache_destroy(kvm_vcpu_cache);
 	sysdev_unregister(&kvm_sysdev);
 	sysdev_class_unregister(&kvm_sysdev_class);
 	unregister_reboot_notifier(&kvm_reboot_notifier);
 	unregister_cpu_notifier(&kvm_cpu_notifier);
 	on_each_cpu(hardware_disable, NULL, 1);
 	kvm_arch_hardware_unsetup();
 	kvm_arch_exit();
 	free_cpumask_var(cpus_hardware_enabled);
 	__free_page(hwpoison_page);
 	__free_page(bad_page);
 }
 EXPORT_SYMBOL_GPL(kvm_exit);