09 Feb, 2017

1 commit

  • commit 11e3b725cfc282efe9d4a354153e99d86a16af08 upstream.

    Update the ARMv8 Crypto Extensions and the plain NEON AES implementations
    in CBC and CTR modes to return the next IV back to the skcipher API client.
    This is necessary for chaining to work correctly.

    Note that for CTR, this is only done if the request is a round multiple of
    the block size, since otherwise, chaining is impossible anyway.

    Signed-off-by: Ard Biesheuvel
    Signed-off-by: Herbert Xu
    Signed-off-by: Greg Kroah-Hartman

    Ard Biesheuvel
     

12 Jan, 2017

7 commits

  • commit 1803b9a52c4e5a5dbb8a27126f6bc06939359753 upstream.

    The core AES cipher implementation that uses ARMv8 Crypto Extensions
    instructions erroneously loads the round keys as 64-bit quantities,
    which causes the algorithm to fail when built for big endian. In
    addition, the key schedule generation routine fails to take endianness
    into account as well, when loading the combining the input key with
    the round constants. So fix both issues.

    Fixes: 12ac3efe74f8 ("arm64/crypto: use crypto instructions to generate AES key schedule")
    Signed-off-by: Ard Biesheuvel
    Signed-off-by: Herbert Xu
    Signed-off-by: Greg Kroah-Hartman

    Ard Biesheuvel
     
  • commit caf4b9e2b326cc2a5005a5c557274306536ace61 upstream.

    Emit the XTS tweak literal constants in the appropriate order for a
    single 128-bit scalar literal load.

    Fixes: 49788fe2a128 ("arm64/crypto: AES-ECB/CBC/CTR/XTS using ARMv8 NEON and Crypto Extensions")
    Signed-off-by: Ard Biesheuvel
    Signed-off-by: Herbert Xu
    Signed-off-by: Greg Kroah-Hartman

    Ard Biesheuvel
     
  • commit ee71e5f1e7d25543ee63a80451871f8985b8d431 upstream.

    The SHA1 digest is an array of 5 32-bit quantities, so we should refer
    to them as such in order for this code to work correctly when built for
    big endian. So replace 16 byte scalar loads and stores with 4x4 vector
    ones where appropriate.

    Fixes: 2c98833a42cd ("arm64/crypto: SHA-1 using ARMv8 Crypto Extensions")
    Signed-off-by: Ard Biesheuvel
    Signed-off-by: Herbert Xu
    Signed-off-by: Greg Kroah-Hartman

    Ard Biesheuvel
     
  • commit a2c435cc99862fd3d165e1b66bf48ac72c839c62 upstream.

    The AES implementation using pure NEON instructions relies on the generic
    AES key schedule generation routines, which store the round keys as arrays
    of 32-bit quantities stored in memory using native endianness. This means
    we should refer to these round keys using 4x4 loads rather than 16x1 loads.
    In addition, the ShiftRows tables are loading using a single scalar load,
    which is also affected by endianness, so emit these tables in the correct
    order depending on whether we are building for big endian or not.

    Fixes: 49788fe2a128 ("arm64/crypto: AES-ECB/CBC/CTR/XTS using ARMv8 NEON and Crypto Extensions")
    Signed-off-by: Ard Biesheuvel
    Signed-off-by: Herbert Xu
    Signed-off-by: Greg Kroah-Hartman

    Ard Biesheuvel
     
  • commit 56e4e76c68fcb51547b5299e5b66a135935ff414 upstream.

    The AES-CCM implementation that uses ARMv8 Crypto Extensions instructions
    refers to the AES round keys as pairs of 64-bit quantities, which causes
    failures when building the code for big endian. In addition, it byte swaps
    the input counter unconditionally, while this is only required for little
    endian builds. So fix both issues.

    Fixes: 12ac3efe74f8 ("arm64/crypto: use crypto instructions to generate AES key schedule")
    Signed-off-by: Ard Biesheuvel
    Signed-off-by: Herbert Xu
    Signed-off-by: Greg Kroah-Hartman

    Ard Biesheuvel
     
  • commit 9c433ad5083fd4a4a3c721d86cbfbd0b2a2326a5 upstream.

    The GHASH key and digest are both pairs of 64-bit quantities, but the
    GHASH code does not always refer to them as such, causing failures when
    built for big endian. So replace the 16x1 loads and stores with 2x8 ones.

    Fixes: b913a6404ce2 ("arm64/crypto: improve performance of GHASH algorithm")
    Signed-off-by: Ard Biesheuvel
    Signed-off-by: Herbert Xu
    Signed-off-by: Greg Kroah-Hartman

    Ard Biesheuvel
     
  • commit 174122c39c369ed924d2608fc0be0171997ce800 upstream.

    The SHA256 digest is an array of 8 32-bit quantities, so we should refer
    to them as such in order for this code to work correctly when built for
    big endian. So replace 16 byte scalar loads and stores with 4x32 vector
    ones where appropriate.

    Fixes: 6ba6c74dfc6b ("arm64/crypto: SHA-224/SHA-256 using ARMv8 Crypto Extensions")
    Signed-off-by: Ard Biesheuvel
    Signed-off-by: Herbert Xu
    Signed-off-by: Greg Kroah-Hartman

    Ard Biesheuvel
     

13 Sep, 2016

1 commit

  • The AES-CTR glue code avoids calling into the blkcipher API for the
    tail portion of the walk, by comparing the remainder of walk.nbytes
    modulo AES_BLOCK_SIZE with the residual nbytes, and jumping straight
    into the tail processing block if they are equal. This tail processing
    block checks whether nbytes != 0, and does nothing otherwise.

    However, in case of an allocation failure in the blkcipher layer, we
    may enter this code with walk.nbytes == 0, while nbytes > 0. In this
    case, we should not dereference the source and destination pointers,
    since they may be NULL. So instead of checking for nbytes != 0, check
    for (walk.nbytes % AES_BLOCK_SIZE) != 0, which implies the former in
    non-error conditions.

    Fixes: 49788fe2a128 ("arm64/crypto: AES-ECB/CBC/CTR/XTS using ARMv8 NEON and Crypto Extensions")
    Cc: stable@vger.kernel.org
    Reported-by: xiakaixu
    Signed-off-by: Ard Biesheuvel
    Signed-off-by: Herbert Xu

    Ard Biesheuvel
     

18 Mar, 2016

1 commit

  • Pull crypto update from Herbert Xu:
    "Here is the crypto update for 4.6:

    API:
    - Convert remaining crypto_hash users to shash or ahash, also convert
    blkcipher/ablkcipher users to skcipher.
    - Remove crypto_hash interface.
    - Remove crypto_pcomp interface.
    - Add crypto engine for async cipher drivers.
    - Add akcipher documentation.
    - Add skcipher documentation.

    Algorithms:
    - Rename crypto/crc32 to avoid name clash with lib/crc32.
    - Fix bug in keywrap where we zero the wrong pointer.

    Drivers:
    - Support T5/M5, T7/M7 SPARC CPUs in n2 hwrng driver.
    - Add PIC32 hwrng driver.
    - Support BCM6368 in bcm63xx hwrng driver.
    - Pack structs for 32-bit compat users in qat.
    - Use crypto engine in omap-aes.
    - Add support for sama5d2x SoCs in atmel-sha.
    - Make atmel-sha available again.
    - Make sahara hashing available again.
    - Make ccp hashing available again.
    - Make sha1-mb available again.
    - Add support for multiple devices in ccp.
    - Improve DMA performance in caam.
    - Add hashing support to rockchip"

    * 'linus' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6: (116 commits)
    crypto: qat - remove redundant arbiter configuration
    crypto: ux500 - fix checks of error code returned by devm_ioremap_resource()
    crypto: atmel - fix checks of error code returned by devm_ioremap_resource()
    crypto: qat - Change the definition of icp_qat_uof_regtype
    hwrng: exynos - use __maybe_unused to hide pm functions
    crypto: ccp - Add abstraction for device-specific calls
    crypto: ccp - CCP versioning support
    crypto: ccp - Support for multiple CCPs
    crypto: ccp - Remove check for x86 family and model
    crypto: ccp - memset request context to zero during import
    lib/mpi: use "static inline" instead of "extern inline"
    lib/mpi: avoid assembler warning
    hwrng: bcm63xx - fix non device tree compatibility
    crypto: testmgr - allow rfc3686 aes-ctr variants in fips mode.
    crypto: qat - The AE id should be less than the maximal AE number
    lib/mpi: Endianness fix
    crypto: rockchip - add hash support for crypto engine in rk3288
    crypto: xts - fix compile errors
    crypto: doc - add skcipher API documentation
    crypto: doc - update AEAD AD handling
    ...

    Linus Torvalds
     

17 Feb, 2016

2 commits

  • Commit 28856a9e52c7 missed the addition of the crypto/xts.h include file
    for different architecture-specific AES implementations.

    Signed-off-by: Stephan Mueller
    Signed-off-by: Herbert Xu

    Stephan Mueller
     
  • The patch centralizes the XTS key check logic into the service function
    xts_check_key which is invoked from the different XTS implementations.
    With this, the XTS implementations in ARM, ARM64, PPC and S390 have now
    a sanity check for the XTS keys similar to the other arches.

    In addition, this service function received a check to ensure that the
    key != the tweak key which is mandated by FIPS 140-2 IG A.9. As the
    check is not present in the standards defining XTS, it is only enforced
    in FIPS mode of the kernel.

    Signed-off-by: Stephan Mueller
    Signed-off-by: Herbert Xu

    Stephan Mueller
     

15 Feb, 2016

1 commit


18 Nov, 2015

1 commit

  • The asynchronous, merged implementations of AES in CBC, CTR and XTS
    modes are preferred when available (i.e., when instantiating ablkciphers
    explicitly). However, the synchronous core AES cipher combined with the
    generic CBC mode implementation will produce a 'cbc(aes)' blkcipher that
    is callable asynchronously as well. To prevent this implementation from
    being used when the accelerated asynchronous implemenation is also
    available, lower its priority to 250 (i.e., below the asynchronous
    module's priority of 300).

    Signed-off-by: Ard Biesheuvel
    Signed-off-by: Catalin Marinas

    Ard Biesheuvel
     

17 Aug, 2015

1 commit


17 Jul, 2015

1 commit


19 Jun, 2015

1 commit


28 May, 2015

1 commit


07 May, 2015

3 commits


23 Apr, 2015

1 commit


17 Apr, 2015

1 commit

  • Pull arm64 updates from Will Deacon:
    "Here are the core arm64 updates for 4.1.

    Highlights include a significant rework to head.S (allowing us to boot
    on machines with physical memory at a really high address), an AES
    performance boost on Cortex-A57 and the ability to run a 32-bit
    userspace with 64k pages (although this requires said userspace to be
    built with a recent binutils).

    The head.S rework spilt over into KVM, so there are some changes under
    arch/arm/ which have been acked by Marc Zyngier (KVM co-maintainer).
    In particular, the linker script changes caused us some issues in
    -next, so there are a few merge commits where we had to apply fixes on
    top of a stable branch.

    Other changes include:

    - AES performance boost for Cortex-A57
    - AArch32 (compat) userspace with 64k pages
    - Cortex-A53 erratum workaround for #845719
    - defconfig updates (new platforms, PCI, ...)"

    * tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux: (39 commits)
    arm64: fix midr range for Cortex-A57 erratum 832075
    arm64: errata: add workaround for cortex-a53 erratum #845719
    arm64: Use bool function return values of true/false not 1/0
    arm64: defconfig: updates for 4.1
    arm64: Extract feature parsing code from cpu_errata.c
    arm64: alternative: Allow immediate branch as alternative instruction
    arm64: insn: Add aarch64_insn_decode_immediate
    ARM: kvm: round HYP section to page size instead of log2 upper bound
    ARM: kvm: assert on HYP section boundaries not actual code size
    arm64: head.S: ensure idmap_t0sz is visible
    arm64: pmu: add support for interrupt-affinity property
    dt: pmu: extend ARM PMU binding to allow for explicit interrupt affinity
    arm64: head.S: ensure visibility of page tables
    arm64: KVM: use ID map with increased VA range if required
    arm64: mm: increase VA range of identity map
    ARM: kvm: implement replacement for ld's LOG2CEIL()
    arm64: proc: remove unused cpu_get_pgd macro
    arm64: enforce x1|x2|x3 == 0 upon kernel entry as per boot protocol
    arm64: remove __calc_phys_offset
    arm64: merge __enable_mmu and __turn_mmu_on
    ...

    Linus Torvalds
     

16 Apr, 2015

1 commit

  • Pull crypto update from Herbert Xu:
    "Here is the crypto update for 4.1:

    New interfaces:
    - user-space interface for AEAD
    - user-space interface for RNG (i.e., pseudo RNG)

    New hashes:
    - ARMv8 SHA1/256
    - ARMv8 AES
    - ARMv8 GHASH
    - ARM assembler and NEON SHA256
    - MIPS OCTEON SHA1/256/512
    - MIPS img-hash SHA1/256 and MD5
    - Power 8 VMX AES/CBC/CTR/GHASH
    - PPC assembler AES, SHA1/256 and MD5
    - Broadcom IPROC RNG driver

    Cleanups/fixes:
    - prevent internal helper algos from being exposed to user-space
    - merge common code from assembly/C SHA implementations
    - misc fixes"

    * git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6: (169 commits)
    crypto: arm - workaround for building with old binutils
    crypto: arm/sha256 - avoid sha256 code on ARMv7-M
    crypto: x86/sha512_ssse3 - move SHA-384/512 SSSE3 implementation to base layer
    crypto: x86/sha256_ssse3 - move SHA-224/256 SSSE3 implementation to base layer
    crypto: x86/sha1_ssse3 - move SHA-1 SSSE3 implementation to base layer
    crypto: arm64/sha2-ce - move SHA-224/256 ARMv8 implementation to base layer
    crypto: arm64/sha1-ce - move SHA-1 ARMv8 implementation to base layer
    crypto: arm/sha2-ce - move SHA-224/256 ARMv8 implementation to base layer
    crypto: arm/sha256 - move SHA-224/256 ASM/NEON implementation to base layer
    crypto: arm/sha1-ce - move SHA-1 ARMv8 implementation to base layer
    crypto: arm/sha1_neon - move SHA-1 NEON implementation to base layer
    crypto: arm/sha1 - move SHA-1 ARM asm implementation to base layer
    crypto: sha512-generic - move to generic glue implementation
    crypto: sha256-generic - move to generic glue implementation
    crypto: sha1-generic - move to generic glue implementation
    crypto: sha512 - implement base layer for SHA-512
    crypto: sha256 - implement base layer for SHA-256
    crypto: sha1 - implement base layer for SHA-1
    crypto: api - remove instance when test failed
    crypto: api - Move alg ref count init to crypto_check_alg
    ...

    Linus Torvalds
     

10 Apr, 2015

2 commits


31 Mar, 2015

1 commit


19 Mar, 2015

1 commit

  • This changes the AES core transform implementations to issue aese/aesmc
    (and aesd/aesimc) in pairs. This enables a micro-architectural optimization
    in recent Cortex-A5x cores that improves performance by 50-90%.

    Measured performance in cycles per byte (Cortex-A57):

    CBC enc CBC dec CTR
    before 3.64 1.34 1.32
    after 1.95 0.85 0.93

    Note that this results in a ~5% performance decrease for older cores.

    Signed-off-by: Ard Biesheuvel
    Signed-off-by: Will Deacon

    Ard Biesheuvel
     

27 Feb, 2015

1 commit

  • This patch increases the interleave factor for parallel AES modes
    to 4x. This improves performance on Cortex-A57 by ~35%. This is
    due to the 3-cycle latency of AES instructions on the A57's
    relatively deep pipeline (compared to Cortex-A53 where the AES
    instruction latency is only 2 cycles).

    At the same time, disable inline expansion of the core AES functions,
    as the performance benefit of this feature is negligible.

    Measured on AMD Seattle (using tcrypt.ko mode=500 sec=1):

    Baseline (2x interleave, inline expansion)
    ------------------------------------------
    testing speed of async cbc(aes) (cbc-aes-ce) decryption
    test 4 (128 bit key, 8192 byte blocks): 95545 operations in 1 seconds
    test 14 (256 bit key, 8192 byte blocks): 68496 operations in 1 seconds

    This patch (4x interleave, no inline expansion)
    -----------------------------------------------
    testing speed of async cbc(aes) (cbc-aes-ce) decryption
    test 4 (128 bit key, 8192 byte blocks): 124735 operations in 1 seconds
    test 14 (256 bit key, 8192 byte blocks): 92328 operations in 1 seconds

    Signed-off-by: Ard Biesheuvel
    Signed-off-by: Catalin Marinas

    Ard Biesheuvel
     

14 Dec, 2014

1 commit

  • Pull crypto update from Herbert Xu:
    - The crypto API is now documented :)
    - Disallow arbitrary module loading through crypto API.
    - Allow get request with empty driver name through crypto_user.
    - Allow speed testing of arbitrary hash functions.
    - Add caam support for ctr(aes), gcm(aes) and their derivatives.
    - nx now supports concurrent hashing properly.
    - Add sahara support for SHA1/256.
    - Add ARM64 version of CRC32.
    - Misc fixes.

    * git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6: (77 commits)
    crypto: tcrypt - Allow speed testing of arbitrary hash functions
    crypto: af_alg - add user space interface for AEAD
    crypto: qat - fix problem with coalescing enable logic
    crypto: sahara - add support for SHA1/256
    crypto: sahara - replace tasklets with kthread
    crypto: sahara - add support for i.MX53
    crypto: sahara - fix spinlock initialization
    crypto: arm - replace memset by memzero_explicit
    crypto: powerpc - replace memset by memzero_explicit
    crypto: sha - replace memset by memzero_explicit
    crypto: sparc - replace memset by memzero_explicit
    crypto: algif_skcipher - initialize upon init request
    crypto: algif_skcipher - removed unneeded code
    crypto: algif_skcipher - Fixed blocking recvmsg
    crypto: drbg - use memzero_explicit() for clearing sensitive data
    crypto: drbg - use MODULE_ALIAS_CRYPTO
    crypto: include crypto- module prefix in template
    crypto: user - add MODULE_ALIAS
    crypto: sha-mb - remove a bogus NULL check
    crytpo: qat - Fix 64 bytes requests
    ...

    Linus Torvalds
     

24 Nov, 2014

1 commit


20 Nov, 2014

1 commit

  • This module registers a crc32 algorithm and a crc32c algorithm
    that use the optional CRC32 and CRC32C instructions in ARMv8.

    Tested on AMD Seattle.

    Improvement compared to crc32c-generic algorithm:
    TCRYPT CRC32C speed test shows ~450% speedup.
    Simple dd write tests to btrfs filesystem show ~30% speedup.

    Signed-off-by: Yazen Ghannam
    Acked-by: Steve Capper
    Acked-by: Ard Biesheuvel
    Signed-off-by: Herbert Xu

    Yazen Ghannam
     

07 Nov, 2014

1 commit


26 Aug, 2014

1 commit

  • Originally found by cppcheck:

    [arch/arm64/crypto/sha2-ce-glue.c:153]: (warning) Assignment of
    function parameter has no effect outside the function. Did you
    forget dereferencing it?

    Updating data by blocks * SHA256_BLOCK_SIZE at the end of
    sha2_finup is redundant code and can be removed.

    Acked-by: Ard Biesheuvel
    Signed-off-by: Colin Ian King
    Signed-off-by: Will Deacon

    Colin Ian King
     

05 Aug, 2014

1 commit

  • Pull arm64 updates from Will Deacon:
    "Once again, Catalin's off on holiday and I'm looking after the arm64
    tree. Please can you pull the following arm64 updates for 3.17?

    Note that this branch also includes the new GICv3 driver (merged via a
    stable tag from Jason's irqchip tree), since there is a fix for older
    binutils on top.

    Changes include:
    - context tracking support (NO_HZ_FULL) which narrowly missed 3.16
    - vDSO layout rework following Andy's work on x86
    - TEXT_OFFSET fuzzing for bootloader testing
    - /proc/cpuinfo tidy-up
    - preliminary work to support 48-bit virtual addresses, but this is
    currently disabled until KVM has been ported to use it (the patches
    do, however, bring some nice clean-up)
    - boot-time CPU sanity checks (especially useful on heterogenous
    systems)
    - support for syscall auditing
    - support for CC_STACKPROTECTOR
    - defconfig updates"

    * tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux: (55 commits)
    arm64: add newline to I-cache policy string
    Revert "arm64: dmi: Add SMBIOS/DMI support"
    arm64: fpsimd: fix a typo in fpsimd_save_partial_state ENDPROC
    arm64: don't call break hooks for BRK exceptions from EL0
    arm64: defconfig: enable devtmpfs mount option
    arm64: vdso: fix build error when switching from LE to BE
    arm64: defconfig: add virtio support for running as a kvm guest
    arm64: gicv3: Allow GICv3 compilation with older binutils
    arm64: fix soft lockup due to large tlb flush range
    arm64/crypto: fix makefile rule for aes-glue-%.o
    arm64: Do not invoke audit_syscall_* functions if !CONFIG_AUDIT_SYSCALL
    arm64: Fix barriers used for page table modifications
    arm64: Add support for 48-bit VA space with 64KB page configuration
    arm64: asm/pgtable.h pmd/pud definitions clean-up
    arm64: Determine the vmalloc/vmemmap space at build time based on VA_BITS
    arm64: Clean up the initial page table creation in head.S
    arm64: Remove asm/pgtable-*level-types.h files
    arm64: Remove asm/pgtable-*level-hwdef.h files
    arm64: Convert bool ARM64_x_LEVELS to int ARM64_PGTABLE_LEVELS
    arm64: mm: Implement 4 levels of translation tables
    ...

    Linus Torvalds
     

29 Jul, 2014

1 commit


28 Jul, 2014

1 commit

  • cryptsetup fails on arm64 when using kernel encryption via AF_ALG socket.
    See https://bugzilla.redhat.com/show_bug.cgi?id=1122937

    The bug is caused by incorrect handling of unaligned data in
    arch/arm64/crypto/aes-glue.c. Cryptsetup creates a buffer that is aligned
    on 8 bytes, but not on 16 bytes. It opens AF_ALG socket and uses the
    socket to encrypt data in the buffer. The arm64 crypto accelerator causes
    data corruption or crashes in the scatterwalk_pagedone.

    This patch fixes the bug by passing the residue bytes that were not
    processed as the last parameter to blkcipher_walk_done.

    Signed-off-by: Mikulas Patocka
    Acked-by: Ard Biesheuvel
    Signed-off-by: Herbert Xu

    Mikulas Patocka
     

25 Jul, 2014

1 commit

  • This fixes the following build failure when building with CONFIG_MODVERSIONS
    enabled:

    CC [M] arch/arm64/crypto/aes-glue-ce.o
    ld: cannot find arch/arm64/crypto/aes-glue-ce.o: No such file or directory
    make[1]: *** [arch/arm64/crypto/aes-ce-blk.o] Error 1
    make: *** [arch/arm64/crypto] Error 2

    The $(obj)/aes-glue-%.o rule only creates $(obj)/.tmp_aes-glue-ce.o, it
    should use if_changed_rule instead of if_changed_dep.

    Signed-off-by: Andreas Schwab
    [ardb: mention CONFIG_MODVERSIONS in commit log]
    Signed-off-by: Ard Biesheuvel
    Signed-off-by: Catalin Marinas

    Andreas Schwab
     

18 Jun, 2014

2 commits