This patch adds to ioatdma and dca modules
support for Intel I/OAT DMA engine ver.3 (aka CB3 device).
The main features of I/OAT ver.3 are:
 * 8 single channel DMA devices (8 channels total)
 * 8 DCA providers, each can accept 2 requesters
 * 8-bit TAG values and 32-bit extended APIC IDs
Signed-off-by: Maciej Sosnowski <maciej.sosnowski@intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

I/OAT DMA performance tuning showed different optimal values of
tcp_dma_copybreak for different I/OAT versions (4096 for 1.2 and 2048
for 2.0).  This patch lets ioatdma driver set tcp_dma_copybreak value
according to these results.

[dan.j.williams@intel.com: remove some ifdefs]
Signed-off-by: Maciej Sosnowski <maciej.sosnowski@intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

Due to occasional DMA channel hangs observed for I/OAT versions 1.2 and 2.0
a watchdog has been introduced to check every 2 seconds
if all channels progress normally.
If stuck channel is detected, driver resets it.
The reset is done in two parts. The second part is scheduled
by the first one to reinitialize the channel after the restart.
Signed-off-by: Maciej Sosnowski <maciej.sosnowski@intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

- use a table for iop13xx, trade text for data
- shrink the iop3xx to a cache line
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

Signed-off-by: Dan Williams <dan.j.williams@intel.com>

Force callers that trigger an "out of descriptors" condition to run the
cleanup loop directly.  Alleviates the requirement to have soft-irqs
enabled when polling for a descriptor in async_xor.
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

Signed-off-by: Dan Williams <dan.j.williams@intel.com>

All callers of async_tx_sync_epilog have called async_tx_quiesce on the
depend_tx, so async_tx_sync_epilog need only call the callback to
complete the operation.
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

Replace open coded "wait and acknowledge" instances with async_tx_quiesce.
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

Ensure forward progress is made when a dmaengine driver is unable to
allocate an xor descriptor by breaking the dependency chain with
async_tx_quisce() and issue any pending descriptors.

Tested with iop-adma by setting device->max_xor = 2 to force multiple
calls to device_prep_dma_xor for each call to async_xor and limiting the
descriptor slot pool to 5.  Discovered that the minimum descriptor pool
size for iop-adma is 2 * iop_chan_xor_slot_cnt(device->max_xor) + 1.
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

When the number of source buffers for an xor operation exceeds the hardware
channel maximum async_xor creates a chain of dependent operations.  The result
of one operation is reused as an input to the next to continue the xor
calculation.  The destination buffer should remain mapped for the duration of
the entire chain.  To provide this guarantee the code must no longer be allowed
to fallback to the synchronous path as this will preclude the buffer from being
unmapped, i.e. the dma-driver will potentially miss the descriptor with
!DMA_COMPL_SKIP_DEST_UNMAP.

Cc: Neil Brown <neilb@suse.de>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

In the rcu update side, don't use list_for_each_entry_rcu().
Signed-off-by: Li Zefan <lizf@cn.fujitsu.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

This adds a driver for the Synopsys DesignWare DMA controller (aka
DMACA on AVR32 systems.) This DMA controller can be found integrated
on the AT32AP7000 chip and is primarily meant for peripheral DMA
transfer, but can also be used for memory-to-memory transfers.

This patch is based on a driver from David Brownell which was based on
an older version of the DMA Engine framework. It also implements the
proposed extensions to the DMA Engine API for slave DMA operations.

The dmatest client shows no problems, but there may still be room for
improvement performance-wise. DMA slave transfer performance is
definitely "good enough"; reading 100 MiB from an SD card running at ~20
MHz yields ~7.2 MiB/s average transfer rate.

Full documentation for this controller can be found in the Synopsys
DW AHB DMAC Databook:

http://www.synopsys.com/designware/docs/iip/DW_ahb_dmac/latest/doc/dw_ahb_dmac_db.pdf

The controller has lots of implementation options, so it's usually a
good idea to check the data sheet of the chip it's intergrated on as
well. The AT32AP7000 data sheet can be found here:

http://www.atmel.com/dyn/products/datasheets.asp?family_id=682


Changes since v4:
  * Use client_count instead of dma_chan_is_in_use()
  * Add missing include
  * Unmap buffers unless client told us not to

Changes since v3:
  * Update to latest DMA engine and DMA slave APIs
  * Embed the hw descriptor into the sw descriptor
  * Clean up and update MODULE_DESCRIPTION, copyright date, etc.

Changes since v2:
  * Dequeue all pending transfers in terminate_all()
  * Rename dw_dmac.h -> dw_dmac_regs.h
  * Define and use controller-specific dma_slave data
  * Fix up a few outdated comments
  * Define hardware registers as structs (doesn't generate better
    code, unfortunately, but it looks nicer.)
  * Get number of channels from platform_data instead of hardcoding it
    based on CONFIG_WHATEVER_CPU.
  * Give slave clients exclusive access to the channel

Acked-by: Maciej Sosnowski <maciej.sosnowski@intel.com>,
Signed-off-by: Haavard Skinnemoen <haavard.skinnemoen@atmel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

This patch adds the necessary interfaces to the DMA Engine framework
to use functionality found on most embedded DMA controllers: DMA from
and to I/O registers with hardware handshaking.

In this context, hardware hanshaking means that the peripheral that
owns the I/O registers in question is able to tell the DMA controller
when more data is available for reading, or when there is room for
more data to be written. This usually happens internally on the chip,
but these signals may also be exported outside the chip for things
like IDE DMA, etc.

A new struct dma_slave is introduced. This contains information that
the DMA engine driver needs to set up slave transfers to and from a
slave device. Most engines supporting DMA slave transfers will want to
extend this structure with controller-specific parameters.  This
additional information is usually passed from the platform/board code
through the client driver.

A "slave" pointer is added to the dma_client struct. This must point
to a valid dma_slave structure iff the DMA_SLAVE capability is
requested.  The DMA engine driver may use this information in its
device_alloc_chan_resources hook to configure the DMA controller for
slave transfers from and to the given slave device.

A new operation for preparing slave DMA transfers is added to struct
dma_device. This takes a scatterlist and returns a single descriptor
representing the whole transfer.

Another new operation for terminating all pending transfers is added as
well. The latter is needed because there may be errors outside the scope
of the DMA Engine framework that may require DMA operations to be
terminated prematurely.

DMA Engine drivers may extend the dma_device, dma_chan and/or
dma_slave_descriptor structures to allow controller-specific
operations. The client driver can detect such extensions by looking at
the DMA Engine's struct device, or it can request a specific DMA
Engine device by setting the dma_dev field in struct dma_slave.

dmaslave interface changes since v4:
  * Fix checkpatch errors
  * Fix changelog (there are no slave descriptors anymore)

dmaslave interface changes since v3:
  * Use dma_data_direction instead of a new enum
  * Submit slave transfers as scatterlists
  * Remove the DMA slave descriptor struct

dmaslave interface changes since v2:
  * Add a dma_dev field to struct dma_slave. If set, the client can
    only be bound to the DMA controller that corresponds to this
    device.  This allows controller-specific extensions of the
    dma_slave structure; if the device matches, the controller may
    safely assume its extensions are present.
  * Move reg_width into struct dma_slave as there are currently no
    users that need to be able to set the width on a per-transfer
    basis.

dmaslave interface changes since v1:
  * Drop the set_direction and set_width descriptor hooks. Pass the
    direction and width to the prep function instead.
  * Declare a dma_slave struct with fixed information about a slave,
    i.e. register addresses, handshake interfaces and such.
  * Add pointer to a dma_slave struct to dma_client. Can be NULL if
    the DMA_SLAVE capability isn't requested.
  * Drop the set_slave device hook since the alloc_chan_resources hook
    now has enough information to set up the channel for slave
    transfers.
Acked-by: Maciej Sosnowski <maciej.sosnowski@intel.com>
Signed-off-by: Haavard Skinnemoen <haavard.skinnemoen@atmel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

In some cases client code may need the dma-driver to skip the unmap of source
and/or destination buffers.  Setting these flags indicates to the driver to
skip the unmap step.  In this regard async_xor is currently broken in that it
allows the destination buffer to be unmapped while an operation is still in
progress, i.e. when the number of sources exceeds the hardware channel's
maximum (fixed in a subsequent patch).
Acked-by: Saeed Bishara <saeed@marvell.com>
Acked-by: Maciej Sosnowski <maciej.sosnowski@intel.com>
Acked-by: Haavard Skinnemoen <haavard.skinnemoen@atmel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

A DMA controller capable of doing slave transfers may need to know a
few things about the slave when preparing the channel. We don't want
to add this information to struct dma_channel since the channel hasn't
yet been bound to a client at this point.

Instead, pass a reference to the client requesting the channel to the
driver's device_alloc_chan_resources hook so that it can pick the
necessary information from the dma_client struct by itself.

[dan.j.williams@intel.com: fixed up fsldma and mv_xor]
Acked-by: Maciej Sosnowski <maciej.sosnowski@intel.com>
Signed-off-by: Haavard Skinnemoen <haavard.skinnemoen@atmel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

This client tests DMA memcpy using various lengths and various offsets
into the source and destination buffers. It will initialize both
buffers with a repeatable pattern and verify that the DMA engine copies
the requested region and nothing more. It will also verify that the
bytes aren't swapped around, and that the source buffer isn't modified.

The dmatest module can be configured to test a specific device, a
specific channel. It can also test multiple channels at the same time,
and it can start multiple threads competing for the same channel.

Changes since v2:
  * Support testing multiple channels at the same time
  * Support testing with multiple threads competing for the same channel
  * Use counting test patterns in order to catch byte ordering issues

Changes since v1:
  * Remove extra dashes around "help"
  * Remove "default n" from Kconfig
  * Turn TEST_BUF_SIZE into a module parameter
  * Return DMA_NAK instead of DMA_DUP
  * Print unhandled events
  * Support testing specific channels and devices
  * Move to the end of the Makefile
Acked-by: Maciej Sosnowski <maciej.sosnowski@intel.com>
Signed-off-by: Haavard Skinnemoen <hskinnemoen@atmel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

The XOR engine found in Marvell's SoCs and system controllers
provides XOR and DMA operation, iSCSI CRC32C calculation, memory
initialization, and memory ECC error cleanup operation support.

This driver implements the DMA engine API and supports the following
capabilities:
- memcpy
- xor
- memset

The XOR engine can be used by DMA engine clients implemented in the
kernel, one of those clients is the RAID module.  In that case, I
observed 20% improvement in the raid5 write throughput, and 40%
decrease in the CPU utilization when doing array construction, those
results obtained on an 5182 running at 500Mhz.

When enabling the NET DMA client, the performance decreased, so
meanwhile it is recommended to keep this client off.
Signed-off-by: Saeed Bishara <saeed@marvell.com>
Signed-off-by: Lennert Buytenhek <buytenh@marvell.com>
Signed-off-by: Nicolas Pitre <nico@marvell.com>
Acked-by: Maciej Sosnowski <maciej.sosnowski@intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

Since 43cc71ee, the platform
modalias is prefixed with "platform:". Add MODULE_ALIAS() to most
of the hotpluggable platform drivers, to re-enable auto loading.

Cc: <stable@kernel.org>
Signed-off-by: Kay Sievers <kay.sievers@vrfy.org>
Signed-off-by: David Brownell <dbrownell@users.sourceforge.net>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

Haavard's dma-slave interface would like to test for exclusive access to a
channel.  The standard channel refcounting is not sufficient in that it
tracks more than just client references, it is also inaccurate as reference
counts are percpu until the channel is removed.

This change also enables a future fix to deallocate resources when a client
declines to use a capable channel.
Acked-by: Haavard Skinnemoen <haavard.skinnemoen@atmel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

The dependency is redundant since all drivers set their specific arch
dependencies.  The NET_DMA option is modified to be enabled only on platforms
where it is known to have a positive effect.  HAS_DMA is added as an explicit
dependency for the DMADEVICES menu.
Acked-by: Adrian Bunk <bunk@kernel.org>
Acked-by: Haavard Skinnemoen <haavard.skinnemoen@atmel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

Set the 'parent' field of channel class devices to point to the
physical DMA device initialized by the DMA engine driver.

This allows drivers to use chan->dev.parent for syncing DMA buffers
and adds a 'device' symlink to the real device in
/sys/class/dma/dmaXchanY.
Signed-off-by: Haavard Skinnemoen <hskinnemoen@atmel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

commit 636bdeaa 'dmaengine: ack to flags: make use of the unused bits in
the 'ack' field' missed an ->ack conversion in
crypto/async_tx/async_memset.c
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

Signed-off-by: Li Yang <leoli@freescale.com>
Acked-by: Zhang Wei <zw@zh-kernel.org>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

> +#define	ARCH_KMALLOC_MINALIGN		(sizeof(long) * 2)
> +#define	ARCH_SLAB_MINALIGN		(sizeof(long) * 2)

This doesn't work if SLAB is selected and slab debugging is enabled as
these are passed to the preprocessor, and the preprocessor doesn't
understand sizeof.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

* 'for-linus' of git://git.kernel.dk/linux-2.6-block:
  cfq-iosched: fix RCU problem in cfq_cic_lookup()
  block: make blktrace use per-cpu buffers for message notes
  Added in elevator switch message to blktrace stream
  Added in MESSAGE notes for blktraces
  block: reorder cfq_queue to save space on 64bit builds
  block: Move the second call to get_request to the end of the loop
  splice: handle try_to_release_page() failure
  splice: fix sendfile() issue with relay

Specify the minimum slab/kmalloc alignment to be 8 bytes.  This fixes a
crash when SLOB is selected as the memory allocator.  The FRV arch needs
this so that it can use the load- and store-double instructions without
faulting.  By default SLOB sets the minimum to be 4 bytes.
Signed-off-by: David Howells <dhowells@redhat.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

Fix a typo in the header guard of asm/ipc.h.
Signed-off-by: Vegard Nossum <vegard.nossum@gmail.com>
Signed-off-by: David Howells <dhowells@redhat.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

cfq_cic_lookup() needs to properly protect ioc->ioc_data before
dereferencing it and also exclude updaters of ioc->ioc_data as well.

Also add a number of comments documenting why the existing RCU usage
is OK.

Thanks a lot to "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> for
review and comments!
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>

Currently it uses a single static char array, but that risks
being corrupted when multiple users issue message notes at the
same time. Make the buffers dynamically allocated when the trace
is setup and make them per-cpu instead.

The default max message size of 1k is also very large, the
interface is mainly for small text notes. So shrink it to 128 bytes.
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>

Signed-off-by: Alan D. Brunelle <alan.brunelle@hp.com>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>

Allows messages to be inserted into blktrace streams.
Signed-off-by: Alan D. Brunelle <alan.brunelle@hp.com>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>

saves 8 bytes of padding & increases objects/slab from 30 to 32 on my
AMD64 config
Signed-off-by: Richard Kennedy <richard@rsk.demon.co.uk>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>

In function get_request_wait, the second call to get_request could be
moved to the end of the while loop, because if the first call to
get_request fails, the second call will fail without sleep.
Signed-off-by: Zhang Yanmin <yanmin.zhang@intel.com>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>

splice currently assumes that try_to_release_page() always suceeds,
but it can return failure. If it does, we cannot steal the page.

Acked-by: Mingming Cao <cmm@us.ibm.com
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>

Splice isn't always incrementing the ppos correctly, which broke
relay splice.
Signed-off-by: Tom Zanussi <zanussi@comcast.net>
Tested-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>

* 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jbarnes/pci-2.6:
  pciehp: add message about pciehp_slot_with_bus option
  pci hotplug core: add check of duplicate slot name
  pciehp: move msleep after power off
  pciehp: poll cmd completion if hotplug interrupt is disabled
  pciehp: fix slow probing
  pciehp: fix NULL dereference in interrupt handler
  shpchp: add message about shpchp_slot_with_bus option
  PCI: don't enable ASPM on devices with mixed PCIe/PCI functions

Some (broken?) platform assign the same slot name to multiple hotplug
slots. On such system, slot initialization would fail because of name
collision. The pciehp driver already have a "slot_with_bus" module
option which adds the bus number into the slot name. This patch adds
the message about this module option that will be displayed when slot
name collision is detected.
Signed-off-by: Kenji Kaneshige <kaneshige.kenji@jp.fujitsu.com>
Signed-off-by: Kristen Carlson Accardi <kristen.c.accardi@intel.com>
Signed-off-by: Jesse Barnes <jbarnes@virtuousgeek.org>

Fix the following errors reported by Jan C. Nordholz in
http://bugzilla.kernel.org/show_bug.cgi?id=10751.

kobject_add_internal failed for 2 with -EEXIST, don't try to register things with the same name in the same directory.
Pid: 1, comm: swapper Tainted: G        W 2.6.26-rc3 #1
 [<c0266980>] kobject_add_internal+0x140/0x190
 [<c0266afd>] kobject_init_and_add+0x2d/0x40
 [<c027bc91>] pci_hp_register+0x81/0x2f0
 [<c027fd07>] pciehp_probe+0x1a7/0x470
 [<c01b3b84>] sysfs_add_one+0x44/0xa0
 [<c01b3c1f>] sysfs_addrm_start+0x3f/0xb0
 [<c01b497a>] sysfs_create_link+0x8a/0xf0
 [<c0279570>] pcie_port_probe_service+0x50/0x80
 [<c02e0545>] driver_sysfs_add+0x55/0x70
 [<c02e0662>] driver_probe_device+0x82/0x180
 [<c02e07cc>] __driver_attach+0x6c/0x70
 [<c02dfe0a>] bus_for_each_dev+0x3a/0x60
 [<c05db2d0>] pcied_init+0x0/0x80
 [<c02e04e6>] driver_attach+0x16/0x20
 [<c02e0760>] __driver_attach+0x0/0x70
 [<c02e0341>] bus_add_driver+0x1a1/0x220
 [<c05db2d0>] pcied_init+0x0/0x80
 [<c02e09cd>] driver_register+0x4d/0x120
 [<c05db050>] ibm_acpiphp_init+0x0/0x190
 [<c0125aab>] printk+0x1b/0x20
 [<c05db2d0>] pcied_init+0x0/0x80
 [<c05db2de>] pcied_init+0xe/0x80
 [<c05c751a>] kernel_init+0x10a/0x300
 [<c0120138>] schedule_tail+0x18/0x50
 [<c0103b9a>] ret_from_fork+0x6/0x1c
 [<c05c7410>] kernel_init+0x0/0x300
 [<c05c7410>] kernel_init+0x0/0x300
 [<c010485b>] kernel_thread_helper+0x7/0x1c
 =======================
pci_hotplug: Unable to register kobject '2'<3>pciehp: pci_hp_register failed with error -22

Slot with the same name can be registered multiple times if shpchp or
pciehp driver is loaded after acpiphp is loaded because ACPI based
hotplug driver and Native OS hotplug driver trying to handle the same
physical slot. In this case, current pci_hotplug core will call
kobject_init_and_add() muliple time with the same name. This is the
cause of this problem. To fix this problem, this patch adds the check
into pci_hp_register() to see if the slot with the same name.
Signed-off-by: Kenji Kaneshige <kaneshige.kenji@jp.fujitsu.com>
Signed-off-by: Kristen Carlson Accardi <kristen.c.accardi@intel.com>
Signed-off-by: Jesse Barnes <jbarnes@virtuousgeek.org>

According to the PCI Express specification, we must wait for at least
1 second after turning power off before taking any action that relies
on power having been removed from the slot/adapter. For this, current
pciehp wait for 1 second after issuing the power off command in
hpc_power_off_slot() function. But waiting for 1 second in
hpc_power_off_slot() can make pciehp probing slow-down because pciehp
probe code calls hpc_power_off_slot() if the slot is not occupied just
in case. We don't need to wait for 1 second at the pciehp probe time
because there is no action on that empty slot. So move 1 second wait
from hpc_power_off_slot() to the caller of hpc_power_off_slot().
Signed-off-by: Kenji Kaneshige <kaneshige.kenji@jp.fujitsu.com>
Signed-off-by: Kristen Carlson Accardi <kristen.c.accardi@intel.com>
Signed-off-by: Jesse Barnes <jbarnes@virtuousgeek.org>