[CRYPTO] Add plumbing for multi-block operations

The VIA Padlock device is able to perform much better when multiple
blocks are fed to it at once.  As this device offers an exceptional
throughput rate it is worthwhile to optimise the infrastructure
specifically for it.

We shift the existing page-sized fast path down to the CBC/ECB functions.
We can then replace the CBC/ECB functions with functions provided by the
underlying algorithm that performs the multi-block operations.

As a side-effect this improves the performance of large cipher operations
for all existing algorithm implementations.  I've measured the gain to be
around 5% for 3DES and 15% for AES.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: David S. Miller <davem@davemloft.net>

crypto/scatterwalk.c

@@ -100,7 +100,7 @@ void scatterwalk_done(struct scatter_walk *walk, int out, int more)
 int scatterwalk_copychunks(void *buf, struct scatter_walk *walk,
 			   size_t nbytes, int out)
 {
-	do {
+	while (nbytes > walk->len_this_page) {
 		memcpy_dir(buf, walk->data, walk->len_this_page, out);
 		buf += walk->len_this_page;
 		nbytes -= walk->len_this_page;
@@ -108,7 +108,7 @@ int scatterwalk_copychunks(void *buf, struct scatter_walk *walk,
 		scatterwalk_unmap(walk, out);
 		scatterwalk_pagedone(walk, out, 1);
 		scatterwalk_map(walk, out);
-	} while (nbytes > walk->len_this_page);
+	}
 
 	memcpy_dir(buf, walk->data, nbytes, out);
 	return nbytes;

crypto/scatterwalk.h

@@ -40,10 +40,10 @@ static inline int scatterwalk_samebuf(struct scatter_walk *walk_in,
 	       walk_in->offset == walk_out->offset;
 }
 
-static inline int scatterwalk_across_pages(struct scatter_walk *walk,
-					   unsigned int nbytes)
+static inline unsigned int scatterwalk_clamp(struct scatter_walk *walk,
+					     unsigned int nbytes)
 {
-	return nbytes > walk->len_this_page;
+	return nbytes > walk->len_this_page ? walk->len_this_page : nbytes;
 }
 
 static inline void scatterwalk_advance(struct scatter_walk *walk,