batching.rst 6.0 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183
  1. ========
  2. Batching
  3. ========
  4. Guzzle provides a fairly generic and very customizable batching framework that allows developers to efficiently
  5. transfer requests in parallel.
  6. Sending requests and commands in parallel
  7. -----------------------------------------
  8. You can send HTTP requests in parallel by passing an array of ``Guzzle\Http\Message\RequestInterface`` objects to
  9. ``Guzzle\Http\Client::send()``:
  10. .. code-block:: php
  11. $responses = $client->send(array(
  12. $client->get('http://www.example.com/foo'),
  13. $client->get('http://www.example.com/baz')
  14. $client->get('http://www.example.com/bar')
  15. ));
  16. You can send commands in parallel by passing an array of ``Guzzle\Service\Command\CommandInterface`` objects
  17. ``Guzzle\Service\Client::execute()``:
  18. .. code-block:: php
  19. $commands = $client->execute(array(
  20. $client->getCommand('foo'),
  21. $client->getCommand('baz'),
  22. $client->getCommand('bar')
  23. ));
  24. These approaches work well for most use-cases. When you need more control over the requests that are sent in
  25. parallel or you need to send a large number of requests, you need to use the functionality provided in the
  26. ``Guzzle\Batch`` namespace.
  27. Batching overview
  28. -----------------
  29. The batch object, ``Guzzle\Batch\Batch``, is a queue. You add requests to the queue until you are ready to transfer
  30. all of the requests. In order to efficiently transfer the items in the queue, the batch object delegates the
  31. responsibility of dividing the queue into manageable parts to a divisor (``Guzzle\Batch\BatchDivisorInterface``).
  32. The batch object then iterates over each array of items created by the divisor and sends them to the batch object's
  33. ``Guzzle\Batch\BatchTransferInterface``.
  34. .. code-block:: php
  35. use Guzzle\Batch\Batch;
  36. use Guzzle\Http\BatchRequestTransfer;
  37. // BatchRequestTransfer acts as both the divisor and transfer strategy
  38. $transferStrategy = new BatchRequestTransfer(10);
  39. $divisorStrategy = $transferStrategy;
  40. $batch = new Batch($transferStrategy, $divisorStrategy);
  41. // Add some requests to the batch queue
  42. $batch->add($request1)
  43. ->add($request2)
  44. ->add($request3);
  45. // Flush the queue and retrieve the flushed items
  46. $arrayOfTransferredRequests = $batch->flush();
  47. .. note::
  48. You might find that your transfer strategy will need to act as both the divisor and transfer strategy.
  49. Using the BatchBuilder
  50. ----------------------
  51. The ``Guzzle\Batch\BatchBuilder`` makes it easier to create batch objects. The batch builder also provides an easier
  52. way to add additional behaviors to your batch object.
  53. Transferring requests
  54. ~~~~~~~~~~~~~~~~~~~~~
  55. The ``Guzzle\Http\BatchRequestTransfer`` class efficiently transfers HTTP requests in parallel by grouping batches of
  56. requests by the curl_multi handle that is used to transfer the requests.
  57. .. code-block:: php
  58. use Guzzle\Batch\BatchBuilder;
  59. $batch = BatchBuilder::factory()
  60. ->transferRequests(10)
  61. ->build();
  62. Transferring commands
  63. ~~~~~~~~~~~~~~~~~~~~~
  64. The ``Guzzle\Service\Command\BatchCommandTransfer`` class efficiently transfers service commands by grouping commands
  65. by the client that is used to transfer them. You can add commands to a batch object that are transferred by different
  66. clients, and the batch will handle the rest.
  67. .. code-block:: php
  68. use Guzzle\Batch\BatchBuilder;
  69. $batch = BatchBuilder::factory()
  70. ->transferCommands(10)
  71. ->build();
  72. $batch->add($client->getCommand('foo'))
  73. ->add($client->getCommand('baz'))
  74. ->add($client->getCommand('bar'));
  75. $commands = $batch->flush();
  76. Batch behaviors
  77. ---------------
  78. You can add various behaviors to your batch that allow for more customizable transfers.
  79. Automatically flushing a queue
  80. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  81. Use the ``Guzzle\Batch\FlushingBatch`` decorator when you want to pump a large number of items into a batch queue and
  82. have the queue automatically flush when the size of the queue reaches a certain threshold.
  83. .. code-block:: php
  84. use Guzzle\Batch\BatchBuilder;
  85. $batch = BatchBuilder::factory()
  86. ->transferRequests(10)
  87. ->autoFlushAt(10)
  88. ->build();
  89. Batch builder method: ``autoFlushAt($threshold)``
  90. Notifying on flush
  91. ~~~~~~~~~~~~~~~~~~
  92. Use the ``Guzzle\Batch\NotifyingBatch`` decorator if you want a function to be notified each time the batch queue is
  93. flushed. This is useful when paired with the flushing batch decorator. Pass a callable to the ``notify()`` method of
  94. a batch builder to use this decorator with the builder.
  95. .. code-block:: php
  96. use Guzzle\Batch\BatchBuilder;
  97. $batch = BatchBuilder::factory()
  98. ->transferRequests(10)
  99. ->autoFlushAt(10)
  100. ->notify(function (array $transferredItems) {
  101. echo 'Transferred ' . count($transferredItems) . "items\n";
  102. })
  103. ->build();
  104. Batch builder method:: ``notify(callable $callback)``
  105. Keeping a history
  106. ~~~~~~~~~~~~~~~~~
  107. Use the ``Guzzle\Batch\HistoryBatch`` decorator if you want to maintain a history of all the items transferred with
  108. the batch queue.
  109. .. code-block:: php
  110. use Guzzle\Batch\BatchBuilder;
  111. $batch = BatchBuilder::factory()
  112. ->transferRequests(10)
  113. ->keepHistory()
  114. ->build();
  115. After transferring items, you can use the ``getHistory()`` of a batch to retrieve an array of transferred items. Be
  116. sure to periodically clear the history using ``clearHistory()``.
  117. Batch builder method: ``keepHistory()``
  118. Exception buffering
  119. ~~~~~~~~~~~~~~~~~~~
  120. Use the ``Guzzle\Batch\ExceptionBufferingBatch`` decorator to buffer exceptions during a transfer so that you can
  121. transfer as many items as possible then deal with the errored batches after the transfer completes. After transfer,
  122. use the ``getExceptions()`` method of a batch to retrieve an array of
  123. ``Guzzle\Batch\Exception\BatchTransferException`` objects. You can use these exceptions to attempt to retry the
  124. failed batches. Be sure to clear the buffered exceptions when you are done with them by using the
  125. ``clearExceptions()`` method.
  126. Batch builder method: ``bufferExceptions()``