3

k��_5n�@sz
dZd
dl
Z
d
dlZd
dlZd
dlZd
dlZd
dlZd
dlZd
dlZd
dl	Z
d
dlmZ
d
dl
mZ
d
dlmZ
d
dlZd
dlmZmZ
dZdZGd	d
�d
ej�Zeje�
Zeje��

ejjZd%Z e!�Z"d&d
d�
Z#dd�Z$dd�Z%Gdd�de&�Z'Gdd�de!�Z(Gdd�de!�Z)Gdd�de!�Z*Gdd�de!�Z+Gdd�dej,�Z-Gdd �d e!�Z.Gd!d"�d"e!�Z/Gd#d$�d$e!�Z0dS)'a�Abstractions over S3's upload/download operations.

This module provides high level abstractions for efficient
uploads/downloads.  It handles several things for the user:

* Automatically switching to multipart transfers when
  a file is over a specific size threshold
* Uploading/downloading a file in parallel
* Throttling based on max bandwidth
* Progress callbacks to monitor transfers
* Retries.  While botocore handles retries for streaming uploads,
  it is not possible for it to handle retries for streaming
  downloads.  This module handles retries for both cases so
  you don't need to implement any retry logic yourself.

This module has a reasonable set of defaults.  It also allows you
to configure many aspects of the transfer process including:

* Multipart threshold size
* Max parallel downloads
* Max bandwidth
* Socket timeouts
* Retry amounts

There is no support for s3->s3 multipart copies at this
time.


.. _ref_s3transfer_usage:

Usage
=====

The simplest way to use this module is:

.. code-block:: python

    client = boto3.client('s3', 'us-west-2')
    transfer = S3Transfer(client)
    # Upload /tmp/myfile to s3://bucket/key
    transfer.upload_file('/tmp/myfile', 'bucket', 'key')

    # Download s3://bucket/key to /tmp/myfile
    transfer.download_file('bucket', 'key', '/tmp/myfile')

The ``upload_file`` and ``download_file`` methods also accept
``**kwargs``, which will be forwarded through to the corresponding
client operation.  Here are a few examples using ``upload_file``::

    # Making the object public
    transfer.upload_file('/tmp/myfile', 'bucket', 'key',
                         extra_args={'ACL': 'public-read'})

    # Setting metadata
    transfer.upload_file('/tmp/myfile', 'bucket', 'key',
                         extra_args={'Metadata': {'a': 'b', 'c': 'd'}})

    # Setting content type
    transfer.upload_file('/tmp/myfile.json', 'bucket', 'key',
                         extra_args={'ContentType': "application/json"})


The ``S3Transfer`` clas also supports progress callbacks so you can
provide transfer progress to users.  Both the ``upload_file`` and
``download_file`` methods take an optional ``callback`` parameter.
Here's an example of how to print a simple progress percentage
to the user:

.. code-block:: python

    class ProgressPercentage(object):
        def __init__(self, filename):
            self._filename = filename
            self._size = float(os.path.getsize(filename))
            self._seen_so_far = 0
            self._lock = threading.Lock()

        def __call__(self, bytes_amount):
            # To simplify we'll assume this is hooked up
            # to a single filename.
            with self._lock:
                self._seen_so_far += bytes_amount
                percentage = (self._seen_so_far / self._size) * 100
                sys.stdout.write(
                    "
%s  %s / %s  (%.2f%%)" % (self._filename, self._seen_so_far,
                                                 self._size, percentage))
                sys.stdout.flush()


    transfer = S3Transfer(boto3.client('s3', 'us-west-2'))
    # Upload /tmp/myfile to s3://bucket/key and print upload progress.
    transfer.upload_file('/tmp/myfile', 'bucket', 'key',
                         callback=ProgressPercentage('/tmp/myfile'))



You can also provide a TransferConfig object to the S3Transfer
object that gives you more fine grained control over the
transfer.  For example:

.. code-block:: python

    client = boto3.client('s3', 'us-west-2')
    config = TransferConfig(
        multipart_threshold=8 * 1024 * 1024,
        max_concurrency=10,
        num_download_attempts=10,
    )
    transfer = S3Transfer(client, config)
    transfer.upload_file('/tmp/foo', 'bucket', 'key')


�N)
�six)
�ReadTimeoutError)
�IncompleteReadError)�RetriesExceededError�S3UploadFailedErrorzAmazon Web Servicesz0.3.4c@seZ
dZd
d�ZdS)�NullHandlerc
CsdS)
N�)�self�recordrr�/usr/lib/python3.6/__init__.py�emit�s
zNullHandler.emitN)�__name__�
__module__�__qualname__rrrrrr�s
ri�c

Csd
jdd�t
|�
D�
�
S)N�c
ss|]}
tj
tj�
V
qdS)
N)�randomZchoice�stringZ	hexdigits)�.0�
_rrr�	<genexpr>�sz(random_file_extension.<locals>.<genexpr>)�join�range)
Z
num_digitsrrr�random_file_extension�s
rcKs"|
dkrt|j
d�r|j
j�
dS)N�	PutObject�
UploadPart�disable_callback)rr)�hasattr�bodyr)�request�operation_name�kwargsrrr�disable_upload_callbacks�s


r"cKs"|
dkrt|j
d�r|j
j�
dS)Nrr�enable_callback)rr)rrr#)rr r!rrr�enable_upload_callbacks�s


r$c
@seZ
dZd
S)�QueueShutdownErrorN)r
rrrrrrr%�s
r%c@s~eZ
dZddd�
Zeddd�
�
Zdd�Zdd	d
�
Zdd�Zd
d�Z	dd�Z
dd�Zdd�Zdd�Z
dd�Zdd�Zdd�Zd
S) �
ReadFileChunkNTcCsF|
|_||_
|j|j|||d
�|_|jj|j
�

d|_||_||_dS)a�

        Given a file object shown below:

            |___________________________________________________|
            0          |                 |                 full_file_size
                       |----chunk_size---|
                 start_byte

        :type fileobj: file
        :param fileobj: File like object

        :type start_byte: int
        :param start_byte: The first byte from which to start reading.

        :type chunk_size: int
        :param chunk_size: The max chunk size to read.  Trying to read
            pass the end of the chunk size will behave like you've
            reached the end of the file.

        :type full_file_size: int
        :param full_file_size: The entire content length associated
            with ``fileobj``.

        :type callback: function(amount_read)
        :param callback: Called whenever data is read from this object.

        )�requested_size�
start_byte�actual_file_sizer
N)�_fileobj�_start_byte�_calculate_file_size�_size�seek�_amount_read�	_callback�_callback_enabled)r	�fileobjr(�
chunk_sizeZfull_file_size�callbackr#rrr�__init__�s







zReadFileChunk.__init__cCs,t|
d
�}t
j|j��
j}|||||||�S)aWConvenience factory function to create from a filename.

        :type start_byte: int
        :param start_byte: The first byte from which to start reading.

        :type chunk_size: int
        :param chunk_size: The max chunk size to read.  Trying to read
            pass the end of the chunk size will behave like you've
            reached the end of the file.

        :type full_file_size: int
        :param full_file_size: The entire content length associated
            with ``fileobj``.

        :type callback: function(amount_read)
        :param callback: Called whenever data is read from this object.

        :type enable_callback: bool
        :param enable_callback: Indicate whether to invoke callback
            during read() calls.

        :rtype: ``ReadFileChunk``
        :return: A new instance of ``ReadFileChunk``

        �rb)�open�os�fstat�fileno�st_size)�cls�filenamer(r3r4r#�
fZ	file_sizerrr�
from_filename�s



zReadFileChunk.from_filenamecCs||}t||�S)
N)
�min)r	r2r'r(r)Zmax_chunk_sizerrrr,

s
z"ReadFileChunk._calculate_file_sizecCsh|
dkr|j|j
}nt|j|j
|
�}|jj|�
}|j
t|�
7_
|jdk	rd|jrd|jt|�
�

|S)
N)r-r/r@r*�read�lenr0r1)r	ZamountZamount_to_read�datarrrrA
s






zReadFileChunk.readc

Cs
d
|_dS)NT)
r1)
r	rrrr#
s
zReadFileChunk.enable_callbackc

Cs
d
|_dS)NF)
r1)
r	rrrr
s
zReadFileChunk.disable_callbackcCs<|jj
|j|
�

|jdk	r2|jr2|j|
|j�

|
|_dS)
N)r*r.r+r0r1r/)r	�whererrrr.
s


zReadFileChunk.seekc


Cs|jj
�
dS)
N)r*�close)
r	rrrrE
s
zReadFileChunk.closec


Cs|jS)
N)
r/)
r	rrr�tell!
s
zReadFileChunk.tellc


Cs|jS)
N)
r-)
r	rrr�__len__$
szReadFileChunk.__len__c


Cs|S)
Nr)
r	rrr�	__enter__,
s
zReadFileChunk.__enter__c

Os|j�
dS)
N)
rE)r	�argsr!rrr�__exit__/
s
zReadFileChunk.__exit__c

Cstg�
S)
N)
�iter)
r	rrr�__iter__2
szReadFileChunk.__iter__)NT)NT)
N)r
rrr5�classmethodr?r,rAr#rr.rErFrGrHrJrLrrrrr&�s
'


r&c@s"eZ
dZd
Zddd�
Zdd�ZdS)�StreamReaderProgressz<Wrapper for a read only stream that adds progress callbacks.NcCs|
|_||_
dS)
N)�_streamr0)r	�streamr4rrrr5=
s

zStreamReaderProgress.__init__c
Os*|jj
|
|�
}|jdk	r&|jt|�
�

|S)
N)rOrAr0rB)r	rIr!�valuerrrrAA
s




zStreamReaderProgress.read)
N)r
rr�__doc__r5rArrrrrN;
s


rNc@s4eZ
dZd
d�Zdd�Zdd�Zdd�Zd	d
�ZdS)�OSUtilscCstj
j|
�
S)
N)r8�path�getsize)r	r=rrr�
get_file_sizeI
s
zOSUtils.get_file_sizecCstj
|
|||d
d�S)NF)
r#)r&r?)r	r=r(�sizer4rrr�open_file_chunk_readerL
s


zOSUtils.open_file_chunk_readercCs
t|
|�S)
N)
r7)r	r=�moderrrr7Q
s
zOSUtils.opencCs(ytj
|
�

Wntk
r"


YnXd
S)z+Remove a file, noop if file does not exist.N)r8�remove�OSError)r	r=rrr�remove_fileT
s


zOSUtils.remove_filecCstj
j|
|�
dS)
N)�
s3transfer�compat�rename_file)r	Zcurrent_filenameZnew_filenamerrrr_]
s
zOSUtils.rename_fileN)r
rrrVrXr7r\r_rrrrrSH
s

	rSc@sHeZ
dZd
dddgZejjf
dd�
Zdd�Zd	d
�Z	dd�Z
d
d�ZdS)�MultipartUploader�SSECustomerKey�SSECustomerAlgorithm�SSECustomerKeyMD5�RequestPayercCs|
|_||_
||_||_dS)
N)�_client�_config�_os�
_executor_cls)r	�client�config�osutil�executor_clsrrrr5k
s


zMultipartUploader.__init__cCs0i}x&|
j�D]\}}||j
kr|||<qW|S)
N)�items�UPLOAD_PART_ARGS)r	�
extra_argsZupload_parts_args�keyrQrrr�_extra_upload_part_argsr
s





z)MultipartUploader._extra_upload_part_argsc
Cs�|jj
f||d
�|��
}|d}y|j||
||||�}Wn^tk
r�
}	
zBtjddd�
|jj|||d�
td|
dj||g�
|	f�
�
WYdd}	~	XnX|jj	|||d	|i
d
�
dS)N)�Bucket�Key�UploadIdzBException raised while uploading parts, aborting multipart upload.T)
�exc_info)rrrsrtzFailed to upload %s to %s: %s�
/ZParts)rrrsrtZMultipartUpload)
reZcreate_multipart_upload�
_upload_parts�	Exception�logger�debugZabort_multipart_uploadrrZcomplete_multipart_upload)
r	r=�bucketrpr4ro�response�	upload_id�parts�
errr�upload_file{
s"














*


zMultipartUploader.upload_filecCs�|j|�
}g}|j
j}	ttj|jj|�
t|	�
�
�
}
|j
j	}|j
|d
�
�J}tj|j
||||
|	||�}
x(|j|
td|
d��D]}|j|�

q|WWdQRX|S)N)
�max_workers�
)rqrf�multipart_chunksize�int�math�ceilrgrV�float�max_concurrencyrh�	functools�partial�_upload_one_part�mapr�append)r	r}r=r{rpr4roZupload_parts_extra_argsr~�	part_size�	num_partsr��executorZupload_partial�partrrrrw�
s













zMultipartUploader._upload_partsc	
CsZ|jj
}	|	|
||d
||��2}
|jjf|||||
d�|��
}|d}||d�SQRXdS)Nr�)rrrsrt�
PartNumber�Body�ETag)r�r�)rgrXreZupload_part)
r	r=r{rpr}r�ror4Zpart_number�open_chunk_readerrr|Zetagrrrr��
s









z"MultipartUploader._upload_one_partN)r
rrrn�
concurrent�futures�ThreadPoolExecutorr5rqr�rwr�rrrrr`a
s


	r`c@s(eZ
dZd
Zdd�Zdd�Zdd�ZdS)	�
ShutdownQueueaY
A queue implementation that can be shutdown.

    Shutting down a queue means that this class adds a
    trigger_shutdown method that will trigger all subsequent
    calls to put() to fail with a ``QueueShutdownError``.

    It purposefully deviates from queue.Queue, and is *not* meant
    to be a drop in replacement for ``queue.Queue``.

    cCsd
|_t
j�|_tjj||
�S)NF)�	_shutdown�	threadingZLock�_shutdown_lock�queue�Queue�_init)r	�maxsizerrrr��
s


zShutdownQueue._initc

	Cs&|j�
d
|_
tjd�

WdQRXdS)NTzThe IO queue is now shutdown.)r�r�ryrz)
r	rrr�trigger_shutdown�
s


zShutdownQueue.trigger_shutdownc
Cs.|j�
|j
rtd
�
�
WdQRXtjj||
�S)Nz6Cannot put item to queue when queue has been shutdown.)r�r�r%r�r��put)r	�itemrrrr��
s

zShutdownQueue.putN)r
rrrRr�r�r�rrrrr��
s

r�c@sNeZ
dZejjf
d
d�
Zddd�
Zdd�Zdd	�Z	d
d�Z
dd
�Zdd�ZdS)�MultipartDownloadercCs*|
|_||_
||_||_t|j
j�
|_dS)
N)rerfrgrhr��max_io_queue�_ioqueue)r	rirjrkrlrrrr5�
s




zMultipartDownloader.__init__Nc
Csv|jd
d�
�`}t
j|j|
||||�}|j|�
}	t
j|j|�}
|j|
�
}tjj|	|gtjj	d�}|j
|�

WdQRXdS)N�)
r�)
Zreturn_when)rhr�r��_download_file_as_futureZsubmit�_perform_io_writesr�r��waitZFIRST_EXCEPTION�_process_future_results)
r	r{rpr=�object_sizeror4Z
controllerZdownload_parts_handlerZparts_futureZio_writes_handlerZ	io_future�resultsrrr�
download_file�
s











z!MultipartDownloader.download_filecCs"|
\}}x|D]}|j�
qWdS)
N)
�result)r	r�ZfinishedZ
unfinishedZfuturerrrr��
s



z+MultipartDownloader._process_future_resultscCs�|jj
}ttj|t|�
�
�
}|jj}tj|j	|
|||||�}	z0|j
|d
�
�}
t|
j|	t
|�
��

WdQRXWd|jjt�

XdS)N)
r�)rfr�r�r�r�r�r�r�r��_download_rangerh�listr�rr�r��SHUTDOWN_SENTINEL)r	r{rpr=r�r4r�r�r�Zdownload_partialr�rrrr��
s









"z,MultipartDownloader._download_file_as_futurecCs6||
}||d
krd}n||
d
}d||f}|S)Nr�rzbytes=%s-%sr)r	r��
part_indexr�Zstart_rangeZ	end_range�range_paramrrr�_calculate_range_param�
s




z*MultipartDownloader._calculate_range_paramcs

z�|j|||�}|j
j}	d}
x�t|	�
D]�}yttjd
�

|jj|
||d�}t|d|��
d
�||}
x8t	��
fdd�d�D] }|j
j|
|f�

|
t|�
7}
qxWdSt
jt
jttfk
r�
}
z tjd	|||	d
d�
|}
w&WYdd}~Xq&Xq&Wt|
�
�
Wdtjd|�
XdS)NzMaking get_object call.)rrrsZRanger�i�cs
�
j��
S)
N)
rAr)�buffer_size�streaming_bodyrr�<lambda>sz5MultipartDownloader._download_range.<locals>.<lambda>�zCRetrying exception caught (%s), retrying request, (attempt %s / %s)T)
ruz$EXITING _download_range for part: %si@)r�rf�num_download_attemptsrryrzre�
get_objectrNrKr�r�rB�socket�timeout�errorrrr)r	r{rpr=r�r�r4r�r��max_attempts�last_exception�
ir|Z
current_index�chunkrr)r�r�rr�s8





























z#MultipartDownloader._download_rangecCs�|jj
|
d
���}x�|jj�}|tkr2tjd�

dSy |\}}|j|�

|j|�

Wqt	k
r�
}
z"tjd|dd�
|jj
�
�WYdd}~XqXqWWdQRXdS)N�wbzCShutdown sentinel received in IO handler, shutting down IO handler.z!Caught exception in IO thread: %sT)
ru)rgr7r��getr�ryrzr.�writerxr�)r	r=r>Ztask�offsetrCrrrrr�$s

















z&MultipartDownloader._perform_io_writes)
N)
r
rrr�r�r�r5r�r�r�r�r�r�rrrrr��
s
	!r�c@s(eZ
dZd
edd
eddfdd�
ZdS)�TransferConfigr�
��dcCs"|
|_||_
||_||_||_dS)
N)�multipart_thresholdr�r�r�r�)r	r�r�r�r�r�rrrr59s




zTransferConfig.__init__N)r
rr�MBr5rrrrr�8s




r�c@s�eZ
dZd
ddddgZdddd	d
ddd
dddddddddddddgZd.dd�
Zd/dd�
Zdd�Zd0dd�
Zd d!�Z	d"d#�Z
d$d%�Zd&d'�Zd(d)�Z
d*d+�Zd,d-�ZdS)1�
S3TransferZ	VersionIdrbrarcrdZACLZCacheControlZContentDispositionZContentEncodingZContentLanguageZContentTypeZExpiresZGrantFullControlZ	GrantReadZGrantReadACPZ
GrantWriteACLZMetadataZServerSideEncryptionZStorageClassZSSEKMSKeyIdZSSEKMSEncryptionContextZTaggingNcCs2|
|_|dkrt
�}||_|dkr(t�}||_dS)
N)rer�rfrS�_osutil)r	rirjrkrrrr5hs






zS3Transfer.__init__cCs�|d
kri}|j||j
�
|jjj}|jdtdd�
|jdtdd�
|j	j
|
�
|jjkrl|j
|
||||�
n|j|
||||�
d
S)z�Upload a file to an S3 object.

        Variants have also been injected into S3 client, Bucket and Object.
        You don't have to use S3Transfer.upload_file() directly.
        Nzrequest-created.s3zs3upload-callback-disable)
Z	unique_idzs3upload-callback-enable)�_validate_all_known_args�ALLOWED_UPLOAD_ARGSre�meta�eventsZregister_firstr"Z
register_lastr$r�rVrfr��_multipart_upload�_put_object)r	r=r{rpr4ror�rrrr�qs














zS3Transfer.upload_filec
CsJ|jj
}||
d
|jj|
�
|d�� }|jjf|||d�|��

WdQRXdS)Nr
)
r4)rrrsr�)r�rXrVreZ
put_object)r	r=r{rpr4ror�rrrrr��s






zS3Transfer._put_objectc
Cs�|d
kri}|j||j
�
|j|
||�}|tjt�}y|j|
|||||�
Wn2tk
r�


tj	d|dd�
|j
j|�

�YnX|j
j||�
d
S)z�Download an S3 object to a file.

        Variants have also been injected into S3 client, Bucket and Object.
        You don't have to use S3Transfer.download_file() directly.
        Nz<Exception caught in download_file, removing partial file: %sT)
ru)
r��ALLOWED_DOWNLOAD_ARGS�_object_sizer8�extsepr�_download_filerxryrzr�r\r_)r	r{rpr=ror4r�Z
temp_filenamerrrr��s













zS3Transfer.download_filecCs8||jj
kr"|j|
|||||�
n|j|
||||�
dS)
N)rfr��_ranged_download�_get_object)r	r{rpr=r�ror4rrrr��s


zS3Transfer._download_filecCs0x*|
D]"}||krtd
|dj
|�
f�
�
qWdS)Nz/Invalid extra_args key '%s', must be one of: %sz, )�
ValueErrorr)r	�actualZallowedZkwargrrrr��s





z#S3Transfer._validate_all_known_argscCs*t|j
|j|j�}|j|
|||||�
dS)
N)r�rerfr�r�)r	r{rpr=r�ror4Z
downloaderrrrr��s



zS3Transfer._ranged_downloadc
Cs�|jj
}d}xlt|�
D]`}y|j|
||||�Stjtjttfk
rt
}	
z t	j
d
|	||dd�
|	}wWYdd}	~	XqXqWt|�
�
dS)NzCRetrying exception caught (%s), retrying request, (attempt %s / %s)T)
ru)rfr�r�_do_get_objectr�r�r�rrryrzr)
r	r{rpr=ror4r�r�r�rrrrr��s














zS3Transfer._get_objectc	sj|jj
f|
|d
�|��
}t|d|��|jj|d��,}x$t�f
dd�d�D]}|j|�

qJWWdQRXdS)N)rrrsr�r�cs
�jd
�
S)Ni )
rAr)
r�rrr��sz+S3Transfer._do_get_object.<locals>.<lambda>r�)rer�rNr�r7rKr�)	r	r{rpr=ror4r|r>r�r)
r�rr��s






zS3Transfer._do_get_objectcCs|jj
f|
|d
�|��
dS)N)rrrsZ
ContentLength)reZhead_object)r	r{rprorrrr��s

zS3Transfer._object_sizecCs(t|j
|j|j�}|j|
||||�
dS)
N)r`rerfr�r�)r	r=r{rpr4roZuploaderrrrr��s

zS3Transfer._multipart_upload)NN)NN)NN)r
rrr�r�r5r�r�r�r�r�r�r�r�r�r�rrrrr�FsL





























	r�i)
r)1rRr8r�r�Zloggingr�r�rr�concurrent.futuresr�Zbotocore.compatrZ6botocore.vendored.requests.packages.urllib3.exceptionsrZbotocore.exceptionsrZs3transfer.compatr]Zs3transfer.exceptionsrr�
__author__�__version__ZHandlerrZ	getLoggerr
ryZ
addHandlerZmovesr�r��objectr�rr"r$rxr%r&rNrSr`r�r�r�r�r�rrrr�<module>}sF
















K l