File: //proc/thread-self/root/lib/python3.6/site-packages/s3transfer/__pycache__/upload.cpython-36.pyc
3
\��_vs � @ s d dl Z d dlmZ d dlmZmZ d dlmZ d dlm Z d dlm
Z
d dlm Z d dlm Z d d l
mZ d d
l
mZ d d l
mZmZ G d d
� d
e�ZG dd� de�ZG dd� de�ZG dd� de�ZG dd� de�ZG dd� de�ZG dd� de
�ZG dd� de �ZG d d � d e �ZdS ) � N)�six)�seekable�readable)�IN_MEMORY_UPLOAD_TAG)�Task)�SubmissionTask)�CreateMultipartUploadTask)�CompleteMultipartUploadTask)�
get_callbacks)�get_filtered_dict)�DeferredOpenFile�ChunksizeAdjusterc @ s. e Zd Zd
dd�Zdd� Zdd� Zd d
� Zd S )�AggregatedProgressCallbacki � c C s || _ || _d| _dS )a� Aggregates progress updates for every provided progress callback
:type callbacks: A list of functions that accepts bytes_transferred
as a single argument
:param callbacks: The callbacks to invoke when threshold is reached
:type threshold: int
:param threshold: The progress threshold in which to take the
aggregated progress and invoke the progress callback with that
aggregated progress total
r N)�
_callbacks�
_threshold� _bytes_seen)�self� callbacks� threshold� r � /usr/lib/python3.6/upload.py�__init__ s z#AggregatedProgressCallback.__init__c C s& | j |7 _ | j | jkr"| j� d S )N)r r �_trigger_callbacks)r �bytes_transferredr r r �__call__- s z#AggregatedProgressCallback.__call__c C s | j dkr| j� dS )z@Flushes out any progress that has not been sent to its callbacksr N)r r )r r r r �flush2 s
z AggregatedProgressCallback.flushc C s& x| j D ]}|| jd� qW d| _d S )N)r r )r r )r �callbackr r r r 7 s z-AggregatedProgressCallback._trigger_callbacksN� )r )�__name__�
__module__� __qualname__r r r r r r r r r s
r c @ sJ e Zd ZdZdd� Zddd�Zdd� Zd d
� Zd d � Zd
d� Z dd� Z
dS )�InterruptReadera� Wrapper that can interrupt reading using an error
It uses a transfer coordinator to propagate an error if it notices
that a read is being made while the file is being read from.
:type fileobj: file-like obj
:param fileobj: The file-like object to read from
:type transfer_coordinator: s3transfer.futures.TransferCoordinator
:param transfer_coordinator: The transfer coordinator to use if the
reader needs to be interrupted.
c C s || _ || _d S )N)�_fileobj�_transfer_coordinator)r �fileobj�transfer_coordinatorr r r r J s zInterruptReader.__init__Nc C s | j jr| j j�| jj|�S )N)r$ Z exceptionr# �read)r �amountr r r r' N s zInterruptReader.readc C s | j j|� d S )N)r# �seek)r �wherer r r r) X s zInterruptReader.seekc C s
| j j� S )N)r# �tell)r r r r r+ [ s zInterruptReader.tellc C s | j j� d S )N)r# �close)r r r r r, ^ s zInterruptReader.closec C s | S )Nr )r r r r � __enter__a s zInterruptReader.__enter__c O s | j � d S )N)r, )r �args�kwargsr r r �__exit__d s zInterruptReader.__exit__)N) r r r! �__doc__r r' r) r+ r, r- r0 r r r r r" = s
r" c @ sf e Zd ZdZddd�Zedd� �Zdd� Zd d
� Zd d � Z d
d� Z
dd� Z dd� Z dd� Z
dd� ZdS )�UploadInputManageraJ Base manager class for handling various types of files for uploads
This class is typically used for the UploadSubmissionTask class to help
determine the following:
* How to determine the size of the file
* How to determine if a multipart upload is required
* How to retrieve the body for a PutObject
* How to retrieve the bodies for a set of UploadParts
The answers/implementations differ for the various types of file inputs
that may be accepted. All implementations must subclass and override
public methods from this class.
Nc C s || _ || _|| _d S )N)�_osutilr$ �_bandwidth_limiter)r �osutilr& �bandwidth_limiterr r r r w s zUploadInputManager.__init__c C s t d��dS )a Determines if the source for the upload is compatible with manager
:param upload_source: The source for which the upload will pull data
from.
:returns: True if the manager can handle the type of source specified
otherwise returns False.
z must implement _is_compatible()N)�NotImplementedError)�cls�
upload_sourcer r r �
is_compatible| s
z UploadInputManager.is_compatiblec C s t d��dS )a� Whether the body it provides are stored in-memory
:type operation_name: str
:param operation_name: The name of the client operation that the body
is being used for. Valid operation_names are ``put_object`` and
``upload_part``.
:rtype: boolean
:returns: True if the body returned by the manager will be stored in
memory. False if the manager will not directly store the body in
memory.
z%must implement store_body_in_memory()N)�NotImplemented)r �operation_namer r r �stores_body_in_memory� s
z(UploadInputManager.stores_body_in_memoryc C s t d��dS )z�Provides the transfer size of an upload
:type transfer_future: s3transfer.futures.TransferFuture
:param transfer_future: The future associated with upload request
z&must implement provide_transfer_size()N)r7 )r �transfer_futurer r r �provide_transfer_size� s z(UploadInputManager.provide_transfer_sizec C s t d��dS )a� Determines where a multipart upload is required
:type transfer_future: s3transfer.futures.TransferFuture
:param transfer_future: The future associated with upload request
:type config: s3transfer.manager.TransferConfig
:param config: The config associated to the transfer manager
:rtype: boolean
:returns: True, if the upload should be multipart based on
configuartion and size. False, otherwise.
z*must implement requires_multipart_upload()N)r7 )r r> �configr r r �requires_multipart_upload� s
z,UploadInputManager.requires_multipart_uploadc C s t d��dS )a� Returns the body to use for PutObject
:type transfer_future: s3transfer.futures.TransferFuture
:param transfer_future: The future associated with upload request
:type config: s3transfer.manager.TransferConfig
:param config: The config associated to the transfer manager
:rtype: s3transfer.utils.ReadFileChunk
:returns: A ReadFileChunk including all progress callbacks
associated with the transfer future.
z$must implement get_put_object_body()N)r7 )r r> r r r �get_put_object_body� s
z&UploadInputManager.get_put_object_bodyc C s t d��dS )a Yields the part number and body to use for each UploadPart
:type transfer_future: s3transfer.futures.TransferFuture
:param transfer_future: The future associated with upload request
:type chunksize: int
:param chunksize: The chunksize to use for this upload.
:rtype: int, s3transfer.utils.ReadFileChunk
:returns: Yields the part number and the ReadFileChunk including all
progress callbacks associated with the transfer future for that
specific yielded part.
z)must implement yield_upload_part_bodies()N)r7 )r r> � chunksizer r r �yield_upload_part_bodies� s z+UploadInputManager.yield_upload_part_bodiesc C s* t || j�}| jr&| jj|| jdd�}|S )NF)Zenabled)r" r$ r4 Zget_bandwith_limited_stream)r r% r r r �
_wrap_fileobj� s
z UploadInputManager._wrap_fileobjc C s t |d�}|rt|�gS g S )NZprogress)r
r )r r> r r r r �_get_progress_callbacks� s
z*UploadInputManager._get_progress_callbacksc C s dd� |D �S )Nc S s g | ]
}|j �qS r )r )�.0r r r r �
<listcomp>� s z;UploadInputManager._get_close_callbacks.<locals>.<listcomp>r )r Z aggregated_progress_callbacksr r r �_get_close_callbacks� s z'UploadInputManager._get_close_callbacks)N)r r r! r1 r � classmethodr: r= r? rA rB rD rE rF rI r r r r r2 h s
r2 c @ sd e Zd ZdZedd� �Zdd� Zdd� Zdd � Zd
d � Z d d
� Z
dd� Z dd� Z dd� Z
dd� ZdS )�UploadFilenameInputManagerz Upload utility for filenamesc C s t |tj�S )N)�
isinstancer Z string_types)r8 r9 r r r r: � s z(UploadFilenameInputManager.is_compatiblec C s dS )NFr )r r< r r r r= � s z0UploadFilenameInputManager.stores_body_in_memoryc C s |j j| jj|j jj�� d S )N)�metar? r3 Z
get_file_size� call_argsr% )r r> r r r r? � s z0UploadFilenameInputManager.provide_transfer_sizec C s |j j|jkS )N)rM �size�multipart_threshold)r r> r@ r r r rA � s z4UploadFilenameInputManager.requires_multipart_uploadc C sJ | j |�\}}| j|�}| j|�}| j|�}|jj}| jj|||||d�S )N)r% �
chunk_size�full_file_sizer �close_callbacks)�&_get_put_object_fileobj_with_full_sizerE rF rI rM rO r3 �#open_file_chunk_reader_from_fileobj)r r> r% � full_sizer rS rO r r r rB � s
z.UploadFilenameInputManager.get_put_object_bodyc c s� |j j}| j||�}x|td|d �D ]j}| j|�}| j|�}||d }| j|j jj|||d�\} }
| j | �} | j
j | ||
||d�} || fV q$W d S )N� )�
start_byte� part_sizerR )r% rQ rR r rS ) rM rO �_get_num_parts�rangerF rI �'_get_upload_part_fileobj_with_full_sizerN r% rE r3 rU ) r r> rC rR Z num_parts� part_numberr rS rX r% rV Zread_file_chunkr r r rD s
z3UploadFilenameInputManager.yield_upload_part_bodiesc C s t ||| jjd�}|S )N)Z
open_function)r r3 �open)r r% rX r r r �_get_deferred_open_file s z2UploadFilenameInputManager._get_deferred_open_filec C s" |j jj}|j j}| j|d�|fS )Nr )rM rN r% rO r_ )r r> r% rO r r r rT # s
zAUploadFilenameInputManager._get_put_object_fileobj_with_full_sizec K s |d }|d }| j ||�|fS )NrX rR )r_ )r r% r/ rX rV r r r r\ ( s zBUploadFilenameInputManager._get_upload_part_fileobj_with_full_sizec C s t tj|jjt|� ��S )N)�int�mathZceilrM rO �float)r r> rY r r r rZ - s z)UploadFilenameInputManager._get_num_partsN)r r r! r1 rJ r: r= r? rA rB rD r_ rT r\ rZ r r r r rK � s rK c @ s< e Zd ZdZedd� �Zdd� Zdd� Zdd � Zd
d � Z d S )
�UploadSeekableInputManagerz&Upload utility for an open file objectc C s t |�ot|�S )N)r r )r8 r9 r r r r: 4 s z(UploadSeekableInputManager.is_compatiblec C s |dkr dS dS d S )N�
put_objectFTr )r r< r r r r= 8 s z0UploadSeekableInputManager.stores_body_in_memoryc C sD |j jj}|j� }|jdd� |j� }|j|� |j j|| � d S )Nr � )rM rN r% r+ r) r? )r r> r% Zstart_positionZ end_positionr r r r? > s
z0UploadSeekableInputManager.provide_transfer_sizec K s |j |d �}tj|�t|�fS )NrY )r' r �BytesIO�len)r r% r/ �datar r r r\ J s zBUploadSeekableInputManager._get_upload_part_fileobj_with_full_sizec C s" |j jj}|j� |j j }||fS )N)rM rN r% r+ rO )r r> r% rO r r r rT Y s
zAUploadSeekableInputManager._get_put_object_fileobj_with_full_sizeN)
r r r! r1 rJ r: r= r? r\ rT r r r r rc 2 s rc c sh e Zd ZdZd� fdd� Zedd� �Zdd� Zd d
� Zd d � Z d
d� Z
dd� Z ddd�Z dd� Z
� ZS )� UploadNonSeekableInputManagerz7Upload utility for a file-like object that cannot seek.Nc s t t| �j|||� d| _d S )N� )�superri r �
_initial_data)r r5 r& r6 )� __class__r r r c s
z&UploadNonSeekableInputManager.__init__c C s t |�S )N)r )r8 r9 r r r r: h s z+UploadNonSeekableInputManager.is_compatiblec C s dS )NTr )r r< r r r r= l s z3UploadNonSeekableInputManager.stores_body_in_memoryc C s d S )Nr )r r> r r r r? o s z3UploadNonSeekableInputManager.provide_transfer_sizec C sT |j jd k r|j j|jkS |j jj}|j}| j||d�| _t| j�|k rLdS dS d S )NFT)rM rO rP rN r% �_readrl rg )r r> r@ r% r r r r rA t s
z7UploadNonSeekableInputManager.requires_multipart_uploadc C s@ | j |�}| j|�}|jjj}| j| j|j� ||�}d | _|S )N)rF rI rM rN r% �
_wrap_datarl r' )r r> r rS r% �bodyr r r rB � s
z1UploadNonSeekableInputManager.get_put_object_bodyc c sb |j jj}d}xN| j|�}| j|�}|d7 }| j||�}|s>P | j|||�}d }||fV qW d S )Nr rW )rM rN r% rF rI rn ro ) r r> rC Z file_objectr] r rS Z part_contentZ part_objectr r r rD � s
z6UploadNonSeekableInputManager.yield_upload_part_bodiesTc C sx t | j�dkr|j|�S |t | j�krL| jd|� }|rH| j|d� | _|S |t | j� }| j|j|� }|rtd| _|S )a=
Reads a specific amount of data from a stream and returns it. If there
is any data in initial_data, that will be popped out first.
:type fileobj: A file-like object that implements read
:param fileobj: The stream to read from.
:type amount: int
:param amount: The number of bytes to read from the stream.
:type truncate: bool
:param truncate: Whether or not to truncate initial_data after
reading from it.
:return: Generator which generates part bodies from the initial data.
r Nrj )rg rl r' )r r% r( �truncaterh Zamount_to_readr r r rn � s
z#UploadNonSeekableInputManager._readc C s. | j tj|��}| jj|t|�t|�||d�S )a�
Wraps data with the interrupt reader and the file chunk reader.
:type data: bytes
:param data: The data to wrap.
:type callbacks: list
:param callbacks: The callbacks associated with the transfer future.
:type close_callbacks: list
:param close_callbacks: The callbacks to be called when closing the
wrapper for the data.
:return: Fully wrapped data.
)r% rQ rR r rS )rE r rf r3 rU rg )r rh r rS r% r r r ro � s z(UploadNonSeekableInputManager._wrap_data)N)T)r r r! r1 r rJ r: r= r? rA rB rD rn ro �
__classcell__r r )rm r ri a s
*ri c @ s\ e Zd ZdZddddgZdgZdd� Zdd d
�Zd d � Zd
d� Z dd� Z
dd� Z dd� Z dS )�UploadSubmissionTaskz.Task for submitting tasks to execute an uploadZSSECustomerKeyZSSECustomerAlgorithmZSSECustomerKeyMD5Z RequestPayerc C sH t ttg}|jjj}x|D ]}|j|�r|S qW td|t|�f ��dS )ao Retrieves a class for managing input for an upload based on file type
:type transfer_future: s3transfer.futures.TransferFuture
:param transfer_future: The transfer future for the request
:rtype: class of UploadInputManager
:returns: The appropriate class to use for managing a specific type of
input for uploads.
z&Input %s of type: %s is not supported.N) rK rc ri rM rN r% r: � RuntimeError�type)r r> Z upload_manager_resolver_chainr% Zupload_manager_clsr r r � _get_upload_input_manager_cls� s
z2UploadSubmissionTask._get_upload_input_manager_clsNc C sd | j |�|| j|�}|jjdkr*|j|� |j||�sL| j||||||� n| j||||||� dS )a�
:param client: The client associated with the transfer manager
:type config: s3transfer.manager.TransferConfig
:param config: The transfer config associated with the transfer
manager
:type osutil: s3transfer.utils.OSUtil
:param osutil: The os utility associated to the transfer manager
:type request_executor: s3transfer.futures.BoundedExecutor
:param request_executor: The request executor associated with the
transfer manager
:type transfer_future: s3transfer.futures.TransferFuture
:param transfer_future: The transfer future associated with the
transfer request that tasks are being submitted for
N)rv r$ rM rO r? rA �_submit_upload_request�_submit_multipart_request)r �clientr@ r5 �request_executorr> r6 �upload_input_managerr r r �_submit s
z UploadSubmissionTask._submitc
C sN |j j}| j|d�}| jj|t| j||j|�|j|j|j d�dd�|d� d S )Nrd )ry r% �bucket�key�
extra_argsT)r& � main_kwargs�is_final)�tag)
rM rN �_get_upload_task_tagr$ �submit�
PutObjectTaskrB r} r~ r ) r ry r@ r5 rz r> r{ rN Zput_object_tagr r r rw 2 s z+UploadSubmissionTask._submit_upload_requestc
C s |j j}| jj|t| j||j|j|jd�d��}g } | j|j�}
| j |d�} |j j
} t � }
|
j |j
| �}|j||�}xJ|D ]B\}}| j| jj|t| j|||j|j||
d�d|id�| d�� qzW | j|j�}| jj|t| j||j|j|d�|| d�d d
�� d S ) N)ry r} r~ r )r& r� � upload_part)ry r% r} r~ r] r � upload_id)r& r� �pending_main_kwargs)r� )r� �partsT)r& r� r� r� )rM rN r$ r� r r} r~ r �_extra_upload_part_argsr� rO r
Zadjust_chunksizeZmultipart_chunksizerD �append�UploadPartTask� _extra_complete_multipart_argsr )r ry r@ r5 rz r> r{ rN Zcreate_multipart_futureZ part_futuresZextra_part_argsZupload_part_tagrO ZadjusterrC Z
part_iteratorr] r% Z complete_multipart_extra_argsr r r rx L sZ
z.UploadSubmissionTask._submit_multipart_requestc C s t || j�S )N)r �UPLOAD_PART_ARGS)r r r r r r� � s z,UploadSubmissionTask._extra_upload_part_argsc C s t || j�S )N)r �COMPLETE_MULTIPART_ARGS)r r r r r r� � s z3UploadSubmissionTask._extra_complete_multipart_argsc C s d }|j |�rt}|S )N)r= r )r r{ r< r� r r r r� � s
z)UploadSubmissionTask._get_upload_task_tag)N)
r r r! r1 r� r� rv r| rw rx r� r� r� r r r r rs � s
&Mrs c @ s e Zd ZdZdd� ZdS )r� z Task to do a nonmultipart uploadc
C s, |� }|j f |||d�|�� W dQ R X dS )aP
:param client: The client to use when calling PutObject
:param fileobj: The file to upload.
:param bucket: The name of the bucket to upload to
:param key: The name of the key to upload to
:param extra_args: A dictionary of any extra arguments that may be
used in the upload.
)�Bucket�Key�BodyN)rd )r ry r% r} r~ r rp r r r �_main� s zPutObjectTask._mainN)r r r! r1 r� r r r r r� � s r� c @ s e Zd ZdZdd� ZdS )r� z+Task to upload a part in a multipart uploadc C s> |�"}|j f |||||d�|��} W dQ R X | d }
|
|d�S )a�
:param client: The client to use when calling PutObject
:param fileobj: The file to upload.
:param bucket: The name of the bucket to upload to
:param key: The name of the key to upload to
:param upload_id: The id of the upload
:param part_number: The number representing the part of the multipart
upload
:param extra_args: A dictionary of any extra arguments that may be
used in the upload.
:rtype: dict
:returns: A dictionary representing a part::
{'Etag': etag_value, 'PartNumber': part_number}
This value can be appended to a list to be used to complete
the multipart upload.
)r� r� ZUploadId�
PartNumberr� N�ETag)r� r� )r� ) r ry r% r} r~ r� r] r rp ZresponseZetagr r r r� � s zUploadPartTask._mainN)r r r! r1 r� r r r r r� � s r� ) ra Zbotocore.compatr Zs3transfer.compatr r Zs3transfer.futuresr Zs3transfer.tasksr r r r Zs3transfer.utilsr
r r r
�objectr r" r2 rK rc ri rs r� r� r r r r �<module>
s* !+yQ/ D