| Title: | 'Amazon Web Service S3' File System |
|---|---|
| Description: | Access 'Amazon Web Service Simple Storage Service' ('S3') <https://aws.amazon.com/s3/> as if it were a file system. Interface based on the R package 'fs'. |
| Authors: | Dyfan Jones [aut, cre] |
| Maintainer: | Dyfan Jones <[email protected]> |
| License: | MIT + file LICENSE |
| Version: | 0.1.7 |
| Built: | 2024-10-28 05:20:51 UTC |
| Source: | https://github.com/DyfanJones/s3fs |
Access 'Amazon Web Service Simple Storage Service' ('S3') https://aws.amazon.com/s3/ as if it were a file system. Interface based on the R package 'fs'.
Maintainer: Dyfan Jones [email protected]
Useful links:
s3_file_copy copies files
s3_dir_copy copies the directory recursively to the new location
s3_file_copy( path, new_path, max_batch = fs_bytes("100MB"), overwrite = FALSE, ... ) s3_dir_copy( path, new_path, max_batch = fs_bytes("100MB"), overwrite = FALSE, ... )s3_file_copy( path, new_path, max_batch = fs_bytes("100MB"), overwrite = FALSE, ... ) s3_dir_copy( path, new_path, max_batch = fs_bytes("100MB"), overwrite = FALSE, ... )
path |
(character): path to a local directory of file or a uri. |
new_path |
(character): path to a local directory of file or a uri. |
max_batch |
(fs_bytes): Maximum batch size being uploaded with each multipart. |
overwrite |
(logical): Overwrite files if the exist. If this is |
... |
parameters to be passed to |
character vector of s3 uri paths
## Not run: # Require AWS S3 credentials temp_file = "temp.txt" file.create(temp_file) s3_file_copy( temp_file, "s3://MyBucket/temp_file.txt" ) ## End(Not run)## Not run: # Require AWS S3 credentials temp_file = "temp.txt" file.create(temp_file) s3_file_copy( temp_file, "s3://MyBucket/temp_file.txt" ) ## End(Not run)
s3_file_copy copies files
s3_dir_copy copies the directory recursively to the new location
s3_file_copy_async( path, new_path, max_batch = fs_bytes("100MB"), overwrite = FALSE, ... ) s3_dir_copy_async( path, new_path, max_batch = fs_bytes("100MB"), overwrite = FALSE, ... )s3_file_copy_async( path, new_path, max_batch = fs_bytes("100MB"), overwrite = FALSE, ... ) s3_dir_copy_async( path, new_path, max_batch = fs_bytes("100MB"), overwrite = FALSE, ... )
path |
(character): path to a local directory of file or a uri. |
new_path |
(character): path to a local directory of file or a uri. |
max_batch |
(fs_bytes): Maximum batch size being uploaded with each multipart. |
overwrite |
(logical): Overwrite files if the exist. If this is |
... |
parameters to be passed to |
return future object of s3_file_copy(), s3_dir_copy()
future s3_file_copy() s3_dir_copy()
s3_file_create create file on AWS S3, if file already exists it will be left unchanged.
s3_dir_create create empty directory of AWS S3.
s3_file_create(path, overwrite = FALSE, ...) s3_bucket_create( path, region_name = NULL, mode = c("private", "public-read", "public-read-write", "authenticated-read"), versioning = FALSE, ... ) s3_dir_create(path, overwrite = FALSE, ...)s3_file_create(path, overwrite = FALSE, ...) s3_bucket_create( path, region_name = NULL, mode = c("private", "public-read", "public-read-write", "authenticated-read"), versioning = FALSE, ... ) s3_dir_create(path, overwrite = FALSE, ...)
path |
(character): A character vector of path or s3 uri. |
overwrite |
(logical): Overwrite files if the exist. If this is |
... |
parameters to be passed to |
region_name |
(character): region for |
mode |
(character): A character of the mode |
versioning |
(logical) |
character vector of s3 uri paths
## Not run: # Require AWS S3 credentials temp_file = s3_file_temp(tmp_dir= "MyBucket") s3_file_create(temp_file) ## End(Not run)## Not run: # Require AWS S3 credentials temp_file = s3_file_temp(tmp_dir= "MyBucket") s3_file_create(temp_file) ## End(Not run)
s3_file_delete delete files in AWS S3
s3_dir_delete delete directories in AWS S3 recursively.
s3_file_delete(path, ...) s3_dir_delete(path)s3_file_delete(path, ...) s3_dir_delete(path)
path |
(character): A character vector of paths or s3 uris. |
... |
parameters to be passed to |
character vector of s3 uri paths
## Not run: # Require AWS S3 credentials temp_file = s3_file_temp(tmp_dir= "MyBucket") s3_file_create(temp_file) s3_file_delete(temp_file) ## End(Not run)## Not run: # Require AWS S3 credentials temp_file = s3_file_temp(tmp_dir= "MyBucket") s3_file_create(temp_file) s3_file_delete(temp_file) ## End(Not run)
s3_file_delete delete files in AWS S3
s3_dir_delete delete directories in AWS S3 recursively.
s3_file_delete_async(path, ...) s3_dir_delete_async(path)s3_file_delete_async(path, ...) s3_dir_delete_async(path)
path |
(character): A character vector of paths or s3 uris. |
... |
parameters to be passed to |
return future object of s3_file_delete() s3_dir_delete()
future s3_file_delete() s3_dir_delete()
s3_file_download downloads AWS S3 files to local
s3_file_download downloads AWS s3 directory to local
s3_file_download(path, new_path, overwrite = FALSE, ...) s3_dir_download(path, new_path, overwrite = FALSE, ...)s3_file_download(path, new_path, overwrite = FALSE, ...) s3_dir_download(path, new_path, overwrite = FALSE, ...)
path |
(character): A character vector of paths or uris |
new_path |
(character): A character vector of paths to the new locations. |
overwrite |
(logical): Overwrite files if the exist. If this is |
... |
parameters to be passed to |
character vector of s3 uri paths
## Not run: # Require AWS S3 credentials temp_file = s3_file_temp(tmp_dir= "MyBucket") s3_file_create(temp_file) s3_file_download(temp_file, "temp_file.txt") ## End(Not run)## Not run: # Require AWS S3 credentials temp_file = s3_file_temp(tmp_dir= "MyBucket") s3_file_create(temp_file) s3_file_download(temp_file, "temp_file.txt") ## End(Not run)
s3_file_download downloads AWS S3 files to local
s3_file_download downloads AWS s3 directory to local
s3_file_download_async(path, new_path, overwrite = FALSE, ...) s3_dir_download_async(path, new_path, overwrite = FALSE, ...)s3_file_download_async(path, new_path, overwrite = FALSE, ...) s3_dir_download_async(path, new_path, overwrite = FALSE, ...)
path |
(character): A character vector of paths or uris |
new_path |
(character): A character vector of paths to the new locations. |
overwrite |
(logical): Overwrite files if the exist. If this is |
... |
parameters to be passed to |
return future object of s3_file_download() s3_dir_download()
future s3_file_download() s3_dir_download()
s3_file_exists check if file exists in AWS S3
s3_dir_exists check if path is a directory in AWS S3
s3_file_exists(path) s3_dir_exists(path = ".")s3_file_exists(path) s3_dir_exists(path = ".")
path |
(character) s3 path to check |
logical vector if file exists
## Not run: # Require AWS S3 credentials temp_file = s3_file_temp(tmp_dir= "MyBucket") s3_file_create(temp_file) s3_file_exists(temp_file) ## End(Not run)## Not run: # Require AWS S3 credentials temp_file = s3_file_temp(tmp_dir= "MyBucket") s3_file_create(temp_file) s3_file_exists(temp_file) ## End(Not run)
Test for file types
s3_is_file(path) s3_is_dir(path) s3_is_bucket(path, ...) s3_is_file_empty(path)s3_is_file(path) s3_is_dir(path) s3_is_bucket(path, ...) s3_is_file_empty(path)
path |
(character): A character vector of paths or uris |
... |
parameters to be passed to |
s3_file_info returns file information within AWS S3 directory
s3_file_size returns file size in bytes
s3_dir_info returns file name information within AWS S3 directory
s3_dir_ls returns file name within AWS S3 directory
s3_file_info(path) s3_file_size(path) s3_dir_info( path = ".", type = c("any", "bucket", "directory", "file"), glob = NULL, regexp = NULL, invert = FALSE, recurse = FALSE, refresh = FALSE, ... ) s3_dir_ls( path = ".", type = c("any", "bucket", "directory", "file"), glob = NULL, regexp = NULL, invert = FALSE, recurse = FALSE, refresh = FALSE, ... )s3_file_info(path) s3_file_size(path) s3_dir_info( path = ".", type = c("any", "bucket", "directory", "file"), glob = NULL, regexp = NULL, invert = FALSE, recurse = FALSE, refresh = FALSE, ... ) s3_dir_ls( path = ".", type = c("any", "bucket", "directory", "file"), glob = NULL, regexp = NULL, invert = FALSE, recurse = FALSE, refresh = FALSE, ... )
path |
(character):A character vector of one or more paths. Can be path or s3 uri. |
type |
(character): File type(s) to return. Default ("any") returns all AWS S3 object types. |
glob |
(character): A wildcard pattern (e.g. |
regexp |
(character): A regular expression (e.g. |
invert |
(logical): If |
recurse |
(logical): Returns all AWS S3 objects in lower sub directories |
refresh |
(logical): Refresh cached in |
... |
parameters to be passed to |
s3_file_info A data.table with metadata for each file. Columns returned are as follows.
bucket_name (character): AWS S3 bucket of file
key (character): AWS S3 path key of file
uri (character): S3 uri of file
size (numeric): file size in bytes
type (character): file type (file or directory)
etag (character): An entity tag is an opague identifier
last_modified (POSIXct): Created date of file.
delete_marker (logical): Specifies retrieved a logical marker
accept_ranges (character): Indicates that a range of bytes was specified.
expiration (character): File expiration
restore (character): If file is archived
archive_status (character): Archive status
missing_meta (integer): Number of metadata entries not returned in "x-amz-meta" headers
version_id (character): version id of file
cache_control (character): caching behaviour for the request/reply chain
content_disposition (character): presentational information of file
content_encoding (character): file content encodings
content_language (character): what language the content is in
content_type (character): file MIME type
expires (POSIXct): date and time the file is no longer cacheable
website_redirect_location (character): redirects request for file to another
server_side_encryption (character): File server side encryption
metadata (list): metadata of file
sse_customer_algorithm (character): server-side encryption with a customer-provided encryption key
sse_customer_key_md5 (character): server-side encryption with a customer-provided encryption key
ssekms_key_id (character): ID of the Amazon Web Services Key Management Service
bucket_key_enabled (logical): s3 bucket key for server-side encryption with
storage_class (character): file storage class information
request_charged (character): indicates successfully charged for request
replication_status (character): return specific header if request involves a bucket that is either a source or a destination in a replication rule https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.head_object
parts_count (integer): number of count parts the file has
object_lock_mode (character): the file lock mode
object_lock_retain_until_date (POSIXct): date and time of when object_lock_mode expires
object_lock_legal_hold_status (character): file legal holding
s3_dir_info data.table with directory metadata
bucket_name (character): AWS S3 bucket of file
key (character): AWS S3 path key of file
uri (character): S3 uri of file
size (numeric): file size in bytes
version_id (character): version id of file
etag (character): An entity tag is an opague identifier
last_modified (POSIXct): Created date of file
s3_dir_ls character vector of s3 uri paths
## Not run: # Require AWS S3 credentials temp_file = s3_file_temp(tmp_dir= "MyBucket") s3_file_create(temp_file) s3_file_info(temp_file) ## End(Not run)## Not run: # Require AWS S3 credentials temp_file = s3_file_temp(tmp_dir= "MyBucket") s3_file_create(temp_file) s3_file_info(temp_file) ## End(Not run)
Constructs a s3 uri path
s3_path(..., ext = "")s3_path(..., ext = "")
... |
(character): Character vectors |
ext |
(character): An optional extension to append to the generated path |
character vector of s3 uri paths
## Not run: # Require AWS S3 credentials s3_path("my_bucket1", "my_bucket2") ## End(Not run)## Not run: # Require AWS S3 credentials s3_path("my_bucket1", "my_bucket2") ## End(Not run)
s3_path_dir returns the directory portion of s3 uri
s3_path_file returns the file name portion of the s3 uri path
s3_path_ext returns the last extension for a path.
s3_path_ext_remove removes the last extension and return the rest of the s3 uri.
s3_path_ext_set replace the extension with a new extension.
s3_path_dir(path) s3_path_file(path) s3_path_ext(path) s3_path_ext_remove(path) s3_path_ext_set(path, ext)s3_path_dir(path) s3_path_file(path) s3_path_ext(path) s3_path_ext_remove(path) s3_path_ext_set(path, ext)
path |
(character): A character vector of paths |
ext |
(character): New file extension |
## Not run: # Require AWS S3 credentials s3_path_dir("s3://my_bucket1/hi.txt") s3_path_file("s3://my_bucket1/hi.txt") ## End(Not run)## Not run: # Require AWS S3 credentials s3_path_dir("s3://my_bucket1/hi.txt") s3_path_file("s3://my_bucket1/hi.txt") ## End(Not run)
Change file permissions
s3_file_chmod( path, mode = c("private", "public-read", "public-read-write", "authenticated-read", "aws-exec-read", "bucket-owner-read", "bucket-owner-full-control") ) s3_bucket_chmod( path, mode = c("private", "public-read", "public-read-write", "authenticated-read") )s3_file_chmod( path, mode = c("private", "public-read", "public-read-write", "authenticated-read", "aws-exec-read", "bucket-owner-read", "bucket-owner-full-control") ) s3_bucket_chmod( path, mode = c("private", "public-read", "public-read-write", "authenticated-read") )
path |
(character): A character vector of path or s3 uri. |
mode |
(character): A character of the mode |
character vector of s3 uri paths
## Not run: # Require AWS S3 credentials temp_file = s3_file_temp(tmp_dir = "MyBucket") s3_file_create(temp_file) # Reset connection to connect to a different region s3_file_chmod( profile_name = "s3fs_example", region_name = "us-east-1", refresh = TRUE ) ## End(Not run)## Not run: # Require AWS S3 credentials temp_file = s3_file_temp(tmp_dir = "MyBucket") s3_file_create(temp_file) # Reset connection to connect to a different region s3_file_chmod( profile_name = "s3fs_example", region_name = "us-east-1", refresh = TRUE ) ## End(Not run)
Delete AWS S3 bucket including all objects in the bucket itself.
s3_bucket_delete(path)s3_bucket_delete(path)
path |
(character): A character vector of path or s3 uri. |
Generate presigned url to list S3 directories
s3_dir_ls_url(path, expiration = 3600L, recurse = FALSE, ...)s3_dir_ls_url(path, expiration = 3600L, recurse = FALSE, ...)
path |
(character): A character vector of paths or uris |
expiration |
(numeric): The number of seconds the presigned url is valid for. By default it expires in an hour (3600 seconds) |
recurse |
(logical): Returns all AWS S3 objects in lower sub directories |
... |
parameters passed to |
return character of urls
Print contents of directories in a tree-like format
s3_dir_tree(path, recurse = TRUE, ...)s3_dir_tree(path, recurse = TRUE, ...)
path |
(character): path A path to print the tree from |
recurse |
(logical): Returns all AWS S3 objects in lower sub directories |
... |
Additional arguments passed to s3_dir_ls. |
character vector of s3 uri paths
Move files to another location on AWS S3
s3_file_move(path, new_path, max_batch = 100 * MB, overwrite = FALSE, ...)s3_file_move(path, new_path, max_batch = 100 * MB, overwrite = FALSE, ...)
path |
(character): A character vector of s3 uri |
new_path |
(character): A character vector of s3 uri. |
max_batch |
(numeric): Maximum batch size being uploaded with each multipart. |
overwrite |
(logical): Overwrite files if the exist. If this is |
... |
parameters to be passed to |
character vector of s3 uri paths
## Not run: # Require AWS S3 credentials temp_file = s3_file_temp(tmp_dir= "MyBucket") s3_file_create(temp_file) s3_file_move(temp_file, "s3://MyBucket/new_file.txt") ## End(Not run)## Not run: # Require AWS S3 credentials temp_file = s3_file_temp(tmp_dir= "MyBucket") s3_file_create(temp_file) s3_file_move(temp_file, "s3://MyBucket/new_file.txt") ## End(Not run)
Move files to another location on AWS S3
s3_file_move_async( path, new_path, max_batch = 100 * MB, overwrite = FALSE, ... )s3_file_move_async( path, new_path, max_batch = 100 * MB, overwrite = FALSE, ... )
path |
(character): A character vector of s3 uri |
new_path |
(character): A character vector of s3 uri. |
max_batch |
(numeric): Maximum batch size being uploaded with each multipart. |
overwrite |
(logical): Overwrite files if the exist. If this is |
... |
parameters to be passed to |
return future object of s3_file_move()
This creates a file system "like" API based off fs
(e.g. dir_ls, file_copy, etc.) for AWS S3 storage. To set up AWS
credentials please look at
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html
s3_file_system( aws_access_key_id = NULL, aws_secret_access_key = NULL, aws_session_token = NULL, region_name = NULL, profile_name = NULL, endpoint = NULL, disable_ssl = FALSE, multipart_threshold = fs_bytes("2GB"), request_payer = FALSE, anonymous = FALSE, retries = 5, refresh = FALSE, ... )s3_file_system( aws_access_key_id = NULL, aws_secret_access_key = NULL, aws_session_token = NULL, region_name = NULL, profile_name = NULL, endpoint = NULL, disable_ssl = FALSE, multipart_threshold = fs_bytes("2GB"), request_payer = FALSE, anonymous = FALSE, retries = 5, refresh = FALSE, ... )
aws_access_key_id |
(character): AWS access key ID |
aws_secret_access_key |
(character): AWS secret access key |
aws_session_token |
(character): AWS temporary session token |
region_name |
(character): Default region when creating new connections |
profile_name |
(character): The name of a profile to use. If not given, then the default profile is used. |
endpoint |
(character): The complete URL to use for the constructed client. |
disable_ssl |
(logical): Whether or not to use SSL. By default, SSL is used. |
multipart_threshold |
(fs_bytes): Threshold to use multipart instead of standard copy and upload methods. |
request_payer |
(logical): Confirms that the requester knows that they will be charged for the request. |
anonymous |
(logical): Set up anonymous credentials when connecting to AWS S3. |
retries |
(numeric): max number of retry attempts |
refresh |
(logical): Refresh cached S3FileSystem class |
... |
Other parameters within |
S3FileSystem class invisible
## Not run: # Require AWS S3 credentials # Set up connection using profile s3_file_system(profile_name = "s3fs_example") # Reset connection to connect to a different region s3_file_system( profile_name = "s3fs_example", region_name = "us-east-1", refresh = TRUE ) ## End(Not run)## Not run: # Require AWS S3 credentials # Set up connection using profile s3_file_system(profile_name = "s3fs_example") # Reset connection to connect to a different region s3_file_system( profile_name = "s3fs_example", region_name = "us-east-1", refresh = TRUE ) ## End(Not run)
return the name which can be used as a temporary file
s3_file_temp(pattern = "file", tmp_dir = "", ext = "")s3_file_temp(pattern = "file", tmp_dir = "", ext = "")
pattern |
(character): A character vector with the non-random portion of the name. |
tmp_dir |
(character): The directory the file will be created in. By default
the cached s3 bucket will be applied otherwise |
ext |
(character): A character vector of one or more paths. |
character vector of s3 uri paths
## Not run: # Require AWS S3 credentials s3_file_temp(tmp_dir = "MyBucket") ## End(Not run)## Not run: # Require AWS S3 credentials s3_file_temp(tmp_dir = "MyBucket") ## End(Not run)
Generate presigned url for S3 object
s3_file_url(path, expiration = 3600L, ...)s3_file_url(path, expiration = 3600L, ...)
path |
(character): A character vector of paths or uris |
expiration |
(numeric): The number of seconds the presigned url is valid for. By default it expires in an hour (3600 seconds) |
... |
parameters to be passed to |
return character of urls
Get file versions
s3_file_version_info(path, ...)s3_file_version_info(path, ...)
path |
(character): A character vector of paths or uris |
... |
parameters to be passed to |
Construct an s3 uri path from path vector
s3_path_join(path)s3_path_join(path)
path |
(character): A character vector of one or more paths |
character vector of s3 uri paths
## Not run: # Require AWS S3 credentials s3_path_dir(c("s3://my_bucket1/hi.txt", "s3://my_bucket/bye.txt")) ## End(Not run)## Not run: # Require AWS S3 credentials s3_path_dir(c("s3://my_bucket1/hi.txt", "s3://my_bucket/bye.txt")) ## End(Not run)
Split s3 uri path to core components bucket, key and version id
s3_path_split(path)s3_path_split(path)
path |
(character): A character vector of one or more paths or s3 uri |
list character vectors splitting the s3 uri path in "Bucket", "Key" and "VersionId"
## Not run: # Require AWS S3 credentials s3_path_dir("s3://my_bucket1/hi.txt") ## End(Not run)## Not run: # Require AWS S3 credentials s3_path_dir("s3://my_bucket1/hi.txt") ## End(Not run)
This creates a file system "like" API based off fs
(e.g. dir_ls, file_copy, etc.) for AWS S3 storage.
s3_cacheCache AWS S3
s3_cache_bucketCached s3 bucket
s3_clientpaws s3 client
region_nameAWS region when creating new connections
profile_nameThe name of a profile to use
multipart_thresholdThreshold to use multipart
request_payerThreshold to use multipart
pidGet the process ID of the R Session
retriesnumber of retries
new()
Initialize S3FileSystem class
S3FileSystem$new(
aws_access_key_id = NULL,
aws_secret_access_key = NULL,
aws_session_token = NULL,
region_name = NULL,
profile_name = NULL,
endpoint = NULL,
disable_ssl = FALSE,
multipart_threshold = fs_bytes("2GB"),
request_payer = FALSE,
anonymous = FALSE,
...
)aws_access_key_id(character): AWS access key ID
aws_secret_access_key(character): AWS secret access key
aws_session_token(character): AWS temporary session token
region_name(character): Default region when creating new connections
profile_name(character): The name of a profile to use. If not given, then the default profile is used.
endpoint(character): The complete URL to use for the constructed client.
disable_ssl(logical): Whether or not to use SSL. By default, SSL is used.
multipart_threshold(fs_bytes): Threshold to use multipart instead of standard copy and upload methods.
request_payer(logical): Confirms that the requester knows that they will be charged for the request.
anonymous(logical): Set up anonymous credentials when connecting to AWS S3.
...Other parameters within paws client.
file_chmod()
Change file permissions
S3FileSystem$file_chmod(
path,
mode = c("private", "public-read", "public-read-write", "authenticated-read",
"aws-exec-read", "bucket-owner-read", "bucket-owner-full-control")
)path(character): A character vector of path or s3 uri.
mode(character): A character of the mode
character vector of s3 uri paths
file_copy()
copy files
S3FileSystem$file_copy(
path,
new_path,
max_batch = fs_bytes("100MB"),
overwrite = FALSE,
...
)path(character): path to a local directory of file or a uri.
new_path(character): path to a local directory of file or a uri.
max_batch(fs_bytes): Maximum batch size being uploaded with each multipart.
overwrite(logical): Overwrite files if the exist. If this is FALSE
and the file exists an error will be thrown.
...parameters to be passed to s3_put_object
character vector of s3 uri paths
file_create()
Create file on AWS S3, if file already exists it will be left unchanged.
S3FileSystem$file_create(path, overwrite = FALSE, ...)
path(character): A character vector of path or s3 uri.
overwrite(logical): Overwrite files if the exist. If this is FALSE
and the file exists an error will be thrown.
...parameters to be passed to s3_put_object
character vector of s3 uri paths
file_delete()
Delete files in AWS S3
S3FileSystem$file_delete(path, ...)
path(character): A character vector of paths or s3 uris.
...parameters to be passed to s3_delete_objects
character vector of s3 uri paths
file_download()
Downloads AWS S3 files to local
S3FileSystem$file_download(path, new_path, overwrite = FALSE, ...)
path(character): A character vector of paths or uris
new_path(character): A character vector of paths to the new locations.
overwrite(logical): Overwrite files if the exist. If this is FALSE
and the file exists an error will be thrown.
...parameters to be passed to s3_get_object
character vector of s3 uri paths
file_exists()
Check if file exists in AWS S3
S3FileSystem$file_exists(path)
path(character) s3 path to check
logical vector if file exists
file_info()
Returns file information within AWS S3 directory
S3FileSystem$file_info(path)
path(character): A character vector of paths or uris.
A data.table with metadata for each file. Columns returned are as follows.
bucket_name (character): AWS S3 bucket of file
key (character): AWS S3 path key of file
uri (character): S3 uri of file
size (numeric): file size in bytes
type (character): file type (file or directory)
etag (character): An entity tag is an opague identifier
last_modified (POSIXct): Created date of file.
delete_marker (logical): Specifies retrieved a logical marker
accept_ranges (character): Indicates that a range of bytes was specified.
expiration (character): File expiration
restore (character): If file is archived
archive_status (character): Archive status
missing_meta (integer): Number of metadata entries not returned in "x-amz-meta" headers
version_id (character): version id of file
cache_control (character): caching behaviour for the request/reply chain
content_disposition (character): presentational information of file
content_encoding (character): file content encodings
content_language (character): what language the content is in
content_type (character): file MIME type
expires (POSIXct): date and time the file is no longer cacheable
website_redirect_location (character): redirects request for file to another
server_side_encryption (character): File server side encryption
metadata (list): metadata of file
sse_customer_algorithm (character): server-side encryption with a customer-provided encryption key
sse_customer_key_md5 (character): server-side encryption with a customer-provided encryption key
ssekms_key_id (character): ID of the Amazon Web Services Key Management Service
bucket_key_enabled (logical): s3 bucket key for server-side encryption with
storage_class (character): file storage class information
request_charged (character): indicates successfully charged for request
replication_status (character): return specific header if request involves a bucket that is either a source or a destination in a replication rule https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.head_object
parts_count (integer): number of count parts the file has
object_lock_mode (character): the file lock mode
object_lock_retain_until_date (POSIXct): date and time of when object_lock_mode expires
object_lock_legal_hold_status (character): file legal holding
file_move()
Move files to another location on AWS S3
S3FileSystem$file_move(
path,
new_path,
max_batch = fs_bytes("100MB"),
overwrite = FALSE,
...
)path(character): A character vector of s3 uri
new_path(character): A character vector of s3 uri.
max_batch(fs_bytes): Maximum batch size being uploaded with each multipart.
overwrite(logical): Overwrite files if the exist. If this is FALSE
and the file exists an error will be thrown.
...parameters to be passed to s3_copy_object
character vector of s3 uri paths
file_size()
Return file size in bytes
S3FileSystem$file_size(path)
path(character): A character vector of s3 uri
file_stream_in()
Streams in AWS S3 file as a raw vector
S3FileSystem$file_stream_in(path, ...)
path(character): A character vector of paths or s3 uri
...parameters to be passed to s3_get_object
list of raw vectors containing the contents of the file
file_stream_out()
Streams out raw vector to AWS S3 file
S3FileSystem$file_stream_out(
obj,
path,
max_batch = fs_bytes("100MB"),
overwrite = FALSE,
...
)obj(raw|character): A raw vector, rawConnection, url to be streamed up to AWS S3.
path(character): A character vector of paths or s3 uri
max_batch(fs_bytes): Maximum batch size being uploaded with each multipart.
overwrite(logical): Overwrite files if the exist. If this is FALSE
and the file exists an error will be thrown.
...parameters to be passed to s3_put_object
character vector of s3 uri paths
file_temp()
return the name which can be used as a temporary file
S3FileSystem$file_temp(pattern = "file", tmp_dir = "", ext = "")
pattern(character): A character vector with the non-random portion of the name.
tmp_dir(character): The directory the file will be created in.
ext(character): A character vector of one or more paths.
character vector of s3 uri paths
file_tag_delete()
Delete file tags
S3FileSystem$file_tag_delete(path)
path(character): A character vector of paths or s3 uri
...parameters to be passed to s3_put_object
character vector of s3 uri paths
file_tag_info()
Get file tags
S3FileSystem$file_tag_info(path)
path(character): A character vector of paths or s3 uri
data.table of file version metadata
bucket_name (character): AWS S3 bucket of file
key (character): AWS S3 path key of file
uri (character): S3 uri of file
size (numeric): file size in bytes
version_id (character): version id of file
tag_key (character): name of tag
tag_value (character): tag value
file_tag_update()
Update file tags
S3FileSystem$file_tag_update(path, tags, overwrite = FALSE)
path(character): A character vector of paths or s3 uri
tags(list): Tags to be applied
overwrite(logical): To overwrite tagging or to modify inplace. Default will modify inplace.
character vector of s3 uri paths
file_touch()
Similar to fs::file_touch this does not create the file if
it does not exist. Use s3fs$file_create() to do this if needed.
S3FileSystem$file_touch(path, ...)
path(character): A character vector of paths or s3 uri
...parameters to be passed to s3_copy_object
character vector of s3 uri paths
file_upload()
Uploads files to AWS S3
S3FileSystem$file_upload(
path,
new_path,
max_batch = fs_bytes("100MB"),
overwrite = FALSE,
...
)path(character): A character vector of local file paths to upload to AWS S3
new_path(character): A character vector of AWS S3 paths or uri's of the new locations.
max_batch(fs_bytes): Maximum batch size being uploaded with each multipart.
overwrite(logical): Overwrite files if the exist. If this is FALSE
and the file exists an error will be thrown.
...parameters to be passed to s3_put_object
and s3_create_multipart_upload
character vector of s3 uri paths
file_url()
Generate presigned url for S3 object
S3FileSystem$file_url(path, expiration = 3600L, ...)
path(character): A character vector of paths or uris
expiration(numeric): The number of seconds the presigned url is valid for. By default it expires in an hour (3600 seconds)
...parameters passed to s3_get_object
return character of urls
file_version_info()
Get file versions
S3FileSystem$file_version_info(path, ...)
path(character): A character vector of paths or uris
...parameters to be passed to s3_list_object_versions
return data.table with file version info, columns below:
bucket_name (character): AWS S3 bucket of file
key (character): AWS S3 path key of file
uri (character): S3 uri of file
size (numeric): file size in bytes
version_id (character): version id of file
owner (character): file owner
etag (character): An entity tag is an opague identifier
last_modified (POSIXct): Created date of file.
is_file()
Test for file types
S3FileSystem$is_file(path)
path(character): A character vector of paths or uris
logical vector if object is a file
is_dir()
Test for file types
S3FileSystem$is_dir(path)
path(character): A character vector of paths or uris
logical vector if object is a directory
is_bucket()
Test for file types
S3FileSystem$is_bucket(path, ...)
path(character): A character vector of paths or uris
...parameters to be passed to s3_list_objects_v2
logical vector if object is a AWS S3 bucket
is_file_empty()
Test for file types
S3FileSystem$is_file_empty(path)
path(character): A character vector of paths or uris
logical vector if file is empty
bucket_chmod()
Change bucket permissions
S3FileSystem$bucket_chmod(
path,
mode = c("private", "public-read", "public-read-write", "authenticated-read")
)path(character): A character vector of path or s3 uri.
mode(character): A character of the mode
character vector of s3 uri paths
bucket_create()
Create bucket
S3FileSystem$bucket_create(
path,
region_name = NULL,
mode = c("private", "public-read", "public-read-write", "authenticated-read"),
versioning = FALSE,
...
)path(character): A character vector of path or s3 uri.
region_name(character): aws region
mode(character): A character of the mode
versioning(logical): Whether to set the bucket to versioning or not.
...parameters to be passed to s3_create_bucket
character vector of s3 uri paths
bucket_delete()
Delete bucket
S3FileSystem$bucket_delete(path)
path(character): A character vector of path or s3 uri.
dir_copy()
Copies the directory recursively to the new location.
S3FileSystem$dir_copy(
path,
new_path,
max_batch = fs_bytes("100MB"),
overwrite = FALSE,
...
)path(character): path to a local directory of file or a uri.
new_path(character): path to a local directory of file or a uri.
max_batch(fs_bytes): Maximum batch size being uploaded with each multipart.
overwrite(logical): Overwrite files if the exist. If this is FALSE
and the file exists an error will be thrown.
...parameters to be passed to s3_put_object
and s3_create_multipart_upload
character vector of s3 uri paths
dir_create()
Create empty directory
S3FileSystem$dir_create(path, overwrite = FALSE, ...)
path(character): A vector of directory or uri to be created in AWS S3
overwrite(logical): Overwrite files if the exist. If this is FALSE
and the file exists an error will be thrown.
...parameters to be passed to s3_put_object
character vector of s3 uri paths
dir_delete()
Delete contents and directory in AWS S3
S3FileSystem$dir_delete(path)
path(character): A vector of paths or uris to directories to be deleted.
character vector of s3 uri paths
dir_exists()
Check if path exists in AWS S3
S3FileSystem$dir_exists(path = ".")
path(character) aws s3 path to be checked
character vector of s3 uri paths
dir_download()
Downloads AWS S3 files to local
S3FileSystem$dir_download(path, new_path, overwrite = FALSE, ...)
path(character): A character vector of paths or uris
new_path(character): A character vector of paths to the new locations.
Please ensure directories end with a /.
overwrite(logical): Overwrite files if the exist. If this is FALSE
and the file exists an error will be thrown.
...parameters to be passed to s3_get_object
character vector of s3 uri paths
dir_info()
Returns file information within AWS S3 directory
S3FileSystem$dir_info(
path = ".",
type = c("any", "bucket", "directory", "file"),
glob = NULL,
regexp = NULL,
invert = FALSE,
recurse = FALSE,
refresh = FALSE,
...
)path(character):A character vector of one or more paths. Can be path or s3 uri.
type(character): File type(s) to return. Default ("any") returns all AWS S3 object types.
glob(character): A wildcard pattern (e.g. *.csv), passed onto
grep() to filter paths.
regexp(character): A regular expression (e.g. [.]csv$),
passed onto grep() to filter paths.
invert(logical): If code return files which do not match.
recurse(logical): Returns all AWS S3 objects in lower sub directories
refresh(logical): Refresh cached in s3_cache.
...parameters to be passed to s3_list_objects_v2
data.table with directory metadata
bucket_name (character): AWS S3 bucket of file
key (character): AWS S3 path key of file
uri (character): S3 uri of file
size (numeric): file size in bytes
version_id (character): version id of file
etag (character): An entity tag is an opague identifier
last_modified (POSIXct): Created date of file
dir_ls()
Returns file name within AWS S3 directory
S3FileSystem$dir_ls(
path = ".",
type = c("any", "bucket", "directory", "file"),
glob = NULL,
regexp = NULL,
invert = FALSE,
recurse = FALSE,
refresh = FALSE,
...
)path(character):A character vector of one or more paths. Can be path or s3 uri.
type(character): File type(s) to return. Default ("any") returns all AWS S3 object types.
glob(character): A wildcard pattern (e.g. *.csv), passed onto
grep() to filter paths.
regexp(character): A regular expression (e.g. [.]csv$),
passed onto grep() to filter paths.
invert(logical): If code return files which do not match.
recurse(logical): Returns all AWS S3 objects in lower sub directories
refresh(logical): Refresh cached in s3_cache.
...parameters to be passed to s3_list_objects_v2
character vector of s3 uri paths
dir_ls_url()
Generate presigned url to list S3 directories
S3FileSystem$dir_ls_url(path, expiration = 3600L, recurse = FALSE, ...)
path(character): A character vector of paths or uris
expiration(numeric): The number of seconds the presigned url is valid for. By default it expires in an hour (3600 seconds)
recurse(logical): Returns all AWS S3 objects in lower sub directories
...parameters passed to s3_list_objects_v2
return character of urls
dir_tree()
Print contents of directories in a tree-like format
S3FileSystem$dir_tree(path, recurse = TRUE, ...)
path(character): path A path to print the tree from
recurse(logical): Returns all AWS S3 objects in lower sub directories
...Additional arguments passed to s3_dir_ls.
character vector of s3 uri paths
dir_upload()
Uploads local directory to AWS S3
S3FileSystem$dir_upload(
path,
new_path,
max_batch = fs_bytes("100MB"),
overwrite = FALSE,
...
)path(character): A character vector of local file paths to upload to AWS S3
new_path(character): A character vector of AWS S3 paths or uri's of the new locations.
max_batch(fs_bytes): Maximum batch size being uploaded with each multipart.
overwrite(logical): Overwrite files if the exist. If this is FALSE
and the file exists an error will be thrown.
...parameters to be passed to s3_put_object
and s3_create_multipart_upload
character vector of s3 uri paths
path()
Constructs a s3 uri path
S3FileSystem$path(..., ext = "")
...(character): Character vectors
ext(character): An optional extension to append to the generated path
character vector of s3 uri paths
path_dir()
Returns the directory portion of s3 uri
S3FileSystem$path_dir(path)
path(character): A character vector of paths
character vector of s3 uri paths
path_ext()
Returns the last extension for a path.
S3FileSystem$path_ext(path)
path(character): A character vector of paths
character s3 uri file extension
path_ext_remove()
Removes the last extension and return the rest of the s3 uri.
S3FileSystem$path_ext_remove(path)
path(character): A character vector of paths
character vector of s3 uri paths
path_ext_set()
Replace the extension with a new extension.
S3FileSystem$path_ext_set(path, ext)
path(character): A character vector of paths
ext(character): New file extension
character vector of s3 uri paths
path_file()
Returns the file name portion of the s3 uri path
S3FileSystem$path_file(path)
path(character): A character vector of paths
character vector of file names
path_join()
Construct an s3 uri path from path vector
S3FileSystem$path_join(parts)
parts(character): A character vector of one or more paths
character vector of s3 uri paths
path_split()
Split s3 uri path to core components bucket, key and version id
S3FileSystem$path_split(path)
path(character): A character vector of one or more paths or s3 uri
list character vectors splitting the s3 uri path in "Bucket", "Key" and "VersionId"
clear_cache()
Clear S3 Cache
S3FileSystem$clear_cache(path = NULL)
path(character): s3 path to be cl
clone()
The objects of this class are cloneable with this method.
S3FileSystem$clone(deep = FALSE)
deepWhether to make a deep clone.
This method will only update the modification time of the AWS S3 object.
s3_file_stream_in streams in AWS S3 file as a raw vector
s3_file_stream_out streams raw vector out to AWS S3 file
s3_file_stream_in(path, ...) s3_file_stream_out( obj, path, max_batch = fs_bytes("100MB"), overwrite = FALSE, ... )s3_file_stream_in(path, ...) s3_file_stream_out( obj, path, max_batch = fs_bytes("100MB"), overwrite = FALSE, ... )
path |
(character): A character vector of paths or s3 uri |
... |
parameters to be passed to |
obj |
(raw|character): A raw vector, rawConnection, url to be streamed up to AWS S3. |
max_batch |
(fs_bytes): Maximum batch size being uploaded with each multipart. |
overwrite |
(logical): Overwrite files if the exist. If this is |
list of raw vectors containing the contents of the file
## Not run: # Require AWS S3 credentials obj = list(charToRaw("contents1"), charToRaw("contents2")) dir = s3_file_temp(tmp_dir = "MyBucket") path = s3_path(dir, letters[1:2], ext = "txt") s3_file_stream_out(obj, path) s3_file_stream_in(path) ## End(Not run)## Not run: # Require AWS S3 credentials obj = list(charToRaw("contents1"), charToRaw("contents2")) dir = s3_file_temp(tmp_dir = "MyBucket") path = s3_path(dir, letters[1:2], ext = "txt") s3_file_stream_out(obj, path) s3_file_stream_in(path) ## End(Not run)
s3_file_stream_in streams in AWS S3 file as a raw vector
s3_file_stream_out streams raw vector out to AWS S3 file
s3_file_stream_in_async(path, ...) s3_file_stream_out_async( obj, path, max_batch = fs_bytes("100MB"), overwrite = FALSE, ... )s3_file_stream_in_async(path, ...) s3_file_stream_out_async( obj, path, max_batch = fs_bytes("100MB"), overwrite = FALSE, ... )
path |
(character): A character vector of paths or s3 uri |
... |
parameters to be passed to |
obj |
(raw|character): A raw vector, rawConnection, url to be streamed up to AWS S3. |
max_batch |
(fs_bytes): Maximum batch size being uploaded with each multipart. |
overwrite |
(logical): Overwrite files if the exist. If this is |
return future object of s3_file_stream_in() s3_file_stream_out()
future s3_file_move() s3_file_stream_in() s3_file_stream_out()
s3_file_tag_delete delete file tags
s3_file_tag_info get file tags
s3_file_tag_info
s3_file_tag_delete(path) s3_file_tag_info(path) s3_file_tag_update(path, tags, overwrite = FALSE)s3_file_tag_delete(path) s3_file_tag_info(path) s3_file_tag_update(path, tags, overwrite = FALSE)
path |
(character): A character vector of paths or s3 uri |
tags |
(list): Tags to be applied |
overwrite |
(logical): To overwrite tagging or to modify inplace. Default will modify inplace. |
Similar to fs::file_touch this does not create the file if
it does not exist. Use s3_file_create to do this if needed.
s3_file_touch(path, ...)s3_file_touch(path, ...)
path |
(character): A character vector of paths or s3 uri |
... |
parameters to be passed to |
character vector of s3 uri paths
This method will only update the modification time of the AWS S3 object.
## Not run: # Require AWS S3 credentials dir = s3_file_temp(tmp_dir = "MyBucket") path = s3_path(dir, letters[1:2], ext = "txt") s3_file_touch(path) ## End(Not run)## Not run: # Require AWS S3 credentials dir = s3_file_temp(tmp_dir = "MyBucket") path = s3_path(dir, letters[1:2], ext = "txt") s3_file_touch(path) ## End(Not run)
s3_file_upload upload files to AWS S3
s3_dir_upload upload directory to AWS S3
s3_file_upload( path, new_path, max_batch = fs_bytes("100MB"), overwrite = FALSE, ... ) s3_dir_upload(path, new_path, max_batch, overwrite = FALSE, ...)s3_file_upload( path, new_path, max_batch = fs_bytes("100MB"), overwrite = FALSE, ... ) s3_dir_upload(path, new_path, max_batch, overwrite = FALSE, ...)
path |
(character): A character vector of local file paths to upload to AWS S3 |
new_path |
(character): A character vector of AWS S3 paths or uri's of the new locations. |
max_batch |
(fs_bytes): Maximum batch size being uploaded with each multipart. |
overwrite |
(logical): Overwrite files if the exist. If this is |
... |
parameters to be passed to |
character vector of s3 uri paths
s3_file_upload upload files to AWS S3
s3_dir_upload upload directory to AWS S3
s3_file_upload_async( path, new_path, max_batch = fs_bytes("100MB"), overwrite = FALSE, ... ) s3_dir_upload_async(path, new_path, max_batch, overwrite = FALSE, ...)s3_file_upload_async( path, new_path, max_batch = fs_bytes("100MB"), overwrite = FALSE, ... ) s3_dir_upload_async(path, new_path, max_batch, overwrite = FALSE, ...)
path |
(character): A character vector of local file paths to upload to AWS S3 |
new_path |
(character): A character vector of AWS S3 paths or uri's of the new locations. |
max_batch |
(fs_bytes): Maximum batch size being uploaded with each multipart. |
overwrite |
(logical): Overwrite files if the exist. If this is |
... |
parameters to be passed to |
return future object of s3_file_upload() s3_dir_upload()
future s3_file_move() s3_file_upload() s3_dir_upload()