records_mover package
Subpackages
- records_mover.airflow package
- records_mover.records package
Module contents
- class records_mover.Session(default_db_creds_name=None, default_aws_creds_name=PleaseInfer.token, default_gcp_creds_name=PleaseInfer.token, session_type=PleaseInfer.token, scratch_s3_url=PleaseInfer.token, creds=PleaseInfer.token, default_db_facts=PleaseInfer.token, default_boto3_session=PleaseInfer.token, default_gcp_creds=PleaseInfer.token, default_gcs_client=PleaseInfer.token, scratch_gcs_url=PleaseInfer.token)
Bases:
object
- Parameters
default_db_creds_name (Optional[str]) –
default_aws_creds_name (Union[None, str, PleaseInfer]) –
default_gcp_creds_name (Union[None, str, PleaseInfer]) –
session_type (Union[str, PleaseInfer]) –
scratch_s3_url (Union[None, str, PleaseInfer]) –
creds (Union[BaseCreds, PleaseInfer]) –
default_db_facts (Union[PleaseInfer, Dict[str, Any]]) –
default_boto3_session (Optional[Union[PleaseInfer, boto3.session.Session]]) –
default_gcp_creds (Optional[Union[PleaseInfer, google.auth.credentials.Credentials]]) –
default_gcs_client (Optional[Union[PleaseInfer, google.cloud.storage.Client]]) –
scratch_gcs_url (Union[None, str, PleaseInfer]) –
- __init__(default_db_creds_name=None, default_aws_creds_name=PleaseInfer.token, default_gcp_creds_name=PleaseInfer.token, session_type=PleaseInfer.token, scratch_s3_url=PleaseInfer.token, creds=PleaseInfer.token, default_db_facts=PleaseInfer.token, default_boto3_session=PleaseInfer.token, default_gcp_creds=PleaseInfer.token, default_gcs_client=PleaseInfer.token, scratch_gcs_url=PleaseInfer.token)
This is an object which ties together configuration on how to do key things in order to move records.
It tries to autoconfigure as much as possible - in many cases you won’t need to specify any constructor arguments at all.
Generally unless otherwise configured, this class will look up and use the default credentials for things like AWS and GCP if they exist and are needed for an operation. When running in a managed environment like Apache Airflow (session_type = “airflow”), that might mean looking up an Airflow Connection via the Airflow Python API. On the command line (session_type = “cli”), that might mean using e.g., the AWS or GCP Python APIs to pull any default credentials which have been configured. In other environments (e.g., containerized systems) you may way want to use environment variables whenever possible to specify exactly what is desired (sesssion_type = ‘env’).
- Parameters
default_db_creds_name (Optional[str]) – Name of the database credential to used when
records_mover.Session.get_default_db_engine()
is called. If not specified, the default will depend on the session type.default_aws_creds_name (Union[None, str, PleaseInfer]) – Name of the AWS IAM credential to used when needed, e.g. when reading or writing to an s3:// URL. This will be inferred unless directly specified.
default_gcp_creds_name (Union[None, str, PleaseInfer]) – Name of the GCP Cloud IAM credential to used when needed, e.g. when reading or writing to an gs:// URL. This will be inferred unless directly specified.
session_type (Union[str, PleaseInfer]) – What assumptions to use when inferring and/or looking up credentials. Valid values of “airflow” (for code running in Apache Airflow), “cli” (for running on the command-line”, “lpass” (for using the LastPass password manager for credentials), and ‘env’ (for looking up credentials via environment variables). This will be inferred unless directly specified.
scratch_s3_url (Union[None, str, PleaseInfer]) – An s3:// URL used as a base directory where temporary files/directories can be created. This is necessary for Amazon Redshift, which supports only S3 for bulk import/export.
default_db_facts (Union[PleaseInfer, Dict[str, Any]]) – Information about the database connection that should be made. This is a dictionary with string keys of type DBFacts
default_boto3_session (Optional[Union[PleaseInfer, boto3.session.Session]]) – The boto3.Session object used when needed, e.g. when reading or writing to an s3:// URL. This will be inferred unless directly specified.
default_gcp_creds (Optional[Union[PleaseInfer, google.auth.credentials.Credentials]]) – The google.auth.credentials.Credentials object to used when needed, e.g. when reading or writing to an gs:// URL. This will be inferred unless directly specified.
default_gcs_client (Optional[Union[PleaseInfer, google.cloud.storage.Client]]) – The google.cloud.storage.Client object to be used when needed, e.g. when reading or writing to an gs:// URL. This will be inferred unless directly specified.
creds (Union[BaseCreds, PleaseInfer]) – Experimental interface; do not use.
scratch_gcs_url (Union[None, str, PleaseInfer]) – A gs:// URL used as a base directory where temporary files/directories can be created. This can be helpful for large imports into Google BigQuery.
- Return type
None
- get_default_db_engine()
Provide the database object corresponding to the default database credentials. The details of how that credential is looked up depends on the session_type determined in the constructor, but can be overridden using the default_db_creds_name parameter.
- Returns
SQLALchemy Engine object
- Return type
Engine
- get_db_engine(db_creds_name, creds_provider=None)
Provide a database object corresponding to a given credential name. The details of how that credential is looked up depends on the session_type determined in the constructor.
- Parameters
db_creds_name (str) – Credential name to look up using the configured credentials provider.
creds_provider (Optional[BaseCreds]) –
- Returns
SQLALchemy Engine object
- Return type
Engine
- set_stream_logging(name='records_mover', level=20, stream=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, fmt='%(asctime)s - %(message)s', datefmt='%H:%M:%S')
records-mover logs details about its operations using Python logging. This method is a simple way to configure that logging to be output to a stream (by default, stdout).
You can use it for other things (e.g., dependencies of records-mover) by adjusting the ‘name’ argument.
- Parameters
name (str) – Name of the package to set logging under. If set to ‘foo’, you can set a log variable FOO_LOG_LEVEL to the log level threshold you’d like to set (INFO/WARNING/etc) - so you can by default set, say, export RECORDS_MOVER_LOG_LEVEL=WARNING to quiet down loging, or export RECORDS_MOVER_LOG_LEVEL=DEBUG to increase it.
level (int) – Logging more detailed than this will not be output to the stream.
stream (IO[str]) – Stream which logging should be sent (e.g., sys.stdout, sys.stdin, or perhaps a file you open)
fmt (str) – Logging format to send to Python’slogging.Formatter() - determines what details will be sent.
datefmt (str) – Date format to send to Python’slogging.Formatter() - determines how the current date/time will be recorded in the log.
- Return type
None
- property records: Records
Property containing a
records_mover.Records
object pre-configured with configuration using this Session. Once you have a Session object constructed, this is your jumping off point to moving records.
- class records_mover.Records(db_driver=PleaseInfer.token, url_resolver=PleaseInfer.token, session=PleaseInfer.token)
Bases:
object
To move records from one place to another, you can use the methods on this object.
This object should be pulled from the ‘records’ property on a
records_mover.Session
object instead of being constructed directly.To move data, you can call the
records_mover.records.move()
method, which is aliased for your convenience on this object.Example:
records = session.records db_engine = session.get_default_db_engine() url = 's3://some-bucket/some-directory/' source = records.sources.directory_from_url(url=url) target = records.targets.table(schema_name='myschema', table_name='mytable', db_engine=db_engine) results = records.move(source, target)
- Parameters
db_driver (Union[Callable[[Union[Engine, Connection]], DBDriver], PleaseInfer]) –
url_resolver (Union[UrlResolver, PleaseInfer]) –
session (Union[Session, PleaseInfer]) –
- move: Callable
Alias of
records_mover.records.move()
- sources: RecordsSources
Object containing factory methods to create various sources from which to copy records, of type
records_mover.records.sources.RecordsSources
- targets: RecordsTargets
Object containing factory methods to create various targets to which records can be copied, of type
records_mover.records.targets.RecordsTargets
- records_mover.set_stream_logging(name='records_mover', level=20, stream=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, fmt='%(asctime)s - %(message)s', datefmt='%H:%M:%S')
records-mover logs details about its operations using Python logging. This method is a simple way to configure that logging to be output to a stream (by default, stdout).
You can use it for other things (e.g., dependencies of records-mover) by adjusting the ‘name’ argument.
- Parameters
name (str) – Name of the package to set logging under. If set to ‘foo’, you can set a log variable FOO_LOG_LEVEL to the log level threshold you’d like to set (INFO/WARNING/etc) - so you can by default set, say, export RECORDS_MOVER_LOG_LEVEL=WARNING to quiet down loging, or export RECORDS_MOVER_LOG_LEVEL=DEBUG to increase it.
level (int) – Logging more detailed than this will not be output to the stream.
stream (IO[str]) – Stream which logging should be sent (e.g., sys.stdout, sys.stdin, or perhaps a file you open)
fmt (str) – Logging format to send to Python’slogging.Formatter() - determines what details will be sent.
datefmt (str) – Date format to send to Python’slogging.Formatter() - determines how the current date/time will be recorded in the log.
- Return type
None
- records_mover.move(records_source, records_target, processing_instructions=<records_mover.records.processing_instructions.ProcessingInstructions object>)
Copy records from one location to another. Applies a sequence of possible techniques to do this in an efficient way and respects the preferences set in records_source, records_target and processing_instructions.
Example use:
records = session.records db_engine = session.get_default_db_engine() url = 's3://some-bucket/some-directory/' source = records.sources.directory_from_url(url=url) target = records.targets.table(schema_name='myschema', table_name='mytable', db_engine=db_engine) results = records.move(source, target)
- Parameters
records_source (RecordsSource) – object returned by a factory method in
records_mover.records.sources.RecordsSources
which represents the place we’re copying records from.records_target (RecordsTarget) – object returned by a factory method in
records_mover.records.targets.RecordsTargets
which represents the place we’re copying records to.processing_instructions (records_mover.records.ProcessingInstructions) – Directives on how to handle different situations when processing files.
- Return type
mvrec CLI options
Move tabular data (“records”) from one place to another
usage: mvrec [-h] [--no_fail_if_dont_understand]
[--no_fail_if_cant_handle_hint] [--no_fail_if_row_invalid]
[--max_inference_rows MAX_INFERENCE_ROWS]
[--max_failure_rows MAX_FAILURE_ROWS] [-V]
{table2gsheet,table2table,table2recordsdir,table2url,table2file,table2spectrum,gsheet2gsheet,gsheet2table,gsheet2recordsdir,gsheet2url,gsheet2file,gsheet2spectrum,recordsdir2gsheet,recordsdir2table,recordsdir2recordsdir,recordsdir2url,recordsdir2file,recordsdir2spectrum,url2gsheet,url2table,url2recordsdir,url2url,url2file,url2spectrum,file2gsheet,file2table,file2recordsdir,file2url,file2file,file2spectrum}
...
Named Arguments
- --no_fail_if_dont_understand
If True, and a part of the RecordsFormat is not understood while processing, then immediately fail and raise an exception. Otherwise, ignore the misunderstood instruction (e.g., ignore the hint, assume default variant, etc etc)
Default: True
- --no_fail_if_cant_handle_hint
If True, and for whatever reason (e.g., limited options in whatever library/tool/database is being used) a certain hint can’t be handled as specified, raise an exception. Otherwise, ignore the hint and use implementation-specific different behavior.
Default: True
- --no_fail_if_row_invalid
If True, and a particular row of data in the records file cannot be understood by the library, raise an exception. Otherwise, ignore the row and continue and try to load other rows.
Default: True
- --max_inference_rows
If the schema is not provided and we need it (e.g., we’re to load the records into a database and there’s no existing table), we’ll figure it out through ‘type inference’ - looking at a bunch of examples of data and building a specific schema that can load those rows. This can take some time, so this parameter controls the maximum number of rows we’ll look at. Higher values will be more likely to result in a schema that can be loaded into, but will take longer to load. If set to None, the entire file will be processed.
Default: 1000000
- --max_failure_rows
Sets a tolerance level for number of rows of data in the records file that cannot be understood by the library that should be ignored. After reaching level, raise an exception.
- -V, --version
show program’s version number and exit
Sub-commands
table2gsheet
Copy from table to gsheet
mvrec table2gsheet [-h]
source.db_name source.schema_name source.table_name
target.spreadsheet_id target.sheet_name
target.gcp_creds_name
Positional Arguments
- source.db_name
SQLAlchemy database engine to pull data from.
- source.schema_name
Schema name of a table to get data from.
- source.table_name
Table name of a table to get data from.
- target.spreadsheet_id
This is the xyz in https://docs.google.com/spreadsheets/d/xyz/edit?ts=5be5b383#gid=abc
- target.sheet_name
This is the label of the particular tab within the Google Sheets spreadsheet where the data should go.
- target.gcp_creds_name
Credentials object for Google Cloud Platform access.
table2table
Copy from table to table
mvrec table2table [-h]
[--target.existing_table {delete_and_overwrite,truncate_and_overwrite,drop_and_recreate,append}]
[--target.drop_and_recreate_on_load_error]
source.db_name source.schema_name source.table_name
target.db_name target.schema_name target.table_name
Positional Arguments
- source.db_name
SQLAlchemy database engine to pull data from.
- source.schema_name
Schema name of a table to get data from.
- source.table_name
Table name of a table to get data from.
- target.db_name
SQLAlchemy database engine to write data to.
- target.schema_name
Schema name of a table to write data to.
- target.table_name
Table name of a table to write data to.
Named Arguments
- --target.existing_table
Possible choices: delete_and_overwrite, truncate_and_overwrite, drop_and_recreate, append
When loading into a database table, controls how any existing table found will be handled. This must be a
records_mover.records.ExistingTableHandling
object.Default: “delete_and_overwrite”
- --target.drop_and_recreate_on_load_error
If True, table load errors will attempt to be addressed by dropping the target table and reloading the incoming data.
Default: False
table2recordsdir
Copy from table to recordsdir
mvrec table2recordsdir [-h] [--target.variant TARGET.VARIANT]
[--target.format {avro,delimited,parquet}]
[--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
[--target.datetimeformat TARGET.DATETIMEFORMAT]
[--target.no_compression]
[--target.compression {GZIP,BZIP,LZO}]
[--target.no_quoting]
[--target.quoting {all,minimal,nonnumeric}]
[--target.no_escape] [--target.escape {\}]
[--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
[--target.dateformat TARGET.DATEFORMAT]
[--target.timeonlyformat TARGET.TIMEONLYFORMAT]
[--target.no_doublequote] [--target.doublequote]
[--target.no_header_row] [--target.header_row]
[--target.quotechar TARGET.QUOTECHAR]
[--target.record_terminator TARGET.RECORD-TERMINATOR]
[--target.field_delimiter TARGET.FIELD-DELIMITER]
source.db_name source.schema_name source.table_name
target.output_url
Positional Arguments
- source.db_name
SQLAlchemy database engine to pull data from.
- source.schema_name
Schema name of a table to get data from.
- source.table_name
Table name of a table to get data from.
- target.output_url
Location to write the records directory. Must be a URL format understood by the records_mover.url library, and must be a directory URL that ends with a ‘/’.
Named Arguments
- --target.variant
Records format variant - valid for ‘delimited’ records format type
- --target.format
Possible choices: avro, delimited, parquet
Records format type. Note that ‘delimited’ includes CSV/TSV/etc.
- --target.datetimeformattz
Format used to write ‘datetimetz’ values
- --target.datetimeformat
Format used to write ‘datetime’ values
- --target.no_compression
- --target.compression
Possible choices: GZIP, BZIP, LZO
Compression type of the file.
- --target.no_quoting
- --target.quoting
Possible choices: all, minimal, nonnumeric
How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.
- --target.no_escape
- --target.escape
Possible choices:
Character used to escape strings
- --target.encoding
Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252
Text encoding of file
- --target.dateformat
Format used to write ‘date’ values
- --target.timeonlyformat
Format used to write ‘time’ values
- --target.no_doublequote
- --target.doublequote
Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.
- --target.no_header_row
- --target.header_row
True if a header row is provided in the delimited files.
- --target.quotechar
A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
- --target.record_terminator
String used to close out individual rows of data.
- --target.field_delimiter
Character used between fields.
table2url
Copy from table to url
mvrec table2url [-h] [--target.variant TARGET.VARIANT]
[--target.format {avro,delimited,parquet}]
[--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
[--target.datetimeformat TARGET.DATETIMEFORMAT]
[--target.no_compression]
[--target.compression {GZIP,BZIP,LZO}] [--target.no_quoting]
[--target.quoting {all,minimal,nonnumeric}]
[--target.no_escape] [--target.escape {\}]
[--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
[--target.dateformat TARGET.DATEFORMAT]
[--target.timeonlyformat TARGET.TIMEONLYFORMAT]
[--target.no_doublequote] [--target.doublequote]
[--target.no_header_row] [--target.header_row]
[--target.quotechar TARGET.QUOTECHAR]
[--target.record_terminator TARGET.RECORD-TERMINATOR]
[--target.field_delimiter TARGET.FIELD-DELIMITER]
source.db_name source.schema_name source.table_name
target.output_url
Positional Arguments
- source.db_name
SQLAlchemy database engine to pull data from.
- source.schema_name
Schema name of a table to get data from.
- source.table_name
Table name of a table to get data from.
- target.output_url
Location of the data file to write. Must be a URL format understood by the records_mover.url library corresponding to a file, not a directory (i.e., not ending with a ‘/’)
Named Arguments
- --target.variant
Records format variant - valid for ‘delimited’ records format type
- --target.format
Possible choices: avro, delimited, parquet
Records format type. Note that ‘delimited’ includes CSV/TSV/etc.
- --target.datetimeformattz
Format used to write ‘datetimetz’ values
- --target.datetimeformat
Format used to write ‘datetime’ values
- --target.no_compression
- --target.compression
Possible choices: GZIP, BZIP, LZO
Compression type of the file.
- --target.no_quoting
- --target.quoting
Possible choices: all, minimal, nonnumeric
How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.
- --target.no_escape
- --target.escape
Possible choices:
Character used to escape strings
- --target.encoding
Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252
Text encoding of file
- --target.dateformat
Format used to write ‘date’ values
- --target.timeonlyformat
Format used to write ‘time’ values
- --target.no_doublequote
- --target.doublequote
Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.
- --target.no_header_row
- --target.header_row
True if a header row is provided in the delimited files.
- --target.quotechar
A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
- --target.record_terminator
String used to close out individual rows of data.
- --target.field_delimiter
Character used between fields.
table2file
Copy from table to file
mvrec table2file [-h] [--target.variant TARGET.VARIANT]
[--target.format {avro,delimited,parquet}]
[--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
[--target.datetimeformat TARGET.DATETIMEFORMAT]
[--target.no_compression]
[--target.compression {GZIP,BZIP,LZO}] [--target.no_quoting]
[--target.quoting {all,minimal,nonnumeric}]
[--target.no_escape] [--target.escape {\}]
[--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
[--target.dateformat TARGET.DATEFORMAT]
[--target.timeonlyformat TARGET.TIMEONLYFORMAT]
[--target.no_doublequote] [--target.doublequote]
[--target.no_header_row] [--target.header_row]
[--target.quotechar TARGET.QUOTECHAR]
[--target.record_terminator TARGET.RECORD-TERMINATOR]
[--target.field_delimiter TARGET.FIELD-DELIMITER]
source.db_name source.schema_name source.table_name
target.filename
Positional Arguments
- source.db_name
SQLAlchemy database engine to pull data from.
- source.schema_name
Schema name of a table to get data from.
- source.table_name
Table name of a table to get data from.
- target.filename
File path (relative or absolute) of the data file to unload to.
Named Arguments
- --target.variant
Records format variant - valid for ‘delimited’ records format type
- --target.format
Possible choices: avro, delimited, parquet
Records format type. Note that ‘delimited’ includes CSV/TSV/etc.
- --target.datetimeformattz
Format used to write ‘datetimetz’ values
- --target.datetimeformat
Format used to write ‘datetime’ values
- --target.no_compression
- --target.compression
Possible choices: GZIP, BZIP, LZO
Compression type of the file.
- --target.no_quoting
- --target.quoting
Possible choices: all, minimal, nonnumeric
How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.
- --target.no_escape
- --target.escape
Possible choices:
Character used to escape strings
- --target.encoding
Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252
Text encoding of file
- --target.dateformat
Format used to write ‘date’ values
- --target.timeonlyformat
Format used to write ‘time’ values
- --target.no_doublequote
- --target.doublequote
Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.
- --target.no_header_row
- --target.header_row
True if a header row is provided in the delimited files.
- --target.quotechar
A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
- --target.record_terminator
String used to close out individual rows of data.
- --target.field_delimiter
Character used between fields.
table2spectrum
Copy from table to spectrum
mvrec table2spectrum [-h]
[--target.spectrum_base_url TARGET.SPECTRUM_BASE_URL]
[--target.spectrum_rdir_url TARGET.SPECTRUM_RDIR_URL]
[--target.existing_table {delete_and_overwrite,truncate_and_overwrite,drop_and_recreate,append}]
source.db_name source.schema_name source.table_name
target.schema_name target.table_name target.db_name
Positional Arguments
- source.db_name
SQLAlchemy database engine to pull data from.
- source.schema_name
Schema name of a table to get data from.
- source.table_name
Table name of a table to get data from.
- target.schema_name
Schema name of a table to write data to.
- target.table_name
Table name of a table to write data to.
- target.db_name
SQLAlchemy database engine to write data to.
Named Arguments
- --target.spectrum_base_url
Root S3 URL under which a simple directory structure will be created for files to be stored, if spectrum_rdir_url is not specified. Note that when using the mover CLI, db-facts may be used to provide a default.
- --target.spectrum_rdir_url
S3 URL where a records directory with files will be stored; otherwise, use db-facts default if exists. If this is not specified, spectrum_base_url must be.
- --target.existing_table
Possible choices: delete_and_overwrite, truncate_and_overwrite, drop_and_recreate, append
When loading into a database table, controls how any existing table found will be handled. This must be a
records_mover.records.ExistingTableHandling
object.Default: “delete_and_overwrite”
gsheet2gsheet
Copy from gsheet to gsheet
mvrec gsheet2gsheet [-h]
[--source.out_of_band_column_headers [SOURCE.OUT_OF_BAND_COLUMN_HEADERS ...]]
source.spreadsheet_id source.sheet_name_or_range
source.gcp_creds_name target.spreadsheet_id
target.sheet_name target.gcp_creds_name
Positional Arguments
- source.spreadsheet_id
This is the xyz in https://docs.google.com/spreadsheets/d/xyz/edit?ts=5be5b383#gid=abc
- source.sheet_name_or_range
This is the label of the particular tab within the Google Sheets spreadsheet where the data should go, or a valid Google Sheets-style range formula
- source.gcp_creds_name
This is an object representing Google Cloud Platform access credentials.
- target.spreadsheet_id
This is the xyz in https://docs.google.com/spreadsheets/d/xyz/edit?ts=5be5b383#gid=abc
- target.sheet_name
This is the label of the particular tab within the Google Sheets spreadsheet where the data should go.
- target.gcp_creds_name
Credentials object for Google Cloud Platform access.
Named Arguments
- --source.out_of_band_column_headers
If provided, we’ll use these column names instead of the first row of the spreadsheet. If set, the first row will be treated as data.
gsheet2table
Copy from gsheet to table
mvrec gsheet2table [-h]
[--source.out_of_band_column_headers [SOURCE.OUT_OF_BAND_COLUMN_HEADERS ...]]
[--target.existing_table {delete_and_overwrite,truncate_and_overwrite,drop_and_recreate,append}]
[--target.drop_and_recreate_on_load_error]
source.spreadsheet_id source.sheet_name_or_range
source.gcp_creds_name target.db_name target.schema_name
target.table_name
Positional Arguments
- source.spreadsheet_id
This is the xyz in https://docs.google.com/spreadsheets/d/xyz/edit?ts=5be5b383#gid=abc
- source.sheet_name_or_range
This is the label of the particular tab within the Google Sheets spreadsheet where the data should go, or a valid Google Sheets-style range formula
- source.gcp_creds_name
This is an object representing Google Cloud Platform access credentials.
- target.db_name
SQLAlchemy database engine to write data to.
- target.schema_name
Schema name of a table to write data to.
- target.table_name
Table name of a table to write data to.
Named Arguments
- --source.out_of_band_column_headers
If provided, we’ll use these column names instead of the first row of the spreadsheet. If set, the first row will be treated as data.
- --target.existing_table
Possible choices: delete_and_overwrite, truncate_and_overwrite, drop_and_recreate, append
When loading into a database table, controls how any existing table found will be handled. This must be a
records_mover.records.ExistingTableHandling
object.Default: “delete_and_overwrite”
- --target.drop_and_recreate_on_load_error
If True, table load errors will attempt to be addressed by dropping the target table and reloading the incoming data.
Default: False
gsheet2recordsdir
Copy from gsheet to recordsdir
mvrec gsheet2recordsdir [-h]
[--source.out_of_band_column_headers [SOURCE.OUT_OF_BAND_COLUMN_HEADERS ...]]
[--target.variant TARGET.VARIANT]
[--target.format {avro,delimited,parquet}]
[--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
[--target.datetimeformat TARGET.DATETIMEFORMAT]
[--target.no_compression]
[--target.compression {GZIP,BZIP,LZO}]
[--target.no_quoting]
[--target.quoting {all,minimal,nonnumeric}]
[--target.no_escape] [--target.escape {\}]
[--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
[--target.dateformat TARGET.DATEFORMAT]
[--target.timeonlyformat TARGET.TIMEONLYFORMAT]
[--target.no_doublequote] [--target.doublequote]
[--target.no_header_row] [--target.header_row]
[--target.quotechar TARGET.QUOTECHAR]
[--target.record_terminator TARGET.RECORD-TERMINATOR]
[--target.field_delimiter TARGET.FIELD-DELIMITER]
source.spreadsheet_id source.sheet_name_or_range
source.gcp_creds_name target.output_url
Positional Arguments
- source.spreadsheet_id
This is the xyz in https://docs.google.com/spreadsheets/d/xyz/edit?ts=5be5b383#gid=abc
- source.sheet_name_or_range
This is the label of the particular tab within the Google Sheets spreadsheet where the data should go, or a valid Google Sheets-style range formula
- source.gcp_creds_name
This is an object representing Google Cloud Platform access credentials.
- target.output_url
Location to write the records directory. Must be a URL format understood by the records_mover.url library, and must be a directory URL that ends with a ‘/’.
Named Arguments
- --source.out_of_band_column_headers
If provided, we’ll use these column names instead of the first row of the spreadsheet. If set, the first row will be treated as data.
- --target.variant
Records format variant - valid for ‘delimited’ records format type
- --target.format
Possible choices: avro, delimited, parquet
Records format type. Note that ‘delimited’ includes CSV/TSV/etc.
- --target.datetimeformattz
Format used to write ‘datetimetz’ values
- --target.datetimeformat
Format used to write ‘datetime’ values
- --target.no_compression
- --target.compression
Possible choices: GZIP, BZIP, LZO
Compression type of the file.
- --target.no_quoting
- --target.quoting
Possible choices: all, minimal, nonnumeric
How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.
- --target.no_escape
- --target.escape
Possible choices:
Character used to escape strings
- --target.encoding
Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252
Text encoding of file
- --target.dateformat
Format used to write ‘date’ values
- --target.timeonlyformat
Format used to write ‘time’ values
- --target.no_doublequote
- --target.doublequote
Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.
- --target.no_header_row
- --target.header_row
True if a header row is provided in the delimited files.
- --target.quotechar
A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
- --target.record_terminator
String used to close out individual rows of data.
- --target.field_delimiter
Character used between fields.
gsheet2url
Copy from gsheet to url
mvrec gsheet2url [-h]
[--source.out_of_band_column_headers [SOURCE.OUT_OF_BAND_COLUMN_HEADERS ...]]
[--target.variant TARGET.VARIANT]
[--target.format {avro,delimited,parquet}]
[--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
[--target.datetimeformat TARGET.DATETIMEFORMAT]
[--target.no_compression]
[--target.compression {GZIP,BZIP,LZO}] [--target.no_quoting]
[--target.quoting {all,minimal,nonnumeric}]
[--target.no_escape] [--target.escape {\}]
[--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
[--target.dateformat TARGET.DATEFORMAT]
[--target.timeonlyformat TARGET.TIMEONLYFORMAT]
[--target.no_doublequote] [--target.doublequote]
[--target.no_header_row] [--target.header_row]
[--target.quotechar TARGET.QUOTECHAR]
[--target.record_terminator TARGET.RECORD-TERMINATOR]
[--target.field_delimiter TARGET.FIELD-DELIMITER]
source.spreadsheet_id source.sheet_name_or_range
source.gcp_creds_name target.output_url
Positional Arguments
- source.spreadsheet_id
This is the xyz in https://docs.google.com/spreadsheets/d/xyz/edit?ts=5be5b383#gid=abc
- source.sheet_name_or_range
This is the label of the particular tab within the Google Sheets spreadsheet where the data should go, or a valid Google Sheets-style range formula
- source.gcp_creds_name
This is an object representing Google Cloud Platform access credentials.
- target.output_url
Location of the data file to write. Must be a URL format understood by the records_mover.url library corresponding to a file, not a directory (i.e., not ending with a ‘/’)
Named Arguments
- --source.out_of_band_column_headers
If provided, we’ll use these column names instead of the first row of the spreadsheet. If set, the first row will be treated as data.
- --target.variant
Records format variant - valid for ‘delimited’ records format type
- --target.format
Possible choices: avro, delimited, parquet
Records format type. Note that ‘delimited’ includes CSV/TSV/etc.
- --target.datetimeformattz
Format used to write ‘datetimetz’ values
- --target.datetimeformat
Format used to write ‘datetime’ values
- --target.no_compression
- --target.compression
Possible choices: GZIP, BZIP, LZO
Compression type of the file.
- --target.no_quoting
- --target.quoting
Possible choices: all, minimal, nonnumeric
How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.
- --target.no_escape
- --target.escape
Possible choices:
Character used to escape strings
- --target.encoding
Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252
Text encoding of file
- --target.dateformat
Format used to write ‘date’ values
- --target.timeonlyformat
Format used to write ‘time’ values
- --target.no_doublequote
- --target.doublequote
Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.
- --target.no_header_row
- --target.header_row
True if a header row is provided in the delimited files.
- --target.quotechar
A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
- --target.record_terminator
String used to close out individual rows of data.
- --target.field_delimiter
Character used between fields.
gsheet2file
Copy from gsheet to file
mvrec gsheet2file [-h]
[--source.out_of_band_column_headers [SOURCE.OUT_OF_BAND_COLUMN_HEADERS ...]]
[--target.variant TARGET.VARIANT]
[--target.format {avro,delimited,parquet}]
[--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
[--target.datetimeformat TARGET.DATETIMEFORMAT]
[--target.no_compression]
[--target.compression {GZIP,BZIP,LZO}] [--target.no_quoting]
[--target.quoting {all,minimal,nonnumeric}]
[--target.no_escape] [--target.escape {\}]
[--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
[--target.dateformat TARGET.DATEFORMAT]
[--target.timeonlyformat TARGET.TIMEONLYFORMAT]
[--target.no_doublequote] [--target.doublequote]
[--target.no_header_row] [--target.header_row]
[--target.quotechar TARGET.QUOTECHAR]
[--target.record_terminator TARGET.RECORD-TERMINATOR]
[--target.field_delimiter TARGET.FIELD-DELIMITER]
source.spreadsheet_id source.sheet_name_or_range
source.gcp_creds_name target.filename
Positional Arguments
- source.spreadsheet_id
This is the xyz in https://docs.google.com/spreadsheets/d/xyz/edit?ts=5be5b383#gid=abc
- source.sheet_name_or_range
This is the label of the particular tab within the Google Sheets spreadsheet where the data should go, or a valid Google Sheets-style range formula
- source.gcp_creds_name
This is an object representing Google Cloud Platform access credentials.
- target.filename
File path (relative or absolute) of the data file to unload to.
Named Arguments
- --source.out_of_band_column_headers
If provided, we’ll use these column names instead of the first row of the spreadsheet. If set, the first row will be treated as data.
- --target.variant
Records format variant - valid for ‘delimited’ records format type
- --target.format
Possible choices: avro, delimited, parquet
Records format type. Note that ‘delimited’ includes CSV/TSV/etc.
- --target.datetimeformattz
Format used to write ‘datetimetz’ values
- --target.datetimeformat
Format used to write ‘datetime’ values
- --target.no_compression
- --target.compression
Possible choices: GZIP, BZIP, LZO
Compression type of the file.
- --target.no_quoting
- --target.quoting
Possible choices: all, minimal, nonnumeric
How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.
- --target.no_escape
- --target.escape
Possible choices:
Character used to escape strings
- --target.encoding
Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252
Text encoding of file
- --target.dateformat
Format used to write ‘date’ values
- --target.timeonlyformat
Format used to write ‘time’ values
- --target.no_doublequote
- --target.doublequote
Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.
- --target.no_header_row
- --target.header_row
True if a header row is provided in the delimited files.
- --target.quotechar
A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
- --target.record_terminator
String used to close out individual rows of data.
- --target.field_delimiter
Character used between fields.
gsheet2spectrum
Copy from gsheet to spectrum
mvrec gsheet2spectrum [-h]
[--source.out_of_band_column_headers [SOURCE.OUT_OF_BAND_COLUMN_HEADERS ...]]
[--target.spectrum_base_url TARGET.SPECTRUM_BASE_URL]
[--target.spectrum_rdir_url TARGET.SPECTRUM_RDIR_URL]
[--target.existing_table {delete_and_overwrite,truncate_and_overwrite,drop_and_recreate,append}]
source.spreadsheet_id source.sheet_name_or_range
source.gcp_creds_name target.schema_name
target.table_name target.db_name
Positional Arguments
- source.spreadsheet_id
This is the xyz in https://docs.google.com/spreadsheets/d/xyz/edit?ts=5be5b383#gid=abc
- source.sheet_name_or_range
This is the label of the particular tab within the Google Sheets spreadsheet where the data should go, or a valid Google Sheets-style range formula
- source.gcp_creds_name
This is an object representing Google Cloud Platform access credentials.
- target.schema_name
Schema name of a table to write data to.
- target.table_name
Table name of a table to write data to.
- target.db_name
SQLAlchemy database engine to write data to.
Named Arguments
- --source.out_of_band_column_headers
If provided, we’ll use these column names instead of the first row of the spreadsheet. If set, the first row will be treated as data.
- --target.spectrum_base_url
Root S3 URL under which a simple directory structure will be created for files to be stored, if spectrum_rdir_url is not specified. Note that when using the mover CLI, db-facts may be used to provide a default.
- --target.spectrum_rdir_url
S3 URL where a records directory with files will be stored; otherwise, use db-facts default if exists. If this is not specified, spectrum_base_url must be.
- --target.existing_table
Possible choices: delete_and_overwrite, truncate_and_overwrite, drop_and_recreate, append
When loading into a database table, controls how any existing table found will be handled. This must be a
records_mover.records.ExistingTableHandling
object.Default: “delete_and_overwrite”
recordsdir2gsheet
Copy from recordsdir to gsheet
mvrec recordsdir2gsheet [-h] [--no_source.fail_if_dont_understand]
source.url target.spreadsheet_id target.sheet_name
target.gcp_creds_name
Positional Arguments
- source.url
Location of the records directory. Must be a URL format understood by the records_mover.url library, and must be a directory URL that ends with a ‘/’.
- target.spreadsheet_id
This is the xyz in https://docs.google.com/spreadsheets/d/xyz/edit?ts=5be5b383#gid=abc
- target.sheet_name
This is the label of the particular tab within the Google Sheets spreadsheet where the data should go.
- target.gcp_creds_name
Credentials object for Google Cloud Platform access.
Named Arguments
- --no_source.fail_if_dont_understand
If True, and a part of the RecordsFormat is not understood while processing, then immediately fail and raise an exception. Otherwise, ignore the misunderstood instruction (e.g., ignore the hint, assume default variant, etc etc)
Default: True
recordsdir2table
Copy from recordsdir to table
mvrec recordsdir2table [-h] [--no_source.fail_if_dont_understand]
[--target.existing_table {delete_and_overwrite,truncate_and_overwrite,drop_and_recreate,append}]
[--target.drop_and_recreate_on_load_error]
source.url target.db_name target.schema_name
target.table_name
Positional Arguments
- source.url
Location of the records directory. Must be a URL format understood by the records_mover.url library, and must be a directory URL that ends with a ‘/’.
- target.db_name
SQLAlchemy database engine to write data to.
- target.schema_name
Schema name of a table to write data to.
- target.table_name
Table name of a table to write data to.
Named Arguments
- --no_source.fail_if_dont_understand
If True, and a part of the RecordsFormat is not understood while processing, then immediately fail and raise an exception. Otherwise, ignore the misunderstood instruction (e.g., ignore the hint, assume default variant, etc etc)
Default: True
- --target.existing_table
Possible choices: delete_and_overwrite, truncate_and_overwrite, drop_and_recreate, append
When loading into a database table, controls how any existing table found will be handled. This must be a
records_mover.records.ExistingTableHandling
object.Default: “delete_and_overwrite”
- --target.drop_and_recreate_on_load_error
If True, table load errors will attempt to be addressed by dropping the target table and reloading the incoming data.
Default: False
recordsdir2recordsdir
Copy from recordsdir to recordsdir
mvrec recordsdir2recordsdir [-h] [--no_source.fail_if_dont_understand]
[--target.variant TARGET.VARIANT]
[--target.format {avro,delimited,parquet}]
[--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
[--target.datetimeformat TARGET.DATETIMEFORMAT]
[--target.no_compression]
[--target.compression {GZIP,BZIP,LZO}]
[--target.no_quoting]
[--target.quoting {all,minimal,nonnumeric}]
[--target.no_escape] [--target.escape {\}]
[--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
[--target.dateformat TARGET.DATEFORMAT]
[--target.timeonlyformat TARGET.TIMEONLYFORMAT]
[--target.no_doublequote] [--target.doublequote]
[--target.no_header_row] [--target.header_row]
[--target.quotechar TARGET.QUOTECHAR]
[--target.record_terminator TARGET.RECORD-TERMINATOR]
[--target.field_delimiter TARGET.FIELD-DELIMITER]
source.url target.output_url
Positional Arguments
- source.url
Location of the records directory. Must be a URL format understood by the records_mover.url library, and must be a directory URL that ends with a ‘/’.
- target.output_url
Location to write the records directory. Must be a URL format understood by the records_mover.url library, and must be a directory URL that ends with a ‘/’.
Named Arguments
- --no_source.fail_if_dont_understand
If True, and a part of the RecordsFormat is not understood while processing, then immediately fail and raise an exception. Otherwise, ignore the misunderstood instruction (e.g., ignore the hint, assume default variant, etc etc)
Default: True
- --target.variant
Records format variant - valid for ‘delimited’ records format type
- --target.format
Possible choices: avro, delimited, parquet
Records format type. Note that ‘delimited’ includes CSV/TSV/etc.
- --target.datetimeformattz
Format used to write ‘datetimetz’ values
- --target.datetimeformat
Format used to write ‘datetime’ values
- --target.no_compression
- --target.compression
Possible choices: GZIP, BZIP, LZO
Compression type of the file.
- --target.no_quoting
- --target.quoting
Possible choices: all, minimal, nonnumeric
How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.
- --target.no_escape
- --target.escape
Possible choices:
Character used to escape strings
- --target.encoding
Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252
Text encoding of file
- --target.dateformat
Format used to write ‘date’ values
- --target.timeonlyformat
Format used to write ‘time’ values
- --target.no_doublequote
- --target.doublequote
Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.
- --target.no_header_row
- --target.header_row
True if a header row is provided in the delimited files.
- --target.quotechar
A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
- --target.record_terminator
String used to close out individual rows of data.
- --target.field_delimiter
Character used between fields.
recordsdir2url
Copy from recordsdir to url
mvrec recordsdir2url [-h] [--no_source.fail_if_dont_understand]
[--target.variant TARGET.VARIANT]
[--target.format {avro,delimited,parquet}]
[--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
[--target.datetimeformat TARGET.DATETIMEFORMAT]
[--target.no_compression]
[--target.compression {GZIP,BZIP,LZO}]
[--target.no_quoting]
[--target.quoting {all,minimal,nonnumeric}]
[--target.no_escape] [--target.escape {\}]
[--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
[--target.dateformat TARGET.DATEFORMAT]
[--target.timeonlyformat TARGET.TIMEONLYFORMAT]
[--target.no_doublequote] [--target.doublequote]
[--target.no_header_row] [--target.header_row]
[--target.quotechar TARGET.QUOTECHAR]
[--target.record_terminator TARGET.RECORD-TERMINATOR]
[--target.field_delimiter TARGET.FIELD-DELIMITER]
source.url target.output_url
Positional Arguments
- source.url
Location of the records directory. Must be a URL format understood by the records_mover.url library, and must be a directory URL that ends with a ‘/’.
- target.output_url
Location of the data file to write. Must be a URL format understood by the records_mover.url library corresponding to a file, not a directory (i.e., not ending with a ‘/’)
Named Arguments
- --no_source.fail_if_dont_understand
If True, and a part of the RecordsFormat is not understood while processing, then immediately fail and raise an exception. Otherwise, ignore the misunderstood instruction (e.g., ignore the hint, assume default variant, etc etc)
Default: True
- --target.variant
Records format variant - valid for ‘delimited’ records format type
- --target.format
Possible choices: avro, delimited, parquet
Records format type. Note that ‘delimited’ includes CSV/TSV/etc.
- --target.datetimeformattz
Format used to write ‘datetimetz’ values
- --target.datetimeformat
Format used to write ‘datetime’ values
- --target.no_compression
- --target.compression
Possible choices: GZIP, BZIP, LZO
Compression type of the file.
- --target.no_quoting
- --target.quoting
Possible choices: all, minimal, nonnumeric
How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.
- --target.no_escape
- --target.escape
Possible choices:
Character used to escape strings
- --target.encoding
Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252
Text encoding of file
- --target.dateformat
Format used to write ‘date’ values
- --target.timeonlyformat
Format used to write ‘time’ values
- --target.no_doublequote
- --target.doublequote
Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.
- --target.no_header_row
- --target.header_row
True if a header row is provided in the delimited files.
- --target.quotechar
A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
- --target.record_terminator
String used to close out individual rows of data.
- --target.field_delimiter
Character used between fields.
recordsdir2file
Copy from recordsdir to file
mvrec recordsdir2file [-h] [--no_source.fail_if_dont_understand]
[--target.variant TARGET.VARIANT]
[--target.format {avro,delimited,parquet}]
[--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
[--target.datetimeformat TARGET.DATETIMEFORMAT]
[--target.no_compression]
[--target.compression {GZIP,BZIP,LZO}]
[--target.no_quoting]
[--target.quoting {all,minimal,nonnumeric}]
[--target.no_escape] [--target.escape {\}]
[--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
[--target.dateformat TARGET.DATEFORMAT]
[--target.timeonlyformat TARGET.TIMEONLYFORMAT]
[--target.no_doublequote] [--target.doublequote]
[--target.no_header_row] [--target.header_row]
[--target.quotechar TARGET.QUOTECHAR]
[--target.record_terminator TARGET.RECORD-TERMINATOR]
[--target.field_delimiter TARGET.FIELD-DELIMITER]
source.url target.filename
Positional Arguments
- source.url
Location of the records directory. Must be a URL format understood by the records_mover.url library, and must be a directory URL that ends with a ‘/’.
- target.filename
File path (relative or absolute) of the data file to unload to.
Named Arguments
- --no_source.fail_if_dont_understand
If True, and a part of the RecordsFormat is not understood while processing, then immediately fail and raise an exception. Otherwise, ignore the misunderstood instruction (e.g., ignore the hint, assume default variant, etc etc)
Default: True
- --target.variant
Records format variant - valid for ‘delimited’ records format type
- --target.format
Possible choices: avro, delimited, parquet
Records format type. Note that ‘delimited’ includes CSV/TSV/etc.
- --target.datetimeformattz
Format used to write ‘datetimetz’ values
- --target.datetimeformat
Format used to write ‘datetime’ values
- --target.no_compression
- --target.compression
Possible choices: GZIP, BZIP, LZO
Compression type of the file.
- --target.no_quoting
- --target.quoting
Possible choices: all, minimal, nonnumeric
How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.
- --target.no_escape
- --target.escape
Possible choices:
Character used to escape strings
- --target.encoding
Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252
Text encoding of file
- --target.dateformat
Format used to write ‘date’ values
- --target.timeonlyformat
Format used to write ‘time’ values
- --target.no_doublequote
- --target.doublequote
Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.
- --target.no_header_row
- --target.header_row
True if a header row is provided in the delimited files.
- --target.quotechar
A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
- --target.record_terminator
String used to close out individual rows of data.
- --target.field_delimiter
Character used between fields.
recordsdir2spectrum
Copy from recordsdir to spectrum
mvrec recordsdir2spectrum [-h] [--no_source.fail_if_dont_understand]
[--target.spectrum_base_url TARGET.SPECTRUM_BASE_URL]
[--target.spectrum_rdir_url TARGET.SPECTRUM_RDIR_URL]
[--target.existing_table {delete_and_overwrite,truncate_and_overwrite,drop_and_recreate,append}]
source.url target.schema_name target.table_name
target.db_name
Positional Arguments
- source.url
Location of the records directory. Must be a URL format understood by the records_mover.url library, and must be a directory URL that ends with a ‘/’.
- target.schema_name
Schema name of a table to write data to.
- target.table_name
Table name of a table to write data to.
- target.db_name
SQLAlchemy database engine to write data to.
Named Arguments
- --no_source.fail_if_dont_understand
If True, and a part of the RecordsFormat is not understood while processing, then immediately fail and raise an exception. Otherwise, ignore the misunderstood instruction (e.g., ignore the hint, assume default variant, etc etc)
Default: True
- --target.spectrum_base_url
Root S3 URL under which a simple directory structure will be created for files to be stored, if spectrum_rdir_url is not specified. Note that when using the mover CLI, db-facts may be used to provide a default.
- --target.spectrum_rdir_url
S3 URL where a records directory with files will be stored; otherwise, use db-facts default if exists. If this is not specified, spectrum_base_url must be.
- --target.existing_table
Possible choices: delete_and_overwrite, truncate_and_overwrite, drop_and_recreate, append
When loading into a database table, controls how any existing table found will be handled. This must be a
records_mover.records.ExistingTableHandling
object.Default: “delete_and_overwrite”
url2gsheet
Copy from url to gsheet
mvrec url2gsheet [-h] [--source.variant SOURCE.VARIANT]
[--source.format {avro,delimited,parquet}]
[--source.datetimeformattz SOURCE.DATETIMEFORMATTZ]
[--source.datetimeformat SOURCE.DATETIMEFORMAT]
[--source.no_compression]
[--source.compression {GZIP,BZIP,LZO}] [--source.no_quoting]
[--source.quoting {all,minimal,nonnumeric}]
[--source.no_escape] [--source.escape {\}]
[--source.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
[--source.dateformat SOURCE.DATEFORMAT]
[--source.timeonlyformat SOURCE.TIMEONLYFORMAT]
[--source.no_doublequote] [--source.doublequote]
[--source.no_header_row] [--source.header_row]
[--source.quotechar SOURCE.QUOTECHAR]
[--source.record_terminator SOURCE.RECORD-TERMINATOR]
[--source.field_delimiter SOURCE.FIELD-DELIMITER]
source.input_url target.spreadsheet_id target.sheet_name
target.gcp_creds_name
Positional Arguments
- source.input_url
Location of the data file. Must be a URL format understood by the records_mover.url library.
- target.spreadsheet_id
This is the xyz in https://docs.google.com/spreadsheets/d/xyz/edit?ts=5be5b383#gid=abc
- target.sheet_name
This is the label of the particular tab within the Google Sheets spreadsheet where the data should go.
- target.gcp_creds_name
Credentials object for Google Cloud Platform access.
Named Arguments
- --source.variant
Records format variant - valid for ‘delimited’ records format type
- --source.format
Possible choices: avro, delimited, parquet
Records format type. Note that ‘delimited’ includes CSV/TSV/etc.
- --source.datetimeformattz
Format used to write ‘datetimetz’ values
- --source.datetimeformat
Format used to write ‘datetime’ values
- --source.no_compression
- --source.compression
Possible choices: GZIP, BZIP, LZO
Compression type of the file.
- --source.no_quoting
- --source.quoting
Possible choices: all, minimal, nonnumeric
How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.
- --source.no_escape
- --source.escape
Possible choices:
Character used to escape strings
- --source.encoding
Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252
Text encoding of file
- --source.dateformat
Format used to write ‘date’ values
- --source.timeonlyformat
Format used to write ‘time’ values
- --source.no_doublequote
- --source.doublequote
Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.
- --source.no_header_row
- --source.header_row
True if a header row is provided in the delimited files.
- --source.quotechar
A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
- --source.record_terminator
String used to close out individual rows of data.
- --source.field_delimiter
Character used between fields.
url2table
Copy from url to table
mvrec url2table [-h] [--source.variant SOURCE.VARIANT]
[--source.format {avro,delimited,parquet}]
[--source.datetimeformattz SOURCE.DATETIMEFORMATTZ]
[--source.datetimeformat SOURCE.DATETIMEFORMAT]
[--source.no_compression]
[--source.compression {GZIP,BZIP,LZO}] [--source.no_quoting]
[--source.quoting {all,minimal,nonnumeric}]
[--source.no_escape] [--source.escape {\}]
[--source.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
[--source.dateformat SOURCE.DATEFORMAT]
[--source.timeonlyformat SOURCE.TIMEONLYFORMAT]
[--source.no_doublequote] [--source.doublequote]
[--source.no_header_row] [--source.header_row]
[--source.quotechar SOURCE.QUOTECHAR]
[--source.record_terminator SOURCE.RECORD-TERMINATOR]
[--source.field_delimiter SOURCE.FIELD-DELIMITER]
[--target.existing_table {delete_and_overwrite,truncate_and_overwrite,drop_and_recreate,append}]
[--target.drop_and_recreate_on_load_error]
source.input_url target.db_name target.schema_name
target.table_name
Positional Arguments
- source.input_url
Location of the data file. Must be a URL format understood by the records_mover.url library.
- target.db_name
SQLAlchemy database engine to write data to.
- target.schema_name
Schema name of a table to write data to.
- target.table_name
Table name of a table to write data to.
Named Arguments
- --source.variant
Records format variant - valid for ‘delimited’ records format type
- --source.format
Possible choices: avro, delimited, parquet
Records format type. Note that ‘delimited’ includes CSV/TSV/etc.
- --source.datetimeformattz
Format used to write ‘datetimetz’ values
- --source.datetimeformat
Format used to write ‘datetime’ values
- --source.no_compression
- --source.compression
Possible choices: GZIP, BZIP, LZO
Compression type of the file.
- --source.no_quoting
- --source.quoting
Possible choices: all, minimal, nonnumeric
How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.
- --source.no_escape
- --source.escape
Possible choices:
Character used to escape strings
- --source.encoding
Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252
Text encoding of file
- --source.dateformat
Format used to write ‘date’ values
- --source.timeonlyformat
Format used to write ‘time’ values
- --source.no_doublequote
- --source.doublequote
Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.
- --source.no_header_row
- --source.header_row
True if a header row is provided in the delimited files.
- --source.quotechar
A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
- --source.record_terminator
String used to close out individual rows of data.
- --source.field_delimiter
Character used between fields.
- --target.existing_table
Possible choices: delete_and_overwrite, truncate_and_overwrite, drop_and_recreate, append
When loading into a database table, controls how any existing table found will be handled. This must be a
records_mover.records.ExistingTableHandling
object.Default: “delete_and_overwrite”
- --target.drop_and_recreate_on_load_error
If True, table load errors will attempt to be addressed by dropping the target table and reloading the incoming data.
Default: False
url2recordsdir
Copy from url to recordsdir
mvrec url2recordsdir [-h] [--source.variant SOURCE.VARIANT]
[--source.format {avro,delimited,parquet}]
[--source.datetimeformattz SOURCE.DATETIMEFORMATTZ]
[--source.datetimeformat SOURCE.DATETIMEFORMAT]
[--source.no_compression]
[--source.compression {GZIP,BZIP,LZO}]
[--source.no_quoting]
[--source.quoting {all,minimal,nonnumeric}]
[--source.no_escape] [--source.escape {\}]
[--source.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
[--source.dateformat SOURCE.DATEFORMAT]
[--source.timeonlyformat SOURCE.TIMEONLYFORMAT]
[--source.no_doublequote] [--source.doublequote]
[--source.no_header_row] [--source.header_row]
[--source.quotechar SOURCE.QUOTECHAR]
[--source.record_terminator SOURCE.RECORD-TERMINATOR]
[--source.field_delimiter SOURCE.FIELD-DELIMITER]
[--target.variant TARGET.VARIANT]
[--target.format {avro,delimited,parquet}]
[--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
[--target.datetimeformat TARGET.DATETIMEFORMAT]
[--target.no_compression]
[--target.compression {GZIP,BZIP,LZO}]
[--target.no_quoting]
[--target.quoting {all,minimal,nonnumeric}]
[--target.no_escape] [--target.escape {\}]
[--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
[--target.dateformat TARGET.DATEFORMAT]
[--target.timeonlyformat TARGET.TIMEONLYFORMAT]
[--target.no_doublequote] [--target.doublequote]
[--target.no_header_row] [--target.header_row]
[--target.quotechar TARGET.QUOTECHAR]
[--target.record_terminator TARGET.RECORD-TERMINATOR]
[--target.field_delimiter TARGET.FIELD-DELIMITER]
source.input_url target.output_url
Positional Arguments
- source.input_url
Location of the data file. Must be a URL format understood by the records_mover.url library.
- target.output_url
Location to write the records directory. Must be a URL format understood by the records_mover.url library, and must be a directory URL that ends with a ‘/’.
Named Arguments
- --source.variant
Records format variant - valid for ‘delimited’ records format type
- --source.format
Possible choices: avro, delimited, parquet
Records format type. Note that ‘delimited’ includes CSV/TSV/etc.
- --source.datetimeformattz
Format used to write ‘datetimetz’ values
- --source.datetimeformat
Format used to write ‘datetime’ values
- --source.no_compression
- --source.compression
Possible choices: GZIP, BZIP, LZO
Compression type of the file.
- --source.no_quoting
- --source.quoting
Possible choices: all, minimal, nonnumeric
How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.
- --source.no_escape
- --source.escape
Possible choices:
Character used to escape strings
- --source.encoding
Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252
Text encoding of file
- --source.dateformat
Format used to write ‘date’ values
- --source.timeonlyformat
Format used to write ‘time’ values
- --source.no_doublequote
- --source.doublequote
Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.
- --source.no_header_row
- --source.header_row
True if a header row is provided in the delimited files.
- --source.quotechar
A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
- --source.record_terminator
String used to close out individual rows of data.
- --source.field_delimiter
Character used between fields.
- --target.variant
Records format variant - valid for ‘delimited’ records format type
- --target.format
Possible choices: avro, delimited, parquet
Records format type. Note that ‘delimited’ includes CSV/TSV/etc.
- --target.datetimeformattz
Format used to write ‘datetimetz’ values
- --target.datetimeformat
Format used to write ‘datetime’ values
- --target.no_compression
- --target.compression
Possible choices: GZIP, BZIP, LZO
Compression type of the file.
- --target.no_quoting
- --target.quoting
Possible choices: all, minimal, nonnumeric
How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.
- --target.no_escape
- --target.escape
Possible choices:
Character used to escape strings
- --target.encoding
Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252
Text encoding of file
- --target.dateformat
Format used to write ‘date’ values
- --target.timeonlyformat
Format used to write ‘time’ values
- --target.no_doublequote
- --target.doublequote
Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.
- --target.no_header_row
- --target.header_row
True if a header row is provided in the delimited files.
- --target.quotechar
A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
- --target.record_terminator
String used to close out individual rows of data.
- --target.field_delimiter
Character used between fields.
url2url
Copy from url to url
mvrec url2url [-h] [--source.variant SOURCE.VARIANT]
[--source.format {avro,delimited,parquet}]
[--source.datetimeformattz SOURCE.DATETIMEFORMATTZ]
[--source.datetimeformat SOURCE.DATETIMEFORMAT]
[--source.no_compression] [--source.compression {GZIP,BZIP,LZO}]
[--source.no_quoting]
[--source.quoting {all,minimal,nonnumeric}] [--source.no_escape]
[--source.escape {\}]
[--source.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
[--source.dateformat SOURCE.DATEFORMAT]
[--source.timeonlyformat SOURCE.TIMEONLYFORMAT]
[--source.no_doublequote] [--source.doublequote]
[--source.no_header_row] [--source.header_row]
[--source.quotechar SOURCE.QUOTECHAR]
[--source.record_terminator SOURCE.RECORD-TERMINATOR]
[--source.field_delimiter SOURCE.FIELD-DELIMITER]
[--target.variant TARGET.VARIANT]
[--target.format {avro,delimited,parquet}]
[--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
[--target.datetimeformat TARGET.DATETIMEFORMAT]
[--target.no_compression] [--target.compression {GZIP,BZIP,LZO}]
[--target.no_quoting]
[--target.quoting {all,minimal,nonnumeric}] [--target.no_escape]
[--target.escape {\}]
[--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
[--target.dateformat TARGET.DATEFORMAT]
[--target.timeonlyformat TARGET.TIMEONLYFORMAT]
[--target.no_doublequote] [--target.doublequote]
[--target.no_header_row] [--target.header_row]
[--target.quotechar TARGET.QUOTECHAR]
[--target.record_terminator TARGET.RECORD-TERMINATOR]
[--target.field_delimiter TARGET.FIELD-DELIMITER]
source.input_url target.output_url
Positional Arguments
- source.input_url
Location of the data file. Must be a URL format understood by the records_mover.url library.
- target.output_url
Location of the data file to write. Must be a URL format understood by the records_mover.url library corresponding to a file, not a directory (i.e., not ending with a ‘/’)
Named Arguments
- --source.variant
Records format variant - valid for ‘delimited’ records format type
- --source.format
Possible choices: avro, delimited, parquet
Records format type. Note that ‘delimited’ includes CSV/TSV/etc.
- --source.datetimeformattz
Format used to write ‘datetimetz’ values
- --source.datetimeformat
Format used to write ‘datetime’ values
- --source.no_compression
- --source.compression
Possible choices: GZIP, BZIP, LZO
Compression type of the file.
- --source.no_quoting
- --source.quoting
Possible choices: all, minimal, nonnumeric
How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.
- --source.no_escape
- --source.escape
Possible choices:
Character used to escape strings
- --source.encoding
Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252
Text encoding of file
- --source.dateformat
Format used to write ‘date’ values
- --source.timeonlyformat
Format used to write ‘time’ values
- --source.no_doublequote
- --source.doublequote
Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.
- --source.no_header_row
- --source.header_row
True if a header row is provided in the delimited files.
- --source.quotechar
A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
- --source.record_terminator
String used to close out individual rows of data.
- --source.field_delimiter
Character used between fields.
- --target.variant
Records format variant - valid for ‘delimited’ records format type
- --target.format
Possible choices: avro, delimited, parquet
Records format type. Note that ‘delimited’ includes CSV/TSV/etc.
- --target.datetimeformattz
Format used to write ‘datetimetz’ values
- --target.datetimeformat
Format used to write ‘datetime’ values
- --target.no_compression
- --target.compression
Possible choices: GZIP, BZIP, LZO
Compression type of the file.
- --target.no_quoting
- --target.quoting
Possible choices: all, minimal, nonnumeric
How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.
- --target.no_escape
- --target.escape
Possible choices:
Character used to escape strings
- --target.encoding
Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252
Text encoding of file
- --target.dateformat
Format used to write ‘date’ values
- --target.timeonlyformat
Format used to write ‘time’ values
- --target.no_doublequote
- --target.doublequote
Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.
- --target.no_header_row
- --target.header_row
True if a header row is provided in the delimited files.
- --target.quotechar
A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
- --target.record_terminator
String used to close out individual rows of data.
- --target.field_delimiter
Character used between fields.
url2file
Copy from url to file
mvrec url2file [-h] [--source.variant SOURCE.VARIANT]
[--source.format {avro,delimited,parquet}]
[--source.datetimeformattz SOURCE.DATETIMEFORMATTZ]
[--source.datetimeformat SOURCE.DATETIMEFORMAT]
[--source.no_compression]
[--source.compression {GZIP,BZIP,LZO}] [--source.no_quoting]
[--source.quoting {all,minimal,nonnumeric}]
[--source.no_escape] [--source.escape {\}]
[--source.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
[--source.dateformat SOURCE.DATEFORMAT]
[--source.timeonlyformat SOURCE.TIMEONLYFORMAT]
[--source.no_doublequote] [--source.doublequote]
[--source.no_header_row] [--source.header_row]
[--source.quotechar SOURCE.QUOTECHAR]
[--source.record_terminator SOURCE.RECORD-TERMINATOR]
[--source.field_delimiter SOURCE.FIELD-DELIMITER]
[--target.variant TARGET.VARIANT]
[--target.format {avro,delimited,parquet}]
[--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
[--target.datetimeformat TARGET.DATETIMEFORMAT]
[--target.no_compression]
[--target.compression {GZIP,BZIP,LZO}] [--target.no_quoting]
[--target.quoting {all,minimal,nonnumeric}]
[--target.no_escape] [--target.escape {\}]
[--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
[--target.dateformat TARGET.DATEFORMAT]
[--target.timeonlyformat TARGET.TIMEONLYFORMAT]
[--target.no_doublequote] [--target.doublequote]
[--target.no_header_row] [--target.header_row]
[--target.quotechar TARGET.QUOTECHAR]
[--target.record_terminator TARGET.RECORD-TERMINATOR]
[--target.field_delimiter TARGET.FIELD-DELIMITER]
source.input_url target.filename
Positional Arguments
- source.input_url
Location of the data file. Must be a URL format understood by the records_mover.url library.
- target.filename
File path (relative or absolute) of the data file to unload to.
Named Arguments
- --source.variant
Records format variant - valid for ‘delimited’ records format type
- --source.format
Possible choices: avro, delimited, parquet
Records format type. Note that ‘delimited’ includes CSV/TSV/etc.
- --source.datetimeformattz
Format used to write ‘datetimetz’ values
- --source.datetimeformat
Format used to write ‘datetime’ values
- --source.no_compression
- --source.compression
Possible choices: GZIP, BZIP, LZO
Compression type of the file.
- --source.no_quoting
- --source.quoting
Possible choices: all, minimal, nonnumeric
How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.
- --source.no_escape
- --source.escape
Possible choices:
Character used to escape strings
- --source.encoding
Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252
Text encoding of file
- --source.dateformat
Format used to write ‘date’ values
- --source.timeonlyformat
Format used to write ‘time’ values
- --source.no_doublequote
- --source.doublequote
Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.
- --source.no_header_row
- --source.header_row
True if a header row is provided in the delimited files.
- --source.quotechar
A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
- --source.record_terminator
String used to close out individual rows of data.
- --source.field_delimiter
Character used between fields.
- --target.variant
Records format variant - valid for ‘delimited’ records format type
- --target.format
Possible choices: avro, delimited, parquet
Records format type. Note that ‘delimited’ includes CSV/TSV/etc.
- --target.datetimeformattz
Format used to write ‘datetimetz’ values
- --target.datetimeformat
Format used to write ‘datetime’ values
- --target.no_compression
- --target.compression
Possible choices: GZIP, BZIP, LZO
Compression type of the file.
- --target.no_quoting
- --target.quoting
Possible choices: all, minimal, nonnumeric
How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.
- --target.no_escape
- --target.escape
Possible choices:
Character used to escape strings
- --target.encoding
Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252
Text encoding of file
- --target.dateformat
Format used to write ‘date’ values
- --target.timeonlyformat
Format used to write ‘time’ values
- --target.no_doublequote
- --target.doublequote
Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.
- --target.no_header_row
- --target.header_row
True if a header row is provided in the delimited files.
- --target.quotechar
A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
- --target.record_terminator
String used to close out individual rows of data.
- --target.field_delimiter
Character used between fields.
url2spectrum
Copy from url to spectrum
mvrec url2spectrum [-h] [--source.variant SOURCE.VARIANT]
[--source.format {avro,delimited,parquet}]
[--source.datetimeformattz SOURCE.DATETIMEFORMATTZ]
[--source.datetimeformat SOURCE.DATETIMEFORMAT]
[--source.no_compression]
[--source.compression {GZIP,BZIP,LZO}]
[--source.no_quoting]
[--source.quoting {all,minimal,nonnumeric}]
[--source.no_escape] [--source.escape {\}]
[--source.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
[--source.dateformat SOURCE.DATEFORMAT]
[--source.timeonlyformat SOURCE.TIMEONLYFORMAT]
[--source.no_doublequote] [--source.doublequote]
[--source.no_header_row] [--source.header_row]
[--source.quotechar SOURCE.QUOTECHAR]
[--source.record_terminator SOURCE.RECORD-TERMINATOR]
[--source.field_delimiter SOURCE.FIELD-DELIMITER]
[--target.spectrum_base_url TARGET.SPECTRUM_BASE_URL]
[--target.spectrum_rdir_url TARGET.SPECTRUM_RDIR_URL]
[--target.existing_table {delete_and_overwrite,truncate_and_overwrite,drop_and_recreate,append}]
source.input_url target.schema_name target.table_name
target.db_name
Positional Arguments
- source.input_url
Location of the data file. Must be a URL format understood by the records_mover.url library.
- target.schema_name
Schema name of a table to write data to.
- target.table_name
Table name of a table to write data to.
- target.db_name
SQLAlchemy database engine to write data to.
Named Arguments
- --source.variant
Records format variant - valid for ‘delimited’ records format type
- --source.format
Possible choices: avro, delimited, parquet
Records format type. Note that ‘delimited’ includes CSV/TSV/etc.
- --source.datetimeformattz
Format used to write ‘datetimetz’ values
- --source.datetimeformat
Format used to write ‘datetime’ values
- --source.no_compression
- --source.compression
Possible choices: GZIP, BZIP, LZO
Compression type of the file.
- --source.no_quoting
- --source.quoting
Possible choices: all, minimal, nonnumeric
How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.
- --source.no_escape
- --source.escape
Possible choices:
Character used to escape strings
- --source.encoding
Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252
Text encoding of file
- --source.dateformat
Format used to write ‘date’ values
- --source.timeonlyformat
Format used to write ‘time’ values
- --source.no_doublequote
- --source.doublequote
Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.
- --source.no_header_row
- --source.header_row
True if a header row is provided in the delimited files.
- --source.quotechar
A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
- --source.record_terminator
String used to close out individual rows of data.
- --source.field_delimiter
Character used between fields.
- --target.spectrum_base_url
Root S3 URL under which a simple directory structure will be created for files to be stored, if spectrum_rdir_url is not specified. Note that when using the mover CLI, db-facts may be used to provide a default.
- --target.spectrum_rdir_url
S3 URL where a records directory with files will be stored; otherwise, use db-facts default if exists. If this is not specified, spectrum_base_url must be.
- --target.existing_table
Possible choices: delete_and_overwrite, truncate_and_overwrite, drop_and_recreate, append
When loading into a database table, controls how any existing table found will be handled. This must be a
records_mover.records.ExistingTableHandling
object.Default: “delete_and_overwrite”
file2gsheet
Copy from file to gsheet
mvrec file2gsheet [-h] [--source.variant SOURCE.VARIANT]
[--source.format {avro,delimited,parquet}]
[--source.datetimeformattz SOURCE.DATETIMEFORMATTZ]
[--source.datetimeformat SOURCE.DATETIMEFORMAT]
[--source.no_compression]
[--source.compression {GZIP,BZIP,LZO}] [--source.no_quoting]
[--source.quoting {all,minimal,nonnumeric}]
[--source.no_escape] [--source.escape {\}]
[--source.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
[--source.dateformat SOURCE.DATEFORMAT]
[--source.timeonlyformat SOURCE.TIMEONLYFORMAT]
[--source.no_doublequote] [--source.doublequote]
[--source.no_header_row] [--source.header_row]
[--source.quotechar SOURCE.QUOTECHAR]
[--source.record_terminator SOURCE.RECORD-TERMINATOR]
[--source.field_delimiter SOURCE.FIELD-DELIMITER]
source.filename target.spreadsheet_id target.sheet_name
target.gcp_creds_name
Positional Arguments
- source.filename
File path (relative or absolute) of the data file to load.
- target.spreadsheet_id
This is the xyz in https://docs.google.com/spreadsheets/d/xyz/edit?ts=5be5b383#gid=abc
- target.sheet_name
This is the label of the particular tab within the Google Sheets spreadsheet where the data should go.
- target.gcp_creds_name
Credentials object for Google Cloud Platform access.
Named Arguments
- --source.variant
Records format variant - valid for ‘delimited’ records format type
- --source.format
Possible choices: avro, delimited, parquet
Records format type. Note that ‘delimited’ includes CSV/TSV/etc.
- --source.datetimeformattz
Format used to write ‘datetimetz’ values
- --source.datetimeformat
Format used to write ‘datetime’ values
- --source.no_compression
- --source.compression
Possible choices: GZIP, BZIP, LZO
Compression type of the file.
- --source.no_quoting
- --source.quoting
Possible choices: all, minimal, nonnumeric
How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.
- --source.no_escape
- --source.escape
Possible choices:
Character used to escape strings
- --source.encoding
Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252
Text encoding of file
- --source.dateformat
Format used to write ‘date’ values
- --source.timeonlyformat
Format used to write ‘time’ values
- --source.no_doublequote
- --source.doublequote
Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.
- --source.no_header_row
- --source.header_row
True if a header row is provided in the delimited files.
- --source.quotechar
A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
- --source.record_terminator
String used to close out individual rows of data.
- --source.field_delimiter
Character used between fields.
file2table
Copy from file to table
mvrec file2table [-h] [--source.variant SOURCE.VARIANT]
[--source.format {avro,delimited,parquet}]
[--source.datetimeformattz SOURCE.DATETIMEFORMATTZ]
[--source.datetimeformat SOURCE.DATETIMEFORMAT]
[--source.no_compression]
[--source.compression {GZIP,BZIP,LZO}] [--source.no_quoting]
[--source.quoting {all,minimal,nonnumeric}]
[--source.no_escape] [--source.escape {\}]
[--source.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
[--source.dateformat SOURCE.DATEFORMAT]
[--source.timeonlyformat SOURCE.TIMEONLYFORMAT]
[--source.no_doublequote] [--source.doublequote]
[--source.no_header_row] [--source.header_row]
[--source.quotechar SOURCE.QUOTECHAR]
[--source.record_terminator SOURCE.RECORD-TERMINATOR]
[--source.field_delimiter SOURCE.FIELD-DELIMITER]
[--target.existing_table {delete_and_overwrite,truncate_and_overwrite,drop_and_recreate,append}]
[--target.drop_and_recreate_on_load_error]
source.filename target.db_name target.schema_name
target.table_name
Positional Arguments
- source.filename
File path (relative or absolute) of the data file to load.
- target.db_name
SQLAlchemy database engine to write data to.
- target.schema_name
Schema name of a table to write data to.
- target.table_name
Table name of a table to write data to.
Named Arguments
- --source.variant
Records format variant - valid for ‘delimited’ records format type
- --source.format
Possible choices: avro, delimited, parquet
Records format type. Note that ‘delimited’ includes CSV/TSV/etc.
- --source.datetimeformattz
Format used to write ‘datetimetz’ values
- --source.datetimeformat
Format used to write ‘datetime’ values
- --source.no_compression
- --source.compression
Possible choices: GZIP, BZIP, LZO
Compression type of the file.
- --source.no_quoting
- --source.quoting
Possible choices: all, minimal, nonnumeric
How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.
- --source.no_escape
- --source.escape
Possible choices:
Character used to escape strings
- --source.encoding
Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252
Text encoding of file
- --source.dateformat
Format used to write ‘date’ values
- --source.timeonlyformat
Format used to write ‘time’ values
- --source.no_doublequote
- --source.doublequote
Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.
- --source.no_header_row
- --source.header_row
True if a header row is provided in the delimited files.
- --source.quotechar
A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
- --source.record_terminator
String used to close out individual rows of data.
- --source.field_delimiter
Character used between fields.
- --target.existing_table
Possible choices: delete_and_overwrite, truncate_and_overwrite, drop_and_recreate, append
When loading into a database table, controls how any existing table found will be handled. This must be a
records_mover.records.ExistingTableHandling
object.Default: “delete_and_overwrite”
- --target.drop_and_recreate_on_load_error
If True, table load errors will attempt to be addressed by dropping the target table and reloading the incoming data.
Default: False
file2recordsdir
Copy from file to recordsdir
mvrec file2recordsdir [-h] [--source.variant SOURCE.VARIANT]
[--source.format {avro,delimited,parquet}]
[--source.datetimeformattz SOURCE.DATETIMEFORMATTZ]
[--source.datetimeformat SOURCE.DATETIMEFORMAT]
[--source.no_compression]
[--source.compression {GZIP,BZIP,LZO}]
[--source.no_quoting]
[--source.quoting {all,minimal,nonnumeric}]
[--source.no_escape] [--source.escape {\}]
[--source.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
[--source.dateformat SOURCE.DATEFORMAT]
[--source.timeonlyformat SOURCE.TIMEONLYFORMAT]
[--source.no_doublequote] [--source.doublequote]
[--source.no_header_row] [--source.header_row]
[--source.quotechar SOURCE.QUOTECHAR]
[--source.record_terminator SOURCE.RECORD-TERMINATOR]
[--source.field_delimiter SOURCE.FIELD-DELIMITER]
[--target.variant TARGET.VARIANT]
[--target.format {avro,delimited,parquet}]
[--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
[--target.datetimeformat TARGET.DATETIMEFORMAT]
[--target.no_compression]
[--target.compression {GZIP,BZIP,LZO}]
[--target.no_quoting]
[--target.quoting {all,minimal,nonnumeric}]
[--target.no_escape] [--target.escape {\}]
[--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
[--target.dateformat TARGET.DATEFORMAT]
[--target.timeonlyformat TARGET.TIMEONLYFORMAT]
[--target.no_doublequote] [--target.doublequote]
[--target.no_header_row] [--target.header_row]
[--target.quotechar TARGET.QUOTECHAR]
[--target.record_terminator TARGET.RECORD-TERMINATOR]
[--target.field_delimiter TARGET.FIELD-DELIMITER]
source.filename target.output_url
Positional Arguments
- source.filename
File path (relative or absolute) of the data file to load.
- target.output_url
Location to write the records directory. Must be a URL format understood by the records_mover.url library, and must be a directory URL that ends with a ‘/’.
Named Arguments
- --source.variant
Records format variant - valid for ‘delimited’ records format type
- --source.format
Possible choices: avro, delimited, parquet
Records format type. Note that ‘delimited’ includes CSV/TSV/etc.
- --source.datetimeformattz
Format used to write ‘datetimetz’ values
- --source.datetimeformat
Format used to write ‘datetime’ values
- --source.no_compression
- --source.compression
Possible choices: GZIP, BZIP, LZO
Compression type of the file.
- --source.no_quoting
- --source.quoting
Possible choices: all, minimal, nonnumeric
How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.
- --source.no_escape
- --source.escape
Possible choices:
Character used to escape strings
- --source.encoding
Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252
Text encoding of file
- --source.dateformat
Format used to write ‘date’ values
- --source.timeonlyformat
Format used to write ‘time’ values
- --source.no_doublequote
- --source.doublequote
Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.
- --source.no_header_row
- --source.header_row
True if a header row is provided in the delimited files.
- --source.quotechar
A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
- --source.record_terminator
String used to close out individual rows of data.
- --source.field_delimiter
Character used between fields.
- --target.variant
Records format variant - valid for ‘delimited’ records format type
- --target.format
Possible choices: avro, delimited, parquet
Records format type. Note that ‘delimited’ includes CSV/TSV/etc.
- --target.datetimeformattz
Format used to write ‘datetimetz’ values
- --target.datetimeformat
Format used to write ‘datetime’ values
- --target.no_compression
- --target.compression
Possible choices: GZIP, BZIP, LZO
Compression type of the file.
- --target.no_quoting
- --target.quoting
Possible choices: all, minimal, nonnumeric
How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.
- --target.no_escape
- --target.escape
Possible choices:
Character used to escape strings
- --target.encoding
Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252
Text encoding of file
- --target.dateformat
Format used to write ‘date’ values
- --target.timeonlyformat
Format used to write ‘time’ values
- --target.no_doublequote
- --target.doublequote
Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.
- --target.no_header_row
- --target.header_row
True if a header row is provided in the delimited files.
- --target.quotechar
A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
- --target.record_terminator
String used to close out individual rows of data.
- --target.field_delimiter
Character used between fields.
file2url
Copy from file to url
mvrec file2url [-h] [--source.variant SOURCE.VARIANT]
[--source.format {avro,delimited,parquet}]
[--source.datetimeformattz SOURCE.DATETIMEFORMATTZ]
[--source.datetimeformat SOURCE.DATETIMEFORMAT]
[--source.no_compression]
[--source.compression {GZIP,BZIP,LZO}] [--source.no_quoting]
[--source.quoting {all,minimal,nonnumeric}]
[--source.no_escape] [--source.escape {\}]
[--source.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
[--source.dateformat SOURCE.DATEFORMAT]
[--source.timeonlyformat SOURCE.TIMEONLYFORMAT]
[--source.no_doublequote] [--source.doublequote]
[--source.no_header_row] [--source.header_row]
[--source.quotechar SOURCE.QUOTECHAR]
[--source.record_terminator SOURCE.RECORD-TERMINATOR]
[--source.field_delimiter SOURCE.FIELD-DELIMITER]
[--target.variant TARGET.VARIANT]
[--target.format {avro,delimited,parquet}]
[--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
[--target.datetimeformat TARGET.DATETIMEFORMAT]
[--target.no_compression]
[--target.compression {GZIP,BZIP,LZO}] [--target.no_quoting]
[--target.quoting {all,minimal,nonnumeric}]
[--target.no_escape] [--target.escape {\}]
[--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
[--target.dateformat TARGET.DATEFORMAT]
[--target.timeonlyformat TARGET.TIMEONLYFORMAT]
[--target.no_doublequote] [--target.doublequote]
[--target.no_header_row] [--target.header_row]
[--target.quotechar TARGET.QUOTECHAR]
[--target.record_terminator TARGET.RECORD-TERMINATOR]
[--target.field_delimiter TARGET.FIELD-DELIMITER]
source.filename target.output_url
Positional Arguments
- source.filename
File path (relative or absolute) of the data file to load.
- target.output_url
Location of the data file to write. Must be a URL format understood by the records_mover.url library corresponding to a file, not a directory (i.e., not ending with a ‘/’)
Named Arguments
- --source.variant
Records format variant - valid for ‘delimited’ records format type
- --source.format
Possible choices: avro, delimited, parquet
Records format type. Note that ‘delimited’ includes CSV/TSV/etc.
- --source.datetimeformattz
Format used to write ‘datetimetz’ values
- --source.datetimeformat
Format used to write ‘datetime’ values
- --source.no_compression
- --source.compression
Possible choices: GZIP, BZIP, LZO
Compression type of the file.
- --source.no_quoting
- --source.quoting
Possible choices: all, minimal, nonnumeric
How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.
- --source.no_escape
- --source.escape
Possible choices:
Character used to escape strings
- --source.encoding
Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252
Text encoding of file
- --source.dateformat
Format used to write ‘date’ values
- --source.timeonlyformat
Format used to write ‘time’ values
- --source.no_doublequote
- --source.doublequote
Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.
- --source.no_header_row
- --source.header_row
True if a header row is provided in the delimited files.
- --source.quotechar
A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
- --source.record_terminator
String used to close out individual rows of data.
- --source.field_delimiter
Character used between fields.
- --target.variant
Records format variant - valid for ‘delimited’ records format type
- --target.format
Possible choices: avro, delimited, parquet
Records format type. Note that ‘delimited’ includes CSV/TSV/etc.
- --target.datetimeformattz
Format used to write ‘datetimetz’ values
- --target.datetimeformat
Format used to write ‘datetime’ values
- --target.no_compression
- --target.compression
Possible choices: GZIP, BZIP, LZO
Compression type of the file.
- --target.no_quoting
- --target.quoting
Possible choices: all, minimal, nonnumeric
How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.
- --target.no_escape
- --target.escape
Possible choices:
Character used to escape strings
- --target.encoding
Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252
Text encoding of file
- --target.dateformat
Format used to write ‘date’ values
- --target.timeonlyformat
Format used to write ‘time’ values
- --target.no_doublequote
- --target.doublequote
Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.
- --target.no_header_row
- --target.header_row
True if a header row is provided in the delimited files.
- --target.quotechar
A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
- --target.record_terminator
String used to close out individual rows of data.
- --target.field_delimiter
Character used between fields.
file2file
Copy from file to file
mvrec file2file [-h] [--source.variant SOURCE.VARIANT]
[--source.format {avro,delimited,parquet}]
[--source.datetimeformattz SOURCE.DATETIMEFORMATTZ]
[--source.datetimeformat SOURCE.DATETIMEFORMAT]
[--source.no_compression]
[--source.compression {GZIP,BZIP,LZO}] [--source.no_quoting]
[--source.quoting {all,minimal,nonnumeric}]
[--source.no_escape] [--source.escape {\}]
[--source.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
[--source.dateformat SOURCE.DATEFORMAT]
[--source.timeonlyformat SOURCE.TIMEONLYFORMAT]
[--source.no_doublequote] [--source.doublequote]
[--source.no_header_row] [--source.header_row]
[--source.quotechar SOURCE.QUOTECHAR]
[--source.record_terminator SOURCE.RECORD-TERMINATOR]
[--source.field_delimiter SOURCE.FIELD-DELIMITER]
[--target.variant TARGET.VARIANT]
[--target.format {avro,delimited,parquet}]
[--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
[--target.datetimeformat TARGET.DATETIMEFORMAT]
[--target.no_compression]
[--target.compression {GZIP,BZIP,LZO}] [--target.no_quoting]
[--target.quoting {all,minimal,nonnumeric}]
[--target.no_escape] [--target.escape {\}]
[--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
[--target.dateformat TARGET.DATEFORMAT]
[--target.timeonlyformat TARGET.TIMEONLYFORMAT]
[--target.no_doublequote] [--target.doublequote]
[--target.no_header_row] [--target.header_row]
[--target.quotechar TARGET.QUOTECHAR]
[--target.record_terminator TARGET.RECORD-TERMINATOR]
[--target.field_delimiter TARGET.FIELD-DELIMITER]
source.filename target.filename
Positional Arguments
- source.filename
File path (relative or absolute) of the data file to load.
- target.filename
File path (relative or absolute) of the data file to unload to.
Named Arguments
- --source.variant
Records format variant - valid for ‘delimited’ records format type
- --source.format
Possible choices: avro, delimited, parquet
Records format type. Note that ‘delimited’ includes CSV/TSV/etc.
- --source.datetimeformattz
Format used to write ‘datetimetz’ values
- --source.datetimeformat
Format used to write ‘datetime’ values
- --source.no_compression
- --source.compression
Possible choices: GZIP, BZIP, LZO
Compression type of the file.
- --source.no_quoting
- --source.quoting
Possible choices: all, minimal, nonnumeric
How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.
- --source.no_escape
- --source.escape
Possible choices:
Character used to escape strings
- --source.encoding
Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252
Text encoding of file
- --source.dateformat
Format used to write ‘date’ values
- --source.timeonlyformat
Format used to write ‘time’ values
- --source.no_doublequote
- --source.doublequote
Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.
- --source.no_header_row
- --source.header_row
True if a header row is provided in the delimited files.
- --source.quotechar
A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
- --source.record_terminator
String used to close out individual rows of data.
- --source.field_delimiter
Character used between fields.
- --target.variant
Records format variant - valid for ‘delimited’ records format type
- --target.format
Possible choices: avro, delimited, parquet
Records format type. Note that ‘delimited’ includes CSV/TSV/etc.
- --target.datetimeformattz
Format used to write ‘datetimetz’ values
- --target.datetimeformat
Format used to write ‘datetime’ values
- --target.no_compression
- --target.compression
Possible choices: GZIP, BZIP, LZO
Compression type of the file.
- --target.no_quoting
- --target.quoting
Possible choices: all, minimal, nonnumeric
How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.
- --target.no_escape
- --target.escape
Possible choices:
Character used to escape strings
- --target.encoding
Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252
Text encoding of file
- --target.dateformat
Format used to write ‘date’ values
- --target.timeonlyformat
Format used to write ‘time’ values
- --target.no_doublequote
- --target.doublequote
Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.
- --target.no_header_row
- --target.header_row
True if a header row is provided in the delimited files.
- --target.quotechar
A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
- --target.record_terminator
String used to close out individual rows of data.
- --target.field_delimiter
Character used between fields.
file2spectrum
Copy from file to spectrum
mvrec file2spectrum [-h] [--source.variant SOURCE.VARIANT]
[--source.format {avro,delimited,parquet}]
[--source.datetimeformattz SOURCE.DATETIMEFORMATTZ]
[--source.datetimeformat SOURCE.DATETIMEFORMAT]
[--source.no_compression]
[--source.compression {GZIP,BZIP,LZO}]
[--source.no_quoting]
[--source.quoting {all,minimal,nonnumeric}]
[--source.no_escape] [--source.escape {\}]
[--source.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
[--source.dateformat SOURCE.DATEFORMAT]
[--source.timeonlyformat SOURCE.TIMEONLYFORMAT]
[--source.no_doublequote] [--source.doublequote]
[--source.no_header_row] [--source.header_row]
[--source.quotechar SOURCE.QUOTECHAR]
[--source.record_terminator SOURCE.RECORD-TERMINATOR]
[--source.field_delimiter SOURCE.FIELD-DELIMITER]
[--target.spectrum_base_url TARGET.SPECTRUM_BASE_URL]
[--target.spectrum_rdir_url TARGET.SPECTRUM_RDIR_URL]
[--target.existing_table {delete_and_overwrite,truncate_and_overwrite,drop_and_recreate,append}]
source.filename target.schema_name target.table_name
target.db_name
Positional Arguments
- source.filename
File path (relative or absolute) of the data file to load.
- target.schema_name
Schema name of a table to write data to.
- target.table_name
Table name of a table to write data to.
- target.db_name
SQLAlchemy database engine to write data to.
Named Arguments
- --source.variant
Records format variant - valid for ‘delimited’ records format type
- --source.format
Possible choices: avro, delimited, parquet
Records format type. Note that ‘delimited’ includes CSV/TSV/etc.
- --source.datetimeformattz
Format used to write ‘datetimetz’ values
- --source.datetimeformat
Format used to write ‘datetime’ values
- --source.no_compression
- --source.compression
Possible choices: GZIP, BZIP, LZO
Compression type of the file.
- --source.no_quoting
- --source.quoting
Possible choices: all, minimal, nonnumeric
How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.
- --source.no_escape
- --source.escape
Possible choices:
Character used to escape strings
- --source.encoding
Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252
Text encoding of file
- --source.dateformat
Format used to write ‘date’ values
- --source.timeonlyformat
Format used to write ‘time’ values
- --source.no_doublequote
- --source.doublequote
Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.
- --source.no_header_row
- --source.header_row
True if a header row is provided in the delimited files.
- --source.quotechar
A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.
- --source.record_terminator
String used to close out individual rows of data.
- --source.field_delimiter
Character used between fields.
- --target.spectrum_base_url
Root S3 URL under which a simple directory structure will be created for files to be stored, if spectrum_rdir_url is not specified. Note that when using the mover CLI, db-facts may be used to provide a default.
- --target.spectrum_rdir_url
S3 URL where a records directory with files will be stored; otherwise, use db-facts default if exists. If this is not specified, spectrum_base_url must be.
- --target.existing_table
Possible choices: delete_and_overwrite, truncate_and_overwrite, drop_and_recreate, append
When loading into a database table, controls how any existing table found will be handled. This must be a
records_mover.records.ExistingTableHandling
object.Default: “delete_and_overwrite”