records_mover package

Subpackages

• records_mover.airflow package
  • Subpackages
    • records_mover.airflow.hooks package
      • Module contents
• records_mover.records package
  • Subpackages
    • records_mover.records.sources package
      • Module contents
    • records_mover.records.targets package
      • Module contents
  • Submodules
  • records_mover.records.base_records_format module
    • BaseRecordsFormat
  • Module contents
    • DelimitedRecordsFormat
      • DelimitedRecordsFormat.__init__()
    • ParquetRecordsFormat
      • ParquetRecordsFormat.__init__()
    • ProcessingInstructions
      • ProcessingInstructions.__init__()
    • DelimitedVariant
    • AvroRecordsFormat
      • AvroRecordsFormat.format_type
      • AvroRecordsFormat.generate_filename()
    • ExistingTableHandling
      • ExistingTableHandling.DELETE_AND_OVERWRITE
      • ExistingTableHandling.TRUNCATE_AND_OVERWRITE
      • ExistingTableHandling.DROP_AND_RECREATE
      • ExistingTableHandling.APPEND
    • RecordsFolderNonEmptyException
    • RecordsException
    • Records
      • Records.move
      • Records.sources
      • Records.targets
    • move()
    • MoveResult
      • MoveResult.move_count
      • MoveResult.output_urls

Module contents

class records_mover.Session(default_db_creds_name=None, default_aws_creds_name=PleaseInfer.token, default_gcp_creds_name=PleaseInfer.token, session_type=PleaseInfer.token, scratch_s3_url=PleaseInfer.token, creds=PleaseInfer.token, default_db_facts=PleaseInfer.token, default_boto3_session=PleaseInfer.token, default_gcp_creds=PleaseInfer.token, default_gcs_client=PleaseInfer.token, scratch_gcs_url=PleaseInfer.token)

Bases: object

Parameters

• default_db_creds_name (Optional[str]) –

• default_aws_creds_name (Union[None, str, PleaseInfer]) –

• default_gcp_creds_name (Union[None, str, PleaseInfer]) –

• session_type (Union[str, PleaseInfer]) –

• scratch_s3_url (Union[None, str, PleaseInfer]) –

• creds (Union[BaseCreds, PleaseInfer]) –

• default_db_facts (Union[PleaseInfer, Dict[str, Any]]) –

• default_boto3_session (Optional[Union[PleaseInfer, boto3.session.Session]]) –

• default_gcp_creds (Optional[Union[PleaseInfer, google.auth.credentials.Credentials]]) –

• default_gcs_client (Optional[Union[PleaseInfer, google.cloud.storage.Client]]) –

• scratch_gcs_url (Union[None, str, PleaseInfer]) –

__init__(default_db_creds_name=None, default_aws_creds_name=PleaseInfer.token, default_gcp_creds_name=PleaseInfer.token, session_type=PleaseInfer.token, scratch_s3_url=PleaseInfer.token, creds=PleaseInfer.token, default_db_facts=PleaseInfer.token, default_boto3_session=PleaseInfer.token, default_gcp_creds=PleaseInfer.token, default_gcs_client=PleaseInfer.token, scratch_gcs_url=PleaseInfer.token)

This is an object which ties together configuration on how to do key things in order to move records.

It tries to autoconfigure as much as possible - in many cases you won’t need to specify any constructor arguments at all.

Generally unless otherwise configured, this class will look up and use the default credentials for things like AWS and GCP if they exist and are needed for an operation. When running in a managed environment like Apache Airflow (session_type = “airflow”), that might mean looking up an Airflow Connection via the Airflow Python API. On the command line (session_type = “cli”), that might mean using e.g., the AWS or GCP Python APIs to pull any default credentials which have been configured. In other environments (e.g., containerized systems) you may way want to use environment variables whenever possible to specify exactly what is desired (sesssion_type = ‘env’).

Parameters

• default_db_creds_name (Optional[str]) – Name of the database credential to used when records_mover.Session.get_default_db_engine() is called. If not specified, the default will depend on the session type.

• default_aws_creds_name (Union[None, str, PleaseInfer]) – Name of the AWS IAM credential to used when needed, e.g. when reading or writing to an s3:// URL. This will be inferred unless directly specified.

• default_gcp_creds_name (Union[None, str, PleaseInfer]) – Name of the GCP Cloud IAM credential to used when needed, e.g. when reading or writing to an gs:// URL. This will be inferred unless directly specified.

• session_type (Union[str, PleaseInfer]) – What assumptions to use when inferring and/or looking up credentials. Valid values of “airflow” (for code running in Apache Airflow), “cli” (for running on the command-line”, “lpass” (for using the LastPass password manager for credentials), and ‘env’ (for looking up credentials via environment variables). This will be inferred unless directly specified.

• scratch_s3_url (Union[None, str, PleaseInfer]) – An s3:// URL used as a base directory where temporary files/directories can be created. This is necessary for Amazon Redshift, which supports only S3 for bulk import/export.

• default_db_facts (Union[PleaseInfer, Dict[str, Any]]) – Information about the database connection that should be made. This is a dictionary with string keys of type DBFacts

• default_boto3_session (Optional[Union[PleaseInfer, boto3.session.Session]]) – The boto3.Session object used when needed, e.g. when reading or writing to an s3:// URL. This will be inferred unless directly specified.

• default_gcp_creds (Optional[Union[PleaseInfer, google.auth.credentials.Credentials]]) – The google.auth.credentials.Credentials object to used when needed, e.g. when reading or writing to an gs:// URL. This will be inferred unless directly specified.

• default_gcs_client (Optional[Union[PleaseInfer, google.cloud.storage.Client]]) – The google.cloud.storage.Client object to be used when needed, e.g. when reading or writing to an gs:// URL. This will be inferred unless directly specified.

• creds (Union[BaseCreds, PleaseInfer]) – Experimental interface; do not use.

• scratch_gcs_url (Union[None, str, PleaseInfer]) – A gs:// URL used as a base directory where temporary files/directories can be created. This can be helpful for large imports into Google BigQuery.

Return type

None

get_default_db_engine()

Provide the database object corresponding to the default database credentials. The details of how that credential is looked up depends on the session_type determined in the constructor, but can be overridden using the default_db_creds_name parameter.

Returns

SQLALchemy Engine object

Return type

Engine

get_db_engine(db_creds_name, creds_provider=None)

Provide a database object corresponding to a given credential name. The details of how that credential is looked up depends on the session_type determined in the constructor.

Parameters

• db_creds_name (str) – Credential name to look up using the configured credentials provider.

• creds_provider (Optional[BaseCreds]) –

Returns

SQLALchemy Engine object

Return type

Engine

set_stream_logging(name='records_mover', level=20, stream=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, fmt='%(asctime)s - %(message)s', datefmt='%H:%M:%S')

records-mover logs details about its operations using Python logging. This method is a simple way to configure that logging to be output to a stream (by default, stdout).

You can use it for other things (e.g., dependencies of records-mover) by adjusting the ‘name’ argument.

Parameters

• name (str) – Name of the package to set logging under. If set to ‘foo’, you can set a log variable FOO_LOG_LEVEL to the log level threshold you’d like to set (INFO/WARNING/etc) - so you can by default set, say, export RECORDS_MOVER_LOG_LEVEL=WARNING to quiet down loging, or export RECORDS_MOVER_LOG_LEVEL=DEBUG to increase it.

• level (int) – Logging more detailed than this will not be output to the stream.

• stream (IO[str]) – Stream which logging should be sent (e.g., sys.stdout, sys.stdin, or perhaps a file you open)

• fmt (str) – Logging format to send to Python’slogging.Formatter() - determines what details will be sent.

• datefmt (str) – Date format to send to Python’slogging.Formatter() - determines how the current date/time will be recorded in the log.

Return type

None

property records: Records

Property containing a records_mover.Records object pre-configured with configuration using this Session. Once you have a Session object constructed, this is your jumping off point to moving records.

class records_mover.Records(db_driver=PleaseInfer.token, url_resolver=PleaseInfer.token, session=PleaseInfer.token)

Bases: object

To move records from one place to another, you can use the methods on this object.

This object should be pulled from the ‘records’ property on a records_mover.Session object instead of being constructed directly.

To move data, you can call the records_mover.records.move() method, which is aliased for your convenience on this object.

Example:

records = session.records
db_engine = session.get_default_db_engine()
url = 's3://some-bucket/some-directory/'
source = records.sources.directory_from_url(url=url)
target = records.targets.table(schema_name='myschema',
                               table_name='mytable',
                               db_engine=db_engine)
results = records.move(source, target)

Parameters

• db_driver (Union[Callable[[Optional[Union[Engine, Connection]], Optional[Connection], Optional[Engine]], DBDriver], PleaseInfer]) –

• url_resolver (Union[UrlResolver, PleaseInfer]) –

• session (Union[Session, PleaseInfer]) –

move: Callable

Alias of records_mover.records.move()

sources: RecordsSources

Object containing factory methods to create various sources from which to copy records, of type records_mover.records.sources.RecordsSources

targets: RecordsTargets

Object containing factory methods to create various targets to which records can be copied, of type records_mover.records.targets.RecordsTargets

records_mover.set_stream_logging(name='records_mover', level=20, stream=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, fmt='%(asctime)s - %(message)s', datefmt='%H:%M:%S')

records-mover logs details about its operations using Python logging. This method is a simple way to configure that logging to be output to a stream (by default, stdout).

You can use it for other things (e.g., dependencies of records-mover) by adjusting the ‘name’ argument.

Parameters

• name (str) – Name of the package to set logging under. If set to ‘foo’, you can set a log variable FOO_LOG_LEVEL to the log level threshold you’d like to set (INFO/WARNING/etc) - so you can by default set, say, export RECORDS_MOVER_LOG_LEVEL=WARNING to quiet down loging, or export RECORDS_MOVER_LOG_LEVEL=DEBUG to increase it.

• level (int) – Logging more detailed than this will not be output to the stream.

• stream (IO[str]) – Stream which logging should be sent (e.g., sys.stdout, sys.stdin, or perhaps a file you open)

• fmt (str) – Logging format to send to Python’slogging.Formatter() - determines what details will be sent.

• datefmt (str) – Date format to send to Python’slogging.Formatter() - determines how the current date/time will be recorded in the log.

Return type

None

records_mover.move(records_source, records_target, processing_instructions=<records_mover.records.processing_instructions.ProcessingInstructions object>)

Copy records from one location to another. Applies a sequence of possible techniques to do this in an efficient way and respects the preferences set in records_source, records_target and processing_instructions.

Example use:

records = session.records
db_engine = session.get_default_db_engine()
url = 's3://some-bucket/some-directory/'
source = records.sources.directory_from_url(url=url)
target = records.targets.table(schema_name='myschema',
                               table_name='mytable',
                               db_engine=db_engine)
results = records.move(source, target)

Parameters

• records_source (RecordsSource) – object returned by a factory method in records_mover.records.sources.RecordsSources which represents the place we’re copying records from.

• records_target (RecordsTarget) – object returned by a factory method in records_mover.records.targets.RecordsTargets which represents the place we’re copying records to.

• processing_instructions (records_mover.records.ProcessingInstructions) – Directives on how to handle different situations when processing files.

Return type

records_mover.records.MoveResult

mvrec CLI options

Move tabular data (“records”) from one place to another

usage: mvrec [-h] [--no_fail_if_dont_understand]
             [--no_fail_if_cant_handle_hint] [--no_fail_if_row_invalid]
             [--max_inference_rows MAX_INFERENCE_ROWS]
             [--max_failure_rows MAX_FAILURE_ROWS] [-V]
             {table2gsheet,table2table,table2recordsdir,table2url,table2file,table2spectrum,gsheet2gsheet,gsheet2table,gsheet2recordsdir,gsheet2url,gsheet2file,gsheet2spectrum,recordsdir2gsheet,recordsdir2table,recordsdir2recordsdir,recordsdir2url,recordsdir2file,recordsdir2spectrum,url2gsheet,url2table,url2recordsdir,url2url,url2file,url2spectrum,file2gsheet,file2table,file2recordsdir,file2url,file2file,file2spectrum}
             ...

Named Arguments

--no_fail_if_dont_understand

If True, and a part of the RecordsFormat is not understood while processing, then immediately fail and raise an exception. Otherwise, ignore the misunderstood instruction (e.g., ignore the hint, assume default variant, etc etc)

Default: True

--no_fail_if_cant_handle_hint

If True, and for whatever reason (e.g., limited options in whatever library/tool/database is being used) a certain hint can’t be handled as specified, raise an exception. Otherwise, ignore the hint and use implementation-specific different behavior.

Default: True

--no_fail_if_row_invalid

If True, and a particular row of data in the records file cannot be understood by the library, raise an exception. Otherwise, ignore the row and continue and try to load other rows.

Default: True

--max_inference_rows

If the schema is not provided and we need it (e.g., we’re to load the records into a database and there’s no existing table), we’ll figure it out through ‘type inference’ - looking at a bunch of examples of data and building a specific schema that can load those rows. This can take some time, so this parameter controls the maximum number of rows we’ll look at. Higher values will be more likely to result in a schema that can be loaded into, but will take longer to load. If set to None, the entire file will be processed.

Default: 1000000

--max_failure_rows

Sets a tolerance level for number of rows of data in the records file that cannot be understood by the library that should be ignored. After reaching level, raise an exception.

-V, --version

show program’s version number and exit

Sub-commands

table2gsheet

Copy from table to gsheet

mvrec table2gsheet [-h]
                   source.db_name source.schema_name source.table_name
                   target.spreadsheet_id target.sheet_name
                   target.gcp_creds_name

Positional Arguments

source.db_name

SQLAlchemy database engine to pull data from.

source.schema_name

Schema name of a table to get data from.

source.table_name

Table name of a table to get data from.

target.spreadsheet_id

This is the xyz in https://docs.google.com/spreadsheets/d/xyz/edit?ts=5be5b383#gid=abc

target.sheet_name

This is the label of the particular tab within the Google Sheets spreadsheet where the data should go.

target.gcp_creds_name

Credentials object for Google Cloud Platform access.

table2table

Copy from table to table

mvrec table2table [-h]
                  [--target.existing_table {delete_and_overwrite,truncate_and_overwrite,drop_and_recreate,append}]
                  [--target.drop_and_recreate_on_load_error]
                  source.db_name source.schema_name source.table_name
                  target.db_name target.schema_name target.table_name

Positional Arguments

source.db_name

SQLAlchemy database engine to pull data from.

source.schema_name

Schema name of a table to get data from.

source.table_name

Table name of a table to get data from.

target.db_name

SQLAlchemy database engine to write data to.

target.schema_name

Schema name of a table to write data to.

target.table_name

Table name of a table to write data to.

Named Arguments

--target.existing_table

Possible choices: delete_and_overwrite, truncate_and_overwrite, drop_and_recreate, append

When loading into a database table, controls how any existing table found will be handled. This must be a records_mover.records.ExistingTableHandling object.

Default: “delete_and_overwrite”

--target.drop_and_recreate_on_load_error

If True, table load errors will attempt to be addressed by dropping the target table and reloading the incoming data.

Default: False

table2recordsdir

Copy from table to recordsdir

mvrec table2recordsdir [-h] [--target.variant TARGET.VARIANT]
                       [--target.format {avro,delimited,parquet}]
                       [--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
                       [--target.datetimeformat TARGET.DATETIMEFORMAT]
                       [--target.no_compression]
                       [--target.compression {GZIP,BZIP,LZO}]
                       [--target.no_quoting]
                       [--target.quoting {all,minimal,nonnumeric}]
                       [--target.no_escape] [--target.escape {\}]
                       [--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
                       [--target.dateformat TARGET.DATEFORMAT]
                       [--target.timeonlyformat TARGET.TIMEONLYFORMAT]
                       [--target.no_doublequote] [--target.doublequote]
                       [--target.no_header_row] [--target.header_row]
                       [--target.quotechar TARGET.QUOTECHAR]
                       [--target.record_terminator TARGET.RECORD-TERMINATOR]
                       [--target.field_delimiter TARGET.FIELD-DELIMITER]
                       source.db_name source.schema_name source.table_name
                       target.output_url

Positional Arguments

source.db_name

SQLAlchemy database engine to pull data from.

source.schema_name

Schema name of a table to get data from.

source.table_name

Table name of a table to get data from.

target.output_url

Location to write the records directory. Must be a URL format understood by the records_mover.url library, and must be a directory URL that ends with a ‘/’.

Named Arguments

--target.variant

Records format variant - valid for ‘delimited’ records format type

--target.format

Possible choices: avro, delimited, parquet

Records format type. Note that ‘delimited’ includes CSV/TSV/etc.

--target.datetimeformattz

Format used to write ‘datetimetz’ values

--target.datetimeformat

Format used to write ‘datetime’ values

--target.no_compression

--target.compression

Possible choices: GZIP, BZIP, LZO

Compression type of the file.

--target.no_quoting

--target.quoting

Possible choices: all, minimal, nonnumeric

How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.

--target.no_escape

--target.escape

Possible choices:

Character used to escape strings

--target.encoding

Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252

Text encoding of file

--target.dateformat

Format used to write ‘date’ values

--target.timeonlyformat

Format used to write ‘time’ values

--target.no_doublequote

--target.doublequote

Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.

--target.no_header_row

--target.header_row

True if a header row is provided in the delimited files.

--target.quotechar

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

--target.record_terminator

String used to close out individual rows of data.

--target.field_delimiter

Character used between fields.

table2url

Copy from table to url

mvrec table2url [-h] [--target.variant TARGET.VARIANT]
                [--target.format {avro,delimited,parquet}]
                [--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
                [--target.datetimeformat TARGET.DATETIMEFORMAT]
                [--target.no_compression]
                [--target.compression {GZIP,BZIP,LZO}] [--target.no_quoting]
                [--target.quoting {all,minimal,nonnumeric}]
                [--target.no_escape] [--target.escape {\}]
                [--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
                [--target.dateformat TARGET.DATEFORMAT]
                [--target.timeonlyformat TARGET.TIMEONLYFORMAT]
                [--target.no_doublequote] [--target.doublequote]
                [--target.no_header_row] [--target.header_row]
                [--target.quotechar TARGET.QUOTECHAR]
                [--target.record_terminator TARGET.RECORD-TERMINATOR]
                [--target.field_delimiter TARGET.FIELD-DELIMITER]
                source.db_name source.schema_name source.table_name
                target.output_url

Positional Arguments

source.db_name

SQLAlchemy database engine to pull data from.

source.schema_name

Schema name of a table to get data from.

source.table_name

Table name of a table to get data from.

target.output_url

Location of the data file to write. Must be a URL format understood by the records_mover.url library corresponding to a file, not a directory (i.e., not ending with a ‘/’)

Named Arguments

--target.variant

Records format variant - valid for ‘delimited’ records format type

--target.format

Possible choices: avro, delimited, parquet

Records format type. Note that ‘delimited’ includes CSV/TSV/etc.

--target.datetimeformattz

Format used to write ‘datetimetz’ values

--target.datetimeformat

Format used to write ‘datetime’ values

--target.no_compression

--target.compression

Possible choices: GZIP, BZIP, LZO

Compression type of the file.

--target.no_quoting

--target.quoting

Possible choices: all, minimal, nonnumeric

How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.

--target.no_escape

--target.escape

Possible choices:

Character used to escape strings

--target.encoding

Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252

Text encoding of file

--target.dateformat

Format used to write ‘date’ values

--target.timeonlyformat

Format used to write ‘time’ values

--target.no_doublequote

--target.doublequote

Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.

--target.no_header_row

--target.header_row

True if a header row is provided in the delimited files.

--target.quotechar

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

--target.record_terminator

String used to close out individual rows of data.

--target.field_delimiter

Character used between fields.

table2file

Copy from table to file

mvrec table2file [-h] [--target.variant TARGET.VARIANT]
                 [--target.format {avro,delimited,parquet}]
                 [--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
                 [--target.datetimeformat TARGET.DATETIMEFORMAT]
                 [--target.no_compression]
                 [--target.compression {GZIP,BZIP,LZO}] [--target.no_quoting]
                 [--target.quoting {all,minimal,nonnumeric}]
                 [--target.no_escape] [--target.escape {\}]
                 [--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
                 [--target.dateformat TARGET.DATEFORMAT]
                 [--target.timeonlyformat TARGET.TIMEONLYFORMAT]
                 [--target.no_doublequote] [--target.doublequote]
                 [--target.no_header_row] [--target.header_row]
                 [--target.quotechar TARGET.QUOTECHAR]
                 [--target.record_terminator TARGET.RECORD-TERMINATOR]
                 [--target.field_delimiter TARGET.FIELD-DELIMITER]
                 source.db_name source.schema_name source.table_name
                 target.filename

Positional Arguments

source.db_name

SQLAlchemy database engine to pull data from.

source.schema_name

Schema name of a table to get data from.

source.table_name

Table name of a table to get data from.

target.filename

File path (relative or absolute) of the data file to unload to.

Named Arguments

--target.variant

Records format variant - valid for ‘delimited’ records format type

--target.format

Possible choices: avro, delimited, parquet

Records format type. Note that ‘delimited’ includes CSV/TSV/etc.

--target.datetimeformattz

Format used to write ‘datetimetz’ values

--target.datetimeformat

Format used to write ‘datetime’ values

--target.no_compression

--target.compression

Possible choices: GZIP, BZIP, LZO

Compression type of the file.

--target.no_quoting

--target.quoting

Possible choices: all, minimal, nonnumeric

How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.

--target.no_escape

--target.escape

Possible choices:

Character used to escape strings

--target.encoding

Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252

Text encoding of file

--target.dateformat

Format used to write ‘date’ values

--target.timeonlyformat

Format used to write ‘time’ values

--target.no_doublequote

--target.doublequote

Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.

--target.no_header_row

--target.header_row

True if a header row is provided in the delimited files.

--target.quotechar

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

--target.record_terminator

String used to close out individual rows of data.

--target.field_delimiter

Character used between fields.

table2spectrum

Copy from table to spectrum

mvrec table2spectrum [-h]
                     [--target.spectrum_base_url TARGET.SPECTRUM_BASE_URL]
                     [--target.spectrum_rdir_url TARGET.SPECTRUM_RDIR_URL]
                     [--target.existing_table {delete_and_overwrite,truncate_and_overwrite,drop_and_recreate,append}]
                     source.db_name source.schema_name source.table_name
                     target.schema_name target.table_name target.db_name

Positional Arguments

source.db_name

SQLAlchemy database engine to pull data from.

source.schema_name

Schema name of a table to get data from.

source.table_name

Table name of a table to get data from.

target.schema_name

Schema name of a table to write data to.

target.table_name

Table name of a table to write data to.

target.db_name

SQLAlchemy database engine to write data to.

Named Arguments

--target.spectrum_base_url

Root S3 URL under which a simple directory structure will be created for files to be stored, if spectrum_rdir_url is not specified. Note that when using the mover CLI, db-facts may be used to provide a default.

--target.spectrum_rdir_url

S3 URL where a records directory with files will be stored; otherwise, use db-facts default if exists. If this is not specified, spectrum_base_url must be.

--target.existing_table

Possible choices: delete_and_overwrite, truncate_and_overwrite, drop_and_recreate, append

When loading into a database table, controls how any existing table found will be handled. This must be a records_mover.records.ExistingTableHandling object.

Default: “delete_and_overwrite”

gsheet2gsheet

Copy from gsheet to gsheet

mvrec gsheet2gsheet [-h]
                    [--source.out_of_band_column_headers [SOURCE.OUT_OF_BAND_COLUMN_HEADERS ...]]
                    source.spreadsheet_id source.sheet_name_or_range
                    source.gcp_creds_name target.spreadsheet_id
                    target.sheet_name target.gcp_creds_name

Positional Arguments

source.spreadsheet_id

This is the xyz in https://docs.google.com/spreadsheets/d/xyz/edit?ts=5be5b383#gid=abc

source.sheet_name_or_range

This is the label of the particular tab within the Google Sheets spreadsheet where the data should go, or a valid Google Sheets-style range formula

source.gcp_creds_name

This is an object representing Google Cloud Platform access credentials.

target.spreadsheet_id

This is the xyz in https://docs.google.com/spreadsheets/d/xyz/edit?ts=5be5b383#gid=abc

target.sheet_name

This is the label of the particular tab within the Google Sheets spreadsheet where the data should go.

target.gcp_creds_name

Credentials object for Google Cloud Platform access.

Named Arguments

--source.out_of_band_column_headers

If provided, we’ll use these column names instead of the first row of the spreadsheet. If set, the first row will be treated as data.

gsheet2table

Copy from gsheet to table

mvrec gsheet2table [-h]
                   [--source.out_of_band_column_headers [SOURCE.OUT_OF_BAND_COLUMN_HEADERS ...]]
                   [--target.existing_table {delete_and_overwrite,truncate_and_overwrite,drop_and_recreate,append}]
                   [--target.drop_and_recreate_on_load_error]
                   source.spreadsheet_id source.sheet_name_or_range
                   source.gcp_creds_name target.db_name target.schema_name
                   target.table_name

Positional Arguments

source.spreadsheet_id

This is the xyz in https://docs.google.com/spreadsheets/d/xyz/edit?ts=5be5b383#gid=abc

source.sheet_name_or_range

This is the label of the particular tab within the Google Sheets spreadsheet where the data should go, or a valid Google Sheets-style range formula

source.gcp_creds_name

This is an object representing Google Cloud Platform access credentials.

target.db_name

SQLAlchemy database engine to write data to.

target.schema_name

Schema name of a table to write data to.

target.table_name

Table name of a table to write data to.

Named Arguments

--source.out_of_band_column_headers

If provided, we’ll use these column names instead of the first row of the spreadsheet. If set, the first row will be treated as data.

--target.existing_table

Possible choices: delete_and_overwrite, truncate_and_overwrite, drop_and_recreate, append

When loading into a database table, controls how any existing table found will be handled. This must be a records_mover.records.ExistingTableHandling object.

Default: “delete_and_overwrite”

--target.drop_and_recreate_on_load_error

If True, table load errors will attempt to be addressed by dropping the target table and reloading the incoming data.

Default: False

gsheet2recordsdir

Copy from gsheet to recordsdir

mvrec gsheet2recordsdir [-h]
                        [--source.out_of_band_column_headers [SOURCE.OUT_OF_BAND_COLUMN_HEADERS ...]]
                        [--target.variant TARGET.VARIANT]
                        [--target.format {avro,delimited,parquet}]
                        [--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
                        [--target.datetimeformat TARGET.DATETIMEFORMAT]
                        [--target.no_compression]
                        [--target.compression {GZIP,BZIP,LZO}]
                        [--target.no_quoting]
                        [--target.quoting {all,minimal,nonnumeric}]
                        [--target.no_escape] [--target.escape {\}]
                        [--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
                        [--target.dateformat TARGET.DATEFORMAT]
                        [--target.timeonlyformat TARGET.TIMEONLYFORMAT]
                        [--target.no_doublequote] [--target.doublequote]
                        [--target.no_header_row] [--target.header_row]
                        [--target.quotechar TARGET.QUOTECHAR]
                        [--target.record_terminator TARGET.RECORD-TERMINATOR]
                        [--target.field_delimiter TARGET.FIELD-DELIMITER]
                        source.spreadsheet_id source.sheet_name_or_range
                        source.gcp_creds_name target.output_url

Positional Arguments

source.spreadsheet_id

This is the xyz in https://docs.google.com/spreadsheets/d/xyz/edit?ts=5be5b383#gid=abc

source.sheet_name_or_range

This is the label of the particular tab within the Google Sheets spreadsheet where the data should go, or a valid Google Sheets-style range formula

source.gcp_creds_name

This is an object representing Google Cloud Platform access credentials.

target.output_url

Location to write the records directory. Must be a URL format understood by the records_mover.url library, and must be a directory URL that ends with a ‘/’.

Named Arguments

--source.out_of_band_column_headers

If provided, we’ll use these column names instead of the first row of the spreadsheet. If set, the first row will be treated as data.

--target.variant

Records format variant - valid for ‘delimited’ records format type

--target.format

Possible choices: avro, delimited, parquet

Records format type. Note that ‘delimited’ includes CSV/TSV/etc.

--target.datetimeformattz

Format used to write ‘datetimetz’ values

--target.datetimeformat

Format used to write ‘datetime’ values

--target.no_compression

--target.compression

Possible choices: GZIP, BZIP, LZO

Compression type of the file.

--target.no_quoting

--target.quoting

Possible choices: all, minimal, nonnumeric

How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.

--target.no_escape

--target.escape

Possible choices:

Character used to escape strings

--target.encoding

Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252

Text encoding of file

--target.dateformat

Format used to write ‘date’ values

--target.timeonlyformat

Format used to write ‘time’ values

--target.no_doublequote

--target.doublequote

Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.

--target.no_header_row

--target.header_row

True if a header row is provided in the delimited files.

--target.quotechar

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

--target.record_terminator

String used to close out individual rows of data.

--target.field_delimiter

Character used between fields.

gsheet2url

Copy from gsheet to url

mvrec gsheet2url [-h]
                 [--source.out_of_band_column_headers [SOURCE.OUT_OF_BAND_COLUMN_HEADERS ...]]
                 [--target.variant TARGET.VARIANT]
                 [--target.format {avro,delimited,parquet}]
                 [--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
                 [--target.datetimeformat TARGET.DATETIMEFORMAT]
                 [--target.no_compression]
                 [--target.compression {GZIP,BZIP,LZO}] [--target.no_quoting]
                 [--target.quoting {all,minimal,nonnumeric}]
                 [--target.no_escape] [--target.escape {\}]
                 [--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
                 [--target.dateformat TARGET.DATEFORMAT]
                 [--target.timeonlyformat TARGET.TIMEONLYFORMAT]
                 [--target.no_doublequote] [--target.doublequote]
                 [--target.no_header_row] [--target.header_row]
                 [--target.quotechar TARGET.QUOTECHAR]
                 [--target.record_terminator TARGET.RECORD-TERMINATOR]
                 [--target.field_delimiter TARGET.FIELD-DELIMITER]
                 source.spreadsheet_id source.sheet_name_or_range
                 source.gcp_creds_name target.output_url

Positional Arguments

source.spreadsheet_id

This is the xyz in https://docs.google.com/spreadsheets/d/xyz/edit?ts=5be5b383#gid=abc

source.sheet_name_or_range

This is the label of the particular tab within the Google Sheets spreadsheet where the data should go, or a valid Google Sheets-style range formula

source.gcp_creds_name

This is an object representing Google Cloud Platform access credentials.

target.output_url

Location of the data file to write. Must be a URL format understood by the records_mover.url library corresponding to a file, not a directory (i.e., not ending with a ‘/’)

Named Arguments

--source.out_of_band_column_headers

If provided, we’ll use these column names instead of the first row of the spreadsheet. If set, the first row will be treated as data.

--target.variant

Records format variant - valid for ‘delimited’ records format type

--target.format

Possible choices: avro, delimited, parquet

Records format type. Note that ‘delimited’ includes CSV/TSV/etc.

--target.datetimeformattz

Format used to write ‘datetimetz’ values

--target.datetimeformat

Format used to write ‘datetime’ values

--target.no_compression

--target.compression

Possible choices: GZIP, BZIP, LZO

Compression type of the file.

--target.no_quoting

--target.quoting

Possible choices: all, minimal, nonnumeric

How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.

--target.no_escape

--target.escape

Possible choices:

Character used to escape strings

--target.encoding

Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252

Text encoding of file

--target.dateformat

Format used to write ‘date’ values

--target.timeonlyformat

Format used to write ‘time’ values

--target.no_doublequote

--target.doublequote

Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.

--target.no_header_row

--target.header_row

True if a header row is provided in the delimited files.

--target.quotechar

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

--target.record_terminator

String used to close out individual rows of data.

--target.field_delimiter

Character used between fields.

gsheet2file

Copy from gsheet to file

mvrec gsheet2file [-h]
                  [--source.out_of_band_column_headers [SOURCE.OUT_OF_BAND_COLUMN_HEADERS ...]]
                  [--target.variant TARGET.VARIANT]
                  [--target.format {avro,delimited,parquet}]
                  [--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
                  [--target.datetimeformat TARGET.DATETIMEFORMAT]
                  [--target.no_compression]
                  [--target.compression {GZIP,BZIP,LZO}] [--target.no_quoting]
                  [--target.quoting {all,minimal,nonnumeric}]
                  [--target.no_escape] [--target.escape {\}]
                  [--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
                  [--target.dateformat TARGET.DATEFORMAT]
                  [--target.timeonlyformat TARGET.TIMEONLYFORMAT]
                  [--target.no_doublequote] [--target.doublequote]
                  [--target.no_header_row] [--target.header_row]
                  [--target.quotechar TARGET.QUOTECHAR]
                  [--target.record_terminator TARGET.RECORD-TERMINATOR]
                  [--target.field_delimiter TARGET.FIELD-DELIMITER]
                  source.spreadsheet_id source.sheet_name_or_range
                  source.gcp_creds_name target.filename

Positional Arguments

source.spreadsheet_id

This is the xyz in https://docs.google.com/spreadsheets/d/xyz/edit?ts=5be5b383#gid=abc

source.sheet_name_or_range

This is the label of the particular tab within the Google Sheets spreadsheet where the data should go, or a valid Google Sheets-style range formula

source.gcp_creds_name

This is an object representing Google Cloud Platform access credentials.

target.filename

File path (relative or absolute) of the data file to unload to.

Named Arguments

--source.out_of_band_column_headers

If provided, we’ll use these column names instead of the first row of the spreadsheet. If set, the first row will be treated as data.

--target.variant

Records format variant - valid for ‘delimited’ records format type

--target.format

Possible choices: avro, delimited, parquet

Records format type. Note that ‘delimited’ includes CSV/TSV/etc.

--target.datetimeformattz

Format used to write ‘datetimetz’ values

--target.datetimeformat

Format used to write ‘datetime’ values

--target.no_compression

--target.compression

Possible choices: GZIP, BZIP, LZO

Compression type of the file.

--target.no_quoting

--target.quoting

Possible choices: all, minimal, nonnumeric

How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.

--target.no_escape

--target.escape

Possible choices:

Character used to escape strings

--target.encoding

Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252

Text encoding of file

--target.dateformat

Format used to write ‘date’ values

--target.timeonlyformat

Format used to write ‘time’ values

--target.no_doublequote

--target.doublequote

Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.

--target.no_header_row

--target.header_row

True if a header row is provided in the delimited files.

--target.quotechar

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

--target.record_terminator

String used to close out individual rows of data.

--target.field_delimiter

Character used between fields.

gsheet2spectrum

Copy from gsheet to spectrum

mvrec gsheet2spectrum [-h]
                      [--source.out_of_band_column_headers [SOURCE.OUT_OF_BAND_COLUMN_HEADERS ...]]
                      [--target.spectrum_base_url TARGET.SPECTRUM_BASE_URL]
                      [--target.spectrum_rdir_url TARGET.SPECTRUM_RDIR_URL]
                      [--target.existing_table {delete_and_overwrite,truncate_and_overwrite,drop_and_recreate,append}]
                      source.spreadsheet_id source.sheet_name_or_range
                      source.gcp_creds_name target.schema_name
                      target.table_name target.db_name

Positional Arguments

source.spreadsheet_id

This is the xyz in https://docs.google.com/spreadsheets/d/xyz/edit?ts=5be5b383#gid=abc

source.sheet_name_or_range

This is the label of the particular tab within the Google Sheets spreadsheet where the data should go, or a valid Google Sheets-style range formula

source.gcp_creds_name

This is an object representing Google Cloud Platform access credentials.

target.schema_name

Schema name of a table to write data to.

target.table_name

Table name of a table to write data to.

target.db_name

SQLAlchemy database engine to write data to.

Named Arguments

--source.out_of_band_column_headers

If provided, we’ll use these column names instead of the first row of the spreadsheet. If set, the first row will be treated as data.

--target.spectrum_base_url

Root S3 URL under which a simple directory structure will be created for files to be stored, if spectrum_rdir_url is not specified. Note that when using the mover CLI, db-facts may be used to provide a default.

--target.spectrum_rdir_url

S3 URL where a records directory with files will be stored; otherwise, use db-facts default if exists. If this is not specified, spectrum_base_url must be.

--target.existing_table

Possible choices: delete_and_overwrite, truncate_and_overwrite, drop_and_recreate, append

When loading into a database table, controls how any existing table found will be handled. This must be a records_mover.records.ExistingTableHandling object.

Default: “delete_and_overwrite”

recordsdir2gsheet

Copy from recordsdir to gsheet

mvrec recordsdir2gsheet [-h] [--no_source.fail_if_dont_understand]
                        source.url target.spreadsheet_id target.sheet_name
                        target.gcp_creds_name

Positional Arguments

source.url

Location of the records directory. Must be a URL format understood by the records_mover.url library, and must be a directory URL that ends with a ‘/’.

target.spreadsheet_id

This is the xyz in https://docs.google.com/spreadsheets/d/xyz/edit?ts=5be5b383#gid=abc

target.sheet_name

This is the label of the particular tab within the Google Sheets spreadsheet where the data should go.

target.gcp_creds_name

Credentials object for Google Cloud Platform access.

Named Arguments

--no_source.fail_if_dont_understand

If True, and a part of the RecordsFormat is not understood while processing, then immediately fail and raise an exception. Otherwise, ignore the misunderstood instruction (e.g., ignore the hint, assume default variant, etc etc)

Default: True

recordsdir2table

Copy from recordsdir to table

mvrec recordsdir2table [-h] [--no_source.fail_if_dont_understand]
                       [--target.existing_table {delete_and_overwrite,truncate_and_overwrite,drop_and_recreate,append}]
                       [--target.drop_and_recreate_on_load_error]
                       source.url target.db_name target.schema_name
                       target.table_name

Positional Arguments

source.url

Location of the records directory. Must be a URL format understood by the records_mover.url library, and must be a directory URL that ends with a ‘/’.

target.db_name

SQLAlchemy database engine to write data to.

target.schema_name

Schema name of a table to write data to.

target.table_name

Table name of a table to write data to.

Named Arguments

--no_source.fail_if_dont_understand

If True, and a part of the RecordsFormat is not understood while processing, then immediately fail and raise an exception. Otherwise, ignore the misunderstood instruction (e.g., ignore the hint, assume default variant, etc etc)

Default: True

--target.existing_table

Possible choices: delete_and_overwrite, truncate_and_overwrite, drop_and_recreate, append

When loading into a database table, controls how any existing table found will be handled. This must be a records_mover.records.ExistingTableHandling object.

Default: “delete_and_overwrite”

--target.drop_and_recreate_on_load_error

If True, table load errors will attempt to be addressed by dropping the target table and reloading the incoming data.

Default: False

recordsdir2recordsdir

Copy from recordsdir to recordsdir

mvrec recordsdir2recordsdir [-h] [--no_source.fail_if_dont_understand]
                            [--target.variant TARGET.VARIANT]
                            [--target.format {avro,delimited,parquet}]
                            [--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
                            [--target.datetimeformat TARGET.DATETIMEFORMAT]
                            [--target.no_compression]
                            [--target.compression {GZIP,BZIP,LZO}]
                            [--target.no_quoting]
                            [--target.quoting {all,minimal,nonnumeric}]
                            [--target.no_escape] [--target.escape {\}]
                            [--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
                            [--target.dateformat TARGET.DATEFORMAT]
                            [--target.timeonlyformat TARGET.TIMEONLYFORMAT]
                            [--target.no_doublequote] [--target.doublequote]
                            [--target.no_header_row] [--target.header_row]
                            [--target.quotechar TARGET.QUOTECHAR]
                            [--target.record_terminator TARGET.RECORD-TERMINATOR]
                            [--target.field_delimiter TARGET.FIELD-DELIMITER]
                            source.url target.output_url

Positional Arguments

source.url

Location of the records directory. Must be a URL format understood by the records_mover.url library, and must be a directory URL that ends with a ‘/’.

target.output_url

Location to write the records directory. Must be a URL format understood by the records_mover.url library, and must be a directory URL that ends with a ‘/’.

Named Arguments

--no_source.fail_if_dont_understand

If True, and a part of the RecordsFormat is not understood while processing, then immediately fail and raise an exception. Otherwise, ignore the misunderstood instruction (e.g., ignore the hint, assume default variant, etc etc)

Default: True

--target.variant

Records format variant - valid for ‘delimited’ records format type

--target.format

Possible choices: avro, delimited, parquet

Records format type. Note that ‘delimited’ includes CSV/TSV/etc.

--target.datetimeformattz

Format used to write ‘datetimetz’ values

--target.datetimeformat

Format used to write ‘datetime’ values

--target.no_compression

--target.compression

Possible choices: GZIP, BZIP, LZO

Compression type of the file.

--target.no_quoting

--target.quoting

Possible choices: all, minimal, nonnumeric

How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.

--target.no_escape

--target.escape

Possible choices:

Character used to escape strings

--target.encoding

Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252

Text encoding of file

--target.dateformat

Format used to write ‘date’ values

--target.timeonlyformat

Format used to write ‘time’ values

--target.no_doublequote

--target.doublequote

Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.

--target.no_header_row

--target.header_row

True if a header row is provided in the delimited files.

--target.quotechar

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

--target.record_terminator

String used to close out individual rows of data.

--target.field_delimiter

Character used between fields.

recordsdir2url

Copy from recordsdir to url

mvrec recordsdir2url [-h] [--no_source.fail_if_dont_understand]
                     [--target.variant TARGET.VARIANT]
                     [--target.format {avro,delimited,parquet}]
                     [--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
                     [--target.datetimeformat TARGET.DATETIMEFORMAT]
                     [--target.no_compression]
                     [--target.compression {GZIP,BZIP,LZO}]
                     [--target.no_quoting]
                     [--target.quoting {all,minimal,nonnumeric}]
                     [--target.no_escape] [--target.escape {\}]
                     [--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
                     [--target.dateformat TARGET.DATEFORMAT]
                     [--target.timeonlyformat TARGET.TIMEONLYFORMAT]
                     [--target.no_doublequote] [--target.doublequote]
                     [--target.no_header_row] [--target.header_row]
                     [--target.quotechar TARGET.QUOTECHAR]
                     [--target.record_terminator TARGET.RECORD-TERMINATOR]
                     [--target.field_delimiter TARGET.FIELD-DELIMITER]
                     source.url target.output_url

Positional Arguments

source.url

Location of the records directory. Must be a URL format understood by the records_mover.url library, and must be a directory URL that ends with a ‘/’.

target.output_url

Location of the data file to write. Must be a URL format understood by the records_mover.url library corresponding to a file, not a directory (i.e., not ending with a ‘/’)

Named Arguments

--no_source.fail_if_dont_understand

If True, and a part of the RecordsFormat is not understood while processing, then immediately fail and raise an exception. Otherwise, ignore the misunderstood instruction (e.g., ignore the hint, assume default variant, etc etc)

Default: True

--target.variant

Records format variant - valid for ‘delimited’ records format type

--target.format

Possible choices: avro, delimited, parquet

Records format type. Note that ‘delimited’ includes CSV/TSV/etc.

--target.datetimeformattz

Format used to write ‘datetimetz’ values

--target.datetimeformat

Format used to write ‘datetime’ values

--target.no_compression

--target.compression

Possible choices: GZIP, BZIP, LZO

Compression type of the file.

--target.no_quoting

--target.quoting

Possible choices: all, minimal, nonnumeric

How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.

--target.no_escape

--target.escape

Possible choices:

Character used to escape strings

--target.encoding

Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252

Text encoding of file

--target.dateformat

Format used to write ‘date’ values

--target.timeonlyformat

Format used to write ‘time’ values

--target.no_doublequote

--target.doublequote

Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.

--target.no_header_row

--target.header_row

True if a header row is provided in the delimited files.

--target.quotechar

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

--target.record_terminator

String used to close out individual rows of data.

--target.field_delimiter

Character used between fields.

recordsdir2file

Copy from recordsdir to file

mvrec recordsdir2file [-h] [--no_source.fail_if_dont_understand]
                      [--target.variant TARGET.VARIANT]
                      [--target.format {avro,delimited,parquet}]
                      [--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
                      [--target.datetimeformat TARGET.DATETIMEFORMAT]
                      [--target.no_compression]
                      [--target.compression {GZIP,BZIP,LZO}]
                      [--target.no_quoting]
                      [--target.quoting {all,minimal,nonnumeric}]
                      [--target.no_escape] [--target.escape {\}]
                      [--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
                      [--target.dateformat TARGET.DATEFORMAT]
                      [--target.timeonlyformat TARGET.TIMEONLYFORMAT]
                      [--target.no_doublequote] [--target.doublequote]
                      [--target.no_header_row] [--target.header_row]
                      [--target.quotechar TARGET.QUOTECHAR]
                      [--target.record_terminator TARGET.RECORD-TERMINATOR]
                      [--target.field_delimiter TARGET.FIELD-DELIMITER]
                      source.url target.filename

Positional Arguments

source.url

Location of the records directory. Must be a URL format understood by the records_mover.url library, and must be a directory URL that ends with a ‘/’.

target.filename

File path (relative or absolute) of the data file to unload to.

Named Arguments

--no_source.fail_if_dont_understand

If True, and a part of the RecordsFormat is not understood while processing, then immediately fail and raise an exception. Otherwise, ignore the misunderstood instruction (e.g., ignore the hint, assume default variant, etc etc)

Default: True

--target.variant

Records format variant - valid for ‘delimited’ records format type

--target.format

Possible choices: avro, delimited, parquet

Records format type. Note that ‘delimited’ includes CSV/TSV/etc.

--target.datetimeformattz

Format used to write ‘datetimetz’ values

--target.datetimeformat

Format used to write ‘datetime’ values

--target.no_compression

--target.compression

Possible choices: GZIP, BZIP, LZO

Compression type of the file.

--target.no_quoting

--target.quoting

Possible choices: all, minimal, nonnumeric

How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.

--target.no_escape

--target.escape

Possible choices:

Character used to escape strings

--target.encoding

Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252

Text encoding of file

--target.dateformat

Format used to write ‘date’ values

--target.timeonlyformat

Format used to write ‘time’ values

--target.no_doublequote

--target.doublequote

Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.

--target.no_header_row

--target.header_row

True if a header row is provided in the delimited files.

--target.quotechar

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

--target.record_terminator

String used to close out individual rows of data.

--target.field_delimiter

Character used between fields.

recordsdir2spectrum

Copy from recordsdir to spectrum

mvrec recordsdir2spectrum [-h] [--no_source.fail_if_dont_understand]
                          [--target.spectrum_base_url TARGET.SPECTRUM_BASE_URL]
                          [--target.spectrum_rdir_url TARGET.SPECTRUM_RDIR_URL]
                          [--target.existing_table {delete_and_overwrite,truncate_and_overwrite,drop_and_recreate,append}]
                          source.url target.schema_name target.table_name
                          target.db_name

Positional Arguments

source.url

Location of the records directory. Must be a URL format understood by the records_mover.url library, and must be a directory URL that ends with a ‘/’.

target.schema_name

Schema name of a table to write data to.

target.table_name

Table name of a table to write data to.

target.db_name

SQLAlchemy database engine to write data to.

Named Arguments

--no_source.fail_if_dont_understand

If True, and a part of the RecordsFormat is not understood while processing, then immediately fail and raise an exception. Otherwise, ignore the misunderstood instruction (e.g., ignore the hint, assume default variant, etc etc)

Default: True

--target.spectrum_base_url

Root S3 URL under which a simple directory structure will be created for files to be stored, if spectrum_rdir_url is not specified. Note that when using the mover CLI, db-facts may be used to provide a default.

--target.spectrum_rdir_url

S3 URL where a records directory with files will be stored; otherwise, use db-facts default if exists. If this is not specified, spectrum_base_url must be.

--target.existing_table

Possible choices: delete_and_overwrite, truncate_and_overwrite, drop_and_recreate, append

When loading into a database table, controls how any existing table found will be handled. This must be a records_mover.records.ExistingTableHandling object.

Default: “delete_and_overwrite”

url2gsheet

Copy from url to gsheet

mvrec url2gsheet [-h] [--source.variant SOURCE.VARIANT]
                 [--source.format {avro,delimited,parquet}]
                 [--source.datetimeformattz SOURCE.DATETIMEFORMATTZ]
                 [--source.datetimeformat SOURCE.DATETIMEFORMAT]
                 [--source.no_compression]
                 [--source.compression {GZIP,BZIP,LZO}] [--source.no_quoting]
                 [--source.quoting {all,minimal,nonnumeric}]
                 [--source.no_escape] [--source.escape {\}]
                 [--source.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
                 [--source.dateformat SOURCE.DATEFORMAT]
                 [--source.timeonlyformat SOURCE.TIMEONLYFORMAT]
                 [--source.no_doublequote] [--source.doublequote]
                 [--source.no_header_row] [--source.header_row]
                 [--source.quotechar SOURCE.QUOTECHAR]
                 [--source.record_terminator SOURCE.RECORD-TERMINATOR]
                 [--source.field_delimiter SOURCE.FIELD-DELIMITER]
                 source.input_url target.spreadsheet_id target.sheet_name
                 target.gcp_creds_name

Positional Arguments

source.input_url

Location of the data file. Must be a URL format understood by the records_mover.url library.

target.spreadsheet_id

This is the xyz in https://docs.google.com/spreadsheets/d/xyz/edit?ts=5be5b383#gid=abc

target.sheet_name

This is the label of the particular tab within the Google Sheets spreadsheet where the data should go.

target.gcp_creds_name

Credentials object for Google Cloud Platform access.

Named Arguments

--source.variant

Records format variant - valid for ‘delimited’ records format type

--source.format

Possible choices: avro, delimited, parquet

Records format type. Note that ‘delimited’ includes CSV/TSV/etc.

--source.datetimeformattz

Format used to write ‘datetimetz’ values

--source.datetimeformat

Format used to write ‘datetime’ values

--source.no_compression

--source.compression

Possible choices: GZIP, BZIP, LZO

Compression type of the file.

--source.no_quoting

--source.quoting

Possible choices: all, minimal, nonnumeric

How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.

--source.no_escape

--source.escape

Possible choices:

Character used to escape strings

--source.encoding

Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252

Text encoding of file

--source.dateformat

Format used to write ‘date’ values

--source.timeonlyformat

Format used to write ‘time’ values

--source.no_doublequote

--source.doublequote

Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.

--source.no_header_row

--source.header_row

True if a header row is provided in the delimited files.

--source.quotechar

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

--source.record_terminator

String used to close out individual rows of data.

--source.field_delimiter

Character used between fields.

url2table

Copy from url to table

mvrec url2table [-h] [--source.variant SOURCE.VARIANT]
                [--source.format {avro,delimited,parquet}]
                [--source.datetimeformattz SOURCE.DATETIMEFORMATTZ]
                [--source.datetimeformat SOURCE.DATETIMEFORMAT]
                [--source.no_compression]
                [--source.compression {GZIP,BZIP,LZO}] [--source.no_quoting]
                [--source.quoting {all,minimal,nonnumeric}]
                [--source.no_escape] [--source.escape {\}]
                [--source.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
                [--source.dateformat SOURCE.DATEFORMAT]
                [--source.timeonlyformat SOURCE.TIMEONLYFORMAT]
                [--source.no_doublequote] [--source.doublequote]
                [--source.no_header_row] [--source.header_row]
                [--source.quotechar SOURCE.QUOTECHAR]
                [--source.record_terminator SOURCE.RECORD-TERMINATOR]
                [--source.field_delimiter SOURCE.FIELD-DELIMITER]
                [--target.existing_table {delete_and_overwrite,truncate_and_overwrite,drop_and_recreate,append}]
                [--target.drop_and_recreate_on_load_error]
                source.input_url target.db_name target.schema_name
                target.table_name

Positional Arguments

source.input_url

Location of the data file. Must be a URL format understood by the records_mover.url library.

target.db_name

SQLAlchemy database engine to write data to.

target.schema_name

Schema name of a table to write data to.

target.table_name

Table name of a table to write data to.

Named Arguments

--source.variant

Records format variant - valid for ‘delimited’ records format type

--source.format

Possible choices: avro, delimited, parquet

Records format type. Note that ‘delimited’ includes CSV/TSV/etc.

--source.datetimeformattz

Format used to write ‘datetimetz’ values

--source.datetimeformat

Format used to write ‘datetime’ values

--source.no_compression

--source.compression

Possible choices: GZIP, BZIP, LZO

Compression type of the file.

--source.no_quoting

--source.quoting

Possible choices: all, minimal, nonnumeric

How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.

--source.no_escape

--source.escape

Possible choices:

Character used to escape strings

--source.encoding

Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252

Text encoding of file

--source.dateformat

Format used to write ‘date’ values

--source.timeonlyformat

Format used to write ‘time’ values

--source.no_doublequote

--source.doublequote

Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.

--source.no_header_row

--source.header_row

True if a header row is provided in the delimited files.

--source.quotechar

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

--source.record_terminator

String used to close out individual rows of data.

--source.field_delimiter

Character used between fields.

--target.existing_table

Possible choices: delete_and_overwrite, truncate_and_overwrite, drop_and_recreate, append

When loading into a database table, controls how any existing table found will be handled. This must be a records_mover.records.ExistingTableHandling object.

Default: “delete_and_overwrite”

--target.drop_and_recreate_on_load_error

If True, table load errors will attempt to be addressed by dropping the target table and reloading the incoming data.

Default: False

url2recordsdir

Copy from url to recordsdir

mvrec url2recordsdir [-h] [--source.variant SOURCE.VARIANT]
                     [--source.format {avro,delimited,parquet}]
                     [--source.datetimeformattz SOURCE.DATETIMEFORMATTZ]
                     [--source.datetimeformat SOURCE.DATETIMEFORMAT]
                     [--source.no_compression]
                     [--source.compression {GZIP,BZIP,LZO}]
                     [--source.no_quoting]
                     [--source.quoting {all,minimal,nonnumeric}]
                     [--source.no_escape] [--source.escape {\}]
                     [--source.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
                     [--source.dateformat SOURCE.DATEFORMAT]
                     [--source.timeonlyformat SOURCE.TIMEONLYFORMAT]
                     [--source.no_doublequote] [--source.doublequote]
                     [--source.no_header_row] [--source.header_row]
                     [--source.quotechar SOURCE.QUOTECHAR]
                     [--source.record_terminator SOURCE.RECORD-TERMINATOR]
                     [--source.field_delimiter SOURCE.FIELD-DELIMITER]
                     [--target.variant TARGET.VARIANT]
                     [--target.format {avro,delimited,parquet}]
                     [--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
                     [--target.datetimeformat TARGET.DATETIMEFORMAT]
                     [--target.no_compression]
                     [--target.compression {GZIP,BZIP,LZO}]
                     [--target.no_quoting]
                     [--target.quoting {all,minimal,nonnumeric}]
                     [--target.no_escape] [--target.escape {\}]
                     [--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
                     [--target.dateformat TARGET.DATEFORMAT]
                     [--target.timeonlyformat TARGET.TIMEONLYFORMAT]
                     [--target.no_doublequote] [--target.doublequote]
                     [--target.no_header_row] [--target.header_row]
                     [--target.quotechar TARGET.QUOTECHAR]
                     [--target.record_terminator TARGET.RECORD-TERMINATOR]
                     [--target.field_delimiter TARGET.FIELD-DELIMITER]
                     source.input_url target.output_url

Positional Arguments

source.input_url

Location of the data file. Must be a URL format understood by the records_mover.url library.

target.output_url

Location to write the records directory. Must be a URL format understood by the records_mover.url library, and must be a directory URL that ends with a ‘/’.

Named Arguments

--source.variant

Records format variant - valid for ‘delimited’ records format type

--source.format

Possible choices: avro, delimited, parquet

Records format type. Note that ‘delimited’ includes CSV/TSV/etc.

--source.datetimeformattz

Format used to write ‘datetimetz’ values

--source.datetimeformat

Format used to write ‘datetime’ values

--source.no_compression

--source.compression

Possible choices: GZIP, BZIP, LZO

Compression type of the file.

--source.no_quoting

--source.quoting

Possible choices: all, minimal, nonnumeric

How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.

--source.no_escape

--source.escape

Possible choices:

Character used to escape strings

--source.encoding

Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252

Text encoding of file

--source.dateformat

Format used to write ‘date’ values

--source.timeonlyformat

Format used to write ‘time’ values

--source.no_doublequote

--source.doublequote

Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.

--source.no_header_row

--source.header_row

True if a header row is provided in the delimited files.

--source.quotechar

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

--source.record_terminator

String used to close out individual rows of data.

--source.field_delimiter

Character used between fields.

--target.variant

Records format variant - valid for ‘delimited’ records format type

--target.format

Possible choices: avro, delimited, parquet

Records format type. Note that ‘delimited’ includes CSV/TSV/etc.

--target.datetimeformattz

Format used to write ‘datetimetz’ values

--target.datetimeformat

Format used to write ‘datetime’ values

--target.no_compression

--target.compression

Possible choices: GZIP, BZIP, LZO

Compression type of the file.

--target.no_quoting

--target.quoting

Possible choices: all, minimal, nonnumeric

How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.

--target.no_escape

--target.escape

Possible choices:

Character used to escape strings

--target.encoding

Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252

Text encoding of file

--target.dateformat

Format used to write ‘date’ values

--target.timeonlyformat

Format used to write ‘time’ values

--target.no_doublequote

--target.doublequote

Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.

--target.no_header_row

--target.header_row

True if a header row is provided in the delimited files.

--target.quotechar

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

--target.record_terminator

String used to close out individual rows of data.

--target.field_delimiter

Character used between fields.

url2url

Copy from url to url

mvrec url2url [-h] [--source.variant SOURCE.VARIANT]
              [--source.format {avro,delimited,parquet}]
              [--source.datetimeformattz SOURCE.DATETIMEFORMATTZ]
              [--source.datetimeformat SOURCE.DATETIMEFORMAT]
              [--source.no_compression] [--source.compression {GZIP,BZIP,LZO}]
              [--source.no_quoting]
              [--source.quoting {all,minimal,nonnumeric}] [--source.no_escape]
              [--source.escape {\}]
              [--source.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
              [--source.dateformat SOURCE.DATEFORMAT]
              [--source.timeonlyformat SOURCE.TIMEONLYFORMAT]
              [--source.no_doublequote] [--source.doublequote]
              [--source.no_header_row] [--source.header_row]
              [--source.quotechar SOURCE.QUOTECHAR]
              [--source.record_terminator SOURCE.RECORD-TERMINATOR]
              [--source.field_delimiter SOURCE.FIELD-DELIMITER]
              [--target.variant TARGET.VARIANT]
              [--target.format {avro,delimited,parquet}]
              [--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
              [--target.datetimeformat TARGET.DATETIMEFORMAT]
              [--target.no_compression] [--target.compression {GZIP,BZIP,LZO}]
              [--target.no_quoting]
              [--target.quoting {all,minimal,nonnumeric}] [--target.no_escape]
              [--target.escape {\}]
              [--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
              [--target.dateformat TARGET.DATEFORMAT]
              [--target.timeonlyformat TARGET.TIMEONLYFORMAT]
              [--target.no_doublequote] [--target.doublequote]
              [--target.no_header_row] [--target.header_row]
              [--target.quotechar TARGET.QUOTECHAR]
              [--target.record_terminator TARGET.RECORD-TERMINATOR]
              [--target.field_delimiter TARGET.FIELD-DELIMITER]
              source.input_url target.output_url

Positional Arguments

source.input_url

Location of the data file. Must be a URL format understood by the records_mover.url library.

target.output_url

Location of the data file to write. Must be a URL format understood by the records_mover.url library corresponding to a file, not a directory (i.e., not ending with a ‘/’)

Named Arguments

--source.variant

Records format variant - valid for ‘delimited’ records format type

--source.format

Possible choices: avro, delimited, parquet

Records format type. Note that ‘delimited’ includes CSV/TSV/etc.

--source.datetimeformattz

Format used to write ‘datetimetz’ values

--source.datetimeformat

Format used to write ‘datetime’ values

--source.no_compression

--source.compression

Possible choices: GZIP, BZIP, LZO

Compression type of the file.

--source.no_quoting

--source.quoting

Possible choices: all, minimal, nonnumeric

How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.

--source.no_escape

--source.escape

Possible choices:

Character used to escape strings

--source.encoding

Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252

Text encoding of file

--source.dateformat

Format used to write ‘date’ values

--source.timeonlyformat

Format used to write ‘time’ values

--source.no_doublequote

--source.doublequote

Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.

--source.no_header_row

--source.header_row

True if a header row is provided in the delimited files.

--source.quotechar

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

--source.record_terminator

String used to close out individual rows of data.

--source.field_delimiter

Character used between fields.

--target.variant

Records format variant - valid for ‘delimited’ records format type

--target.format

Possible choices: avro, delimited, parquet

Records format type. Note that ‘delimited’ includes CSV/TSV/etc.

--target.datetimeformattz

Format used to write ‘datetimetz’ values

--target.datetimeformat

Format used to write ‘datetime’ values

--target.no_compression

--target.compression

Possible choices: GZIP, BZIP, LZO

Compression type of the file.

--target.no_quoting

--target.quoting

Possible choices: all, minimal, nonnumeric

How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.

--target.no_escape

--target.escape

Possible choices:

Character used to escape strings

--target.encoding

Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252

Text encoding of file

--target.dateformat

Format used to write ‘date’ values

--target.timeonlyformat

Format used to write ‘time’ values

--target.no_doublequote

--target.doublequote

Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.

--target.no_header_row

--target.header_row

True if a header row is provided in the delimited files.

--target.quotechar

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

--target.record_terminator

String used to close out individual rows of data.

--target.field_delimiter

Character used between fields.

url2file

Copy from url to file

mvrec url2file [-h] [--source.variant SOURCE.VARIANT]
               [--source.format {avro,delimited,parquet}]
               [--source.datetimeformattz SOURCE.DATETIMEFORMATTZ]
               [--source.datetimeformat SOURCE.DATETIMEFORMAT]
               [--source.no_compression]
               [--source.compression {GZIP,BZIP,LZO}] [--source.no_quoting]
               [--source.quoting {all,minimal,nonnumeric}]
               [--source.no_escape] [--source.escape {\}]
               [--source.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
               [--source.dateformat SOURCE.DATEFORMAT]
               [--source.timeonlyformat SOURCE.TIMEONLYFORMAT]
               [--source.no_doublequote] [--source.doublequote]
               [--source.no_header_row] [--source.header_row]
               [--source.quotechar SOURCE.QUOTECHAR]
               [--source.record_terminator SOURCE.RECORD-TERMINATOR]
               [--source.field_delimiter SOURCE.FIELD-DELIMITER]
               [--target.variant TARGET.VARIANT]
               [--target.format {avro,delimited,parquet}]
               [--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
               [--target.datetimeformat TARGET.DATETIMEFORMAT]
               [--target.no_compression]
               [--target.compression {GZIP,BZIP,LZO}] [--target.no_quoting]
               [--target.quoting {all,minimal,nonnumeric}]
               [--target.no_escape] [--target.escape {\}]
               [--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
               [--target.dateformat TARGET.DATEFORMAT]
               [--target.timeonlyformat TARGET.TIMEONLYFORMAT]
               [--target.no_doublequote] [--target.doublequote]
               [--target.no_header_row] [--target.header_row]
               [--target.quotechar TARGET.QUOTECHAR]
               [--target.record_terminator TARGET.RECORD-TERMINATOR]
               [--target.field_delimiter TARGET.FIELD-DELIMITER]
               source.input_url target.filename

Positional Arguments

source.input_url

Location of the data file. Must be a URL format understood by the records_mover.url library.

target.filename

File path (relative or absolute) of the data file to unload to.

Named Arguments

--source.variant

Records format variant - valid for ‘delimited’ records format type

--source.format

Possible choices: avro, delimited, parquet

Records format type. Note that ‘delimited’ includes CSV/TSV/etc.

--source.datetimeformattz

Format used to write ‘datetimetz’ values

--source.datetimeformat

Format used to write ‘datetime’ values

--source.no_compression

--source.compression

Possible choices: GZIP, BZIP, LZO

Compression type of the file.

--source.no_quoting

--source.quoting

Possible choices: all, minimal, nonnumeric

How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.

--source.no_escape

--source.escape

Possible choices:

Character used to escape strings

--source.encoding

Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252

Text encoding of file

--source.dateformat

Format used to write ‘date’ values

--source.timeonlyformat

Format used to write ‘time’ values

--source.no_doublequote

--source.doublequote

Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.

--source.no_header_row

--source.header_row

True if a header row is provided in the delimited files.

--source.quotechar

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

--source.record_terminator

String used to close out individual rows of data.

--source.field_delimiter

Character used between fields.

--target.variant

Records format variant - valid for ‘delimited’ records format type

--target.format

Possible choices: avro, delimited, parquet

Records format type. Note that ‘delimited’ includes CSV/TSV/etc.

--target.datetimeformattz

Format used to write ‘datetimetz’ values

--target.datetimeformat

Format used to write ‘datetime’ values

--target.no_compression

--target.compression

Possible choices: GZIP, BZIP, LZO

Compression type of the file.

--target.no_quoting

--target.quoting

Possible choices: all, minimal, nonnumeric

How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.

--target.no_escape

--target.escape

Possible choices:

Character used to escape strings

--target.encoding

Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252

Text encoding of file

--target.dateformat

Format used to write ‘date’ values

--target.timeonlyformat

Format used to write ‘time’ values

--target.no_doublequote

--target.doublequote

Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.

--target.no_header_row

--target.header_row

True if a header row is provided in the delimited files.

--target.quotechar

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

--target.record_terminator

String used to close out individual rows of data.

--target.field_delimiter

Character used between fields.

url2spectrum

Copy from url to spectrum

mvrec url2spectrum [-h] [--source.variant SOURCE.VARIANT]
                   [--source.format {avro,delimited,parquet}]
                   [--source.datetimeformattz SOURCE.DATETIMEFORMATTZ]
                   [--source.datetimeformat SOURCE.DATETIMEFORMAT]
                   [--source.no_compression]
                   [--source.compression {GZIP,BZIP,LZO}]
                   [--source.no_quoting]
                   [--source.quoting {all,minimal,nonnumeric}]
                   [--source.no_escape] [--source.escape {\}]
                   [--source.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
                   [--source.dateformat SOURCE.DATEFORMAT]
                   [--source.timeonlyformat SOURCE.TIMEONLYFORMAT]
                   [--source.no_doublequote] [--source.doublequote]
                   [--source.no_header_row] [--source.header_row]
                   [--source.quotechar SOURCE.QUOTECHAR]
                   [--source.record_terminator SOURCE.RECORD-TERMINATOR]
                   [--source.field_delimiter SOURCE.FIELD-DELIMITER]
                   [--target.spectrum_base_url TARGET.SPECTRUM_BASE_URL]
                   [--target.spectrum_rdir_url TARGET.SPECTRUM_RDIR_URL]
                   [--target.existing_table {delete_and_overwrite,truncate_and_overwrite,drop_and_recreate,append}]
                   source.input_url target.schema_name target.table_name
                   target.db_name

Positional Arguments

source.input_url

Location of the data file. Must be a URL format understood by the records_mover.url library.

target.schema_name

Schema name of a table to write data to.

target.table_name

Table name of a table to write data to.

target.db_name

SQLAlchemy database engine to write data to.

Named Arguments

--source.variant

Records format variant - valid for ‘delimited’ records format type

--source.format

Possible choices: avro, delimited, parquet

Records format type. Note that ‘delimited’ includes CSV/TSV/etc.

--source.datetimeformattz

Format used to write ‘datetimetz’ values

--source.datetimeformat

Format used to write ‘datetime’ values

--source.no_compression

--source.compression

Possible choices: GZIP, BZIP, LZO

Compression type of the file.

--source.no_quoting

--source.quoting

Possible choices: all, minimal, nonnumeric

How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.

--source.no_escape

--source.escape

Possible choices:

Character used to escape strings

--source.encoding

Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252

Text encoding of file

--source.dateformat

Format used to write ‘date’ values

--source.timeonlyformat

Format used to write ‘time’ values

--source.no_doublequote

--source.doublequote

Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.

--source.no_header_row

--source.header_row

True if a header row is provided in the delimited files.

--source.quotechar

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

--source.record_terminator

String used to close out individual rows of data.

--source.field_delimiter

Character used between fields.

--target.spectrum_base_url

Root S3 URL under which a simple directory structure will be created for files to be stored, if spectrum_rdir_url is not specified. Note that when using the mover CLI, db-facts may be used to provide a default.

--target.spectrum_rdir_url

S3 URL where a records directory with files will be stored; otherwise, use db-facts default if exists. If this is not specified, spectrum_base_url must be.

--target.existing_table

Possible choices: delete_and_overwrite, truncate_and_overwrite, drop_and_recreate, append

When loading into a database table, controls how any existing table found will be handled. This must be a records_mover.records.ExistingTableHandling object.

Default: “delete_and_overwrite”

file2gsheet

Copy from file to gsheet

mvrec file2gsheet [-h] [--source.variant SOURCE.VARIANT]
                  [--source.format {avro,delimited,parquet}]
                  [--source.datetimeformattz SOURCE.DATETIMEFORMATTZ]
                  [--source.datetimeformat SOURCE.DATETIMEFORMAT]
                  [--source.no_compression]
                  [--source.compression {GZIP,BZIP,LZO}] [--source.no_quoting]
                  [--source.quoting {all,minimal,nonnumeric}]
                  [--source.no_escape] [--source.escape {\}]
                  [--source.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
                  [--source.dateformat SOURCE.DATEFORMAT]
                  [--source.timeonlyformat SOURCE.TIMEONLYFORMAT]
                  [--source.no_doublequote] [--source.doublequote]
                  [--source.no_header_row] [--source.header_row]
                  [--source.quotechar SOURCE.QUOTECHAR]
                  [--source.record_terminator SOURCE.RECORD-TERMINATOR]
                  [--source.field_delimiter SOURCE.FIELD-DELIMITER]
                  source.filename target.spreadsheet_id target.sheet_name
                  target.gcp_creds_name

Positional Arguments

source.filename

File path (relative or absolute) of the data file to load.

target.spreadsheet_id

This is the xyz in https://docs.google.com/spreadsheets/d/xyz/edit?ts=5be5b383#gid=abc

target.sheet_name

This is the label of the particular tab within the Google Sheets spreadsheet where the data should go.

target.gcp_creds_name

Credentials object for Google Cloud Platform access.

Named Arguments

--source.variant

Records format variant - valid for ‘delimited’ records format type

--source.format

Possible choices: avro, delimited, parquet

Records format type. Note that ‘delimited’ includes CSV/TSV/etc.

--source.datetimeformattz

Format used to write ‘datetimetz’ values

--source.datetimeformat

Format used to write ‘datetime’ values

--source.no_compression

--source.compression

Possible choices: GZIP, BZIP, LZO

Compression type of the file.

--source.no_quoting

--source.quoting

Possible choices: all, minimal, nonnumeric

How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.

--source.no_escape

--source.escape

Possible choices:

Character used to escape strings

--source.encoding

Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252

Text encoding of file

--source.dateformat

Format used to write ‘date’ values

--source.timeonlyformat

Format used to write ‘time’ values

--source.no_doublequote

--source.doublequote

Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.

--source.no_header_row

--source.header_row

True if a header row is provided in the delimited files.

--source.quotechar

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

--source.record_terminator

String used to close out individual rows of data.

--source.field_delimiter

Character used between fields.

file2table

Copy from file to table

mvrec file2table [-h] [--source.variant SOURCE.VARIANT]
                 [--source.format {avro,delimited,parquet}]
                 [--source.datetimeformattz SOURCE.DATETIMEFORMATTZ]
                 [--source.datetimeformat SOURCE.DATETIMEFORMAT]
                 [--source.no_compression]
                 [--source.compression {GZIP,BZIP,LZO}] [--source.no_quoting]
                 [--source.quoting {all,minimal,nonnumeric}]
                 [--source.no_escape] [--source.escape {\}]
                 [--source.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
                 [--source.dateformat SOURCE.DATEFORMAT]
                 [--source.timeonlyformat SOURCE.TIMEONLYFORMAT]
                 [--source.no_doublequote] [--source.doublequote]
                 [--source.no_header_row] [--source.header_row]
                 [--source.quotechar SOURCE.QUOTECHAR]
                 [--source.record_terminator SOURCE.RECORD-TERMINATOR]
                 [--source.field_delimiter SOURCE.FIELD-DELIMITER]
                 [--target.existing_table {delete_and_overwrite,truncate_and_overwrite,drop_and_recreate,append}]
                 [--target.drop_and_recreate_on_load_error]
                 source.filename target.db_name target.schema_name
                 target.table_name

Positional Arguments

source.filename

File path (relative or absolute) of the data file to load.

target.db_name

SQLAlchemy database engine to write data to.

target.schema_name

Schema name of a table to write data to.

target.table_name

Table name of a table to write data to.

Named Arguments

--source.variant

Records format variant - valid for ‘delimited’ records format type

--source.format

Possible choices: avro, delimited, parquet

Records format type. Note that ‘delimited’ includes CSV/TSV/etc.

--source.datetimeformattz

Format used to write ‘datetimetz’ values

--source.datetimeformat

Format used to write ‘datetime’ values

--source.no_compression

--source.compression

Possible choices: GZIP, BZIP, LZO

Compression type of the file.

--source.no_quoting

--source.quoting

Possible choices: all, minimal, nonnumeric

How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.

--source.no_escape

--source.escape

Possible choices:

Character used to escape strings

--source.encoding

Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252

Text encoding of file

--source.dateformat

Format used to write ‘date’ values

--source.timeonlyformat

Format used to write ‘time’ values

--source.no_doublequote

--source.doublequote

Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.

--source.no_header_row

--source.header_row

True if a header row is provided in the delimited files.

--source.quotechar

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

--source.record_terminator

String used to close out individual rows of data.

--source.field_delimiter

Character used between fields.

--target.existing_table

Possible choices: delete_and_overwrite, truncate_and_overwrite, drop_and_recreate, append

When loading into a database table, controls how any existing table found will be handled. This must be a records_mover.records.ExistingTableHandling object.

Default: “delete_and_overwrite”

--target.drop_and_recreate_on_load_error

If True, table load errors will attempt to be addressed by dropping the target table and reloading the incoming data.

Default: False

file2recordsdir

Copy from file to recordsdir

mvrec file2recordsdir [-h] [--source.variant SOURCE.VARIANT]
                      [--source.format {avro,delimited,parquet}]
                      [--source.datetimeformattz SOURCE.DATETIMEFORMATTZ]
                      [--source.datetimeformat SOURCE.DATETIMEFORMAT]
                      [--source.no_compression]
                      [--source.compression {GZIP,BZIP,LZO}]
                      [--source.no_quoting]
                      [--source.quoting {all,minimal,nonnumeric}]
                      [--source.no_escape] [--source.escape {\}]
                      [--source.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
                      [--source.dateformat SOURCE.DATEFORMAT]
                      [--source.timeonlyformat SOURCE.TIMEONLYFORMAT]
                      [--source.no_doublequote] [--source.doublequote]
                      [--source.no_header_row] [--source.header_row]
                      [--source.quotechar SOURCE.QUOTECHAR]
                      [--source.record_terminator SOURCE.RECORD-TERMINATOR]
                      [--source.field_delimiter SOURCE.FIELD-DELIMITER]
                      [--target.variant TARGET.VARIANT]
                      [--target.format {avro,delimited,parquet}]
                      [--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
                      [--target.datetimeformat TARGET.DATETIMEFORMAT]
                      [--target.no_compression]
                      [--target.compression {GZIP,BZIP,LZO}]
                      [--target.no_quoting]
                      [--target.quoting {all,minimal,nonnumeric}]
                      [--target.no_escape] [--target.escape {\}]
                      [--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
                      [--target.dateformat TARGET.DATEFORMAT]
                      [--target.timeonlyformat TARGET.TIMEONLYFORMAT]
                      [--target.no_doublequote] [--target.doublequote]
                      [--target.no_header_row] [--target.header_row]
                      [--target.quotechar TARGET.QUOTECHAR]
                      [--target.record_terminator TARGET.RECORD-TERMINATOR]
                      [--target.field_delimiter TARGET.FIELD-DELIMITER]
                      source.filename target.output_url

Positional Arguments

source.filename

File path (relative or absolute) of the data file to load.

target.output_url

Location to write the records directory. Must be a URL format understood by the records_mover.url library, and must be a directory URL that ends with a ‘/’.

Named Arguments

--source.variant

Records format variant - valid for ‘delimited’ records format type

--source.format

Possible choices: avro, delimited, parquet

Records format type. Note that ‘delimited’ includes CSV/TSV/etc.

--source.datetimeformattz

Format used to write ‘datetimetz’ values

--source.datetimeformat

Format used to write ‘datetime’ values

--source.no_compression

--source.compression

Possible choices: GZIP, BZIP, LZO

Compression type of the file.

--source.no_quoting

--source.quoting

Possible choices: all, minimal, nonnumeric

How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.

--source.no_escape

--source.escape

Possible choices:

Character used to escape strings

--source.encoding

Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252

Text encoding of file

--source.dateformat

Format used to write ‘date’ values

--source.timeonlyformat

Format used to write ‘time’ values

--source.no_doublequote

--source.doublequote

Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.

--source.no_header_row

--source.header_row

True if a header row is provided in the delimited files.

--source.quotechar

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

--source.record_terminator

String used to close out individual rows of data.

--source.field_delimiter

Character used between fields.

--target.variant

Records format variant - valid for ‘delimited’ records format type

--target.format

Possible choices: avro, delimited, parquet

Records format type. Note that ‘delimited’ includes CSV/TSV/etc.

--target.datetimeformattz

Format used to write ‘datetimetz’ values

--target.datetimeformat

Format used to write ‘datetime’ values

--target.no_compression

--target.compression

Possible choices: GZIP, BZIP, LZO

Compression type of the file.

--target.no_quoting

--target.quoting

Possible choices: all, minimal, nonnumeric

How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.

--target.no_escape

--target.escape

Possible choices:

Character used to escape strings

--target.encoding

Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252

Text encoding of file

--target.dateformat

Format used to write ‘date’ values

--target.timeonlyformat

Format used to write ‘time’ values

--target.no_doublequote

--target.doublequote

Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.

--target.no_header_row

--target.header_row

True if a header row is provided in the delimited files.

--target.quotechar

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

--target.record_terminator

String used to close out individual rows of data.

--target.field_delimiter

Character used between fields.

file2url

Copy from file to url

mvrec file2url [-h] [--source.variant SOURCE.VARIANT]
               [--source.format {avro,delimited,parquet}]
               [--source.datetimeformattz SOURCE.DATETIMEFORMATTZ]
               [--source.datetimeformat SOURCE.DATETIMEFORMAT]
               [--source.no_compression]
               [--source.compression {GZIP,BZIP,LZO}] [--source.no_quoting]
               [--source.quoting {all,minimal,nonnumeric}]
               [--source.no_escape] [--source.escape {\}]
               [--source.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
               [--source.dateformat SOURCE.DATEFORMAT]
               [--source.timeonlyformat SOURCE.TIMEONLYFORMAT]
               [--source.no_doublequote] [--source.doublequote]
               [--source.no_header_row] [--source.header_row]
               [--source.quotechar SOURCE.QUOTECHAR]
               [--source.record_terminator SOURCE.RECORD-TERMINATOR]
               [--source.field_delimiter SOURCE.FIELD-DELIMITER]
               [--target.variant TARGET.VARIANT]
               [--target.format {avro,delimited,parquet}]
               [--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
               [--target.datetimeformat TARGET.DATETIMEFORMAT]
               [--target.no_compression]
               [--target.compression {GZIP,BZIP,LZO}] [--target.no_quoting]
               [--target.quoting {all,minimal,nonnumeric}]
               [--target.no_escape] [--target.escape {\}]
               [--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
               [--target.dateformat TARGET.DATEFORMAT]
               [--target.timeonlyformat TARGET.TIMEONLYFORMAT]
               [--target.no_doublequote] [--target.doublequote]
               [--target.no_header_row] [--target.header_row]
               [--target.quotechar TARGET.QUOTECHAR]
               [--target.record_terminator TARGET.RECORD-TERMINATOR]
               [--target.field_delimiter TARGET.FIELD-DELIMITER]
               source.filename target.output_url

Positional Arguments

source.filename

File path (relative or absolute) of the data file to load.

target.output_url

Location of the data file to write. Must be a URL format understood by the records_mover.url library corresponding to a file, not a directory (i.e., not ending with a ‘/’)

Named Arguments

--source.variant

Records format variant - valid for ‘delimited’ records format type

--source.format

Possible choices: avro, delimited, parquet

Records format type. Note that ‘delimited’ includes CSV/TSV/etc.

--source.datetimeformattz

Format used to write ‘datetimetz’ values

--source.datetimeformat

Format used to write ‘datetime’ values

--source.no_compression

--source.compression

Possible choices: GZIP, BZIP, LZO

Compression type of the file.

--source.no_quoting

--source.quoting

Possible choices: all, minimal, nonnumeric

How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.

--source.no_escape

--source.escape

Possible choices:

Character used to escape strings

--source.encoding

Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252

Text encoding of file

--source.dateformat

Format used to write ‘date’ values

--source.timeonlyformat

Format used to write ‘time’ values

--source.no_doublequote

--source.doublequote

Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.

--source.no_header_row

--source.header_row

True if a header row is provided in the delimited files.

--source.quotechar

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

--source.record_terminator

String used to close out individual rows of data.

--source.field_delimiter

Character used between fields.

--target.variant

Records format variant - valid for ‘delimited’ records format type

--target.format

Possible choices: avro, delimited, parquet

Records format type. Note that ‘delimited’ includes CSV/TSV/etc.

--target.datetimeformattz

Format used to write ‘datetimetz’ values

--target.datetimeformat

Format used to write ‘datetime’ values

--target.no_compression

--target.compression

Possible choices: GZIP, BZIP, LZO

Compression type of the file.

--target.no_quoting

--target.quoting

Possible choices: all, minimal, nonnumeric

How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.

--target.no_escape

--target.escape

Possible choices:

Character used to escape strings

--target.encoding

Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252

Text encoding of file

--target.dateformat

Format used to write ‘date’ values

--target.timeonlyformat

Format used to write ‘time’ values

--target.no_doublequote

--target.doublequote

Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.

--target.no_header_row

--target.header_row

True if a header row is provided in the delimited files.

--target.quotechar

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

--target.record_terminator

String used to close out individual rows of data.

--target.field_delimiter

Character used between fields.

file2file

Copy from file to file

mvrec file2file [-h] [--source.variant SOURCE.VARIANT]
                [--source.format {avro,delimited,parquet}]
                [--source.datetimeformattz SOURCE.DATETIMEFORMATTZ]
                [--source.datetimeformat SOURCE.DATETIMEFORMAT]
                [--source.no_compression]
                [--source.compression {GZIP,BZIP,LZO}] [--source.no_quoting]
                [--source.quoting {all,minimal,nonnumeric}]
                [--source.no_escape] [--source.escape {\}]
                [--source.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
                [--source.dateformat SOURCE.DATEFORMAT]
                [--source.timeonlyformat SOURCE.TIMEONLYFORMAT]
                [--source.no_doublequote] [--source.doublequote]
                [--source.no_header_row] [--source.header_row]
                [--source.quotechar SOURCE.QUOTECHAR]
                [--source.record_terminator SOURCE.RECORD-TERMINATOR]
                [--source.field_delimiter SOURCE.FIELD-DELIMITER]
                [--target.variant TARGET.VARIANT]
                [--target.format {avro,delimited,parquet}]
                [--target.datetimeformattz TARGET.DATETIMEFORMATTZ]
                [--target.datetimeformat TARGET.DATETIMEFORMAT]
                [--target.no_compression]
                [--target.compression {GZIP,BZIP,LZO}] [--target.no_quoting]
                [--target.quoting {all,minimal,nonnumeric}]
                [--target.no_escape] [--target.escape {\}]
                [--target.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
                [--target.dateformat TARGET.DATEFORMAT]
                [--target.timeonlyformat TARGET.TIMEONLYFORMAT]
                [--target.no_doublequote] [--target.doublequote]
                [--target.no_header_row] [--target.header_row]
                [--target.quotechar TARGET.QUOTECHAR]
                [--target.record_terminator TARGET.RECORD-TERMINATOR]
                [--target.field_delimiter TARGET.FIELD-DELIMITER]
                source.filename target.filename

Positional Arguments

source.filename

File path (relative or absolute) of the data file to load.

target.filename

File path (relative or absolute) of the data file to unload to.

Named Arguments

--source.variant

Records format variant - valid for ‘delimited’ records format type

--source.format

Possible choices: avro, delimited, parquet

Records format type. Note that ‘delimited’ includes CSV/TSV/etc.

--source.datetimeformattz

Format used to write ‘datetimetz’ values

--source.datetimeformat

Format used to write ‘datetime’ values

--source.no_compression

--source.compression

Possible choices: GZIP, BZIP, LZO

Compression type of the file.

--source.no_quoting

--source.quoting

Possible choices: all, minimal, nonnumeric

How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.

--source.no_escape

--source.escape

Possible choices:

Character used to escape strings

--source.encoding

Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252

Text encoding of file

--source.dateformat

Format used to write ‘date’ values

--source.timeonlyformat

Format used to write ‘time’ values

--source.no_doublequote

--source.doublequote

Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.

--source.no_header_row

--source.header_row

True if a header row is provided in the delimited files.

--source.quotechar

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

--source.record_terminator

String used to close out individual rows of data.

--source.field_delimiter

Character used between fields.

--target.variant

Records format variant - valid for ‘delimited’ records format type

--target.format

Possible choices: avro, delimited, parquet

Records format type. Note that ‘delimited’ includes CSV/TSV/etc.

--target.datetimeformattz

Format used to write ‘datetimetz’ values

--target.datetimeformat

Format used to write ‘datetime’ values

--target.no_compression

--target.compression

Possible choices: GZIP, BZIP, LZO

Compression type of the file.

--target.no_quoting

--target.quoting

Possible choices: all, minimal, nonnumeric

How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.

--target.no_escape

--target.escape

Possible choices:

Character used to escape strings

--target.encoding

Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252

Text encoding of file

--target.dateformat

Format used to write ‘date’ values

--target.timeonlyformat

Format used to write ‘time’ values

--target.no_doublequote

--target.doublequote

Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.

--target.no_header_row

--target.header_row

True if a header row is provided in the delimited files.

--target.quotechar

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

--target.record_terminator

String used to close out individual rows of data.

--target.field_delimiter

Character used between fields.

file2spectrum

Copy from file to spectrum

mvrec file2spectrum [-h] [--source.variant SOURCE.VARIANT]
                    [--source.format {avro,delimited,parquet}]
                    [--source.datetimeformattz SOURCE.DATETIMEFORMATTZ]
                    [--source.datetimeformat SOURCE.DATETIMEFORMAT]
                    [--source.no_compression]
                    [--source.compression {GZIP,BZIP,LZO}]
                    [--source.no_quoting]
                    [--source.quoting {all,minimal,nonnumeric}]
                    [--source.no_escape] [--source.escape {\}]
                    [--source.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
                    [--source.dateformat SOURCE.DATEFORMAT]
                    [--source.timeonlyformat SOURCE.TIMEONLYFORMAT]
                    [--source.no_doublequote] [--source.doublequote]
                    [--source.no_header_row] [--source.header_row]
                    [--source.quotechar SOURCE.QUOTECHAR]
                    [--source.record_terminator SOURCE.RECORD-TERMINATOR]
                    [--source.field_delimiter SOURCE.FIELD-DELIMITER]
                    [--target.spectrum_base_url TARGET.SPECTRUM_BASE_URL]
                    [--target.spectrum_rdir_url TARGET.SPECTRUM_RDIR_URL]
                    [--target.existing_table {delete_and_overwrite,truncate_and_overwrite,drop_and_recreate,append}]
                    source.filename target.schema_name target.table_name
                    target.db_name

Positional Arguments

source.filename

File path (relative or absolute) of the data file to load.

target.schema_name

Schema name of a table to write data to.

target.table_name

Table name of a table to write data to.

target.db_name

SQLAlchemy database engine to write data to.

Named Arguments

--source.variant

Records format variant - valid for ‘delimited’ records format type

--source.format

Possible choices: avro, delimited, parquet

Records format type. Note that ‘delimited’ includes CSV/TSV/etc.

--source.datetimeformattz

Format used to write ‘datetimetz’ values

--source.datetimeformat

Format used to write ‘datetime’ values

--source.no_compression

--source.compression

Possible choices: GZIP, BZIP, LZO

Compression type of the file.

--source.no_quoting

--source.quoting

Possible choices: all, minimal, nonnumeric

How quotes are applied to individual fields. all: quote all fields. minimal: quote only fields that contain ambiguous characters (the delimiter, the escape character, or a line terminator). default: never quote fields.

--source.no_escape

--source.escape

Possible choices:

Character used to escape strings

--source.encoding

Possible choices: UTF8, UTF16, UTF16LE, UTF16BE, UTF16BOM, UTF8BOM, LATIN1, CP1252

Text encoding of file

--source.dateformat

Format used to write ‘date’ values

--source.timeonlyformat

Format used to write ‘time’ values

--source.no_doublequote

--source.doublequote

Controls how instances of quotechar appearing inside a field should themselves be quoted. When True, the character is doubled. When False, the escapechar is used as a prefix to the quotechar.

--source.no_header_row

--source.header_row

True if a header row is provided in the delimited files.

--source.quotechar

A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.

--source.record_terminator

String used to close out individual rows of data.

--source.field_delimiter

Character used between fields.

--target.spectrum_base_url

Root S3 URL under which a simple directory structure will be created for files to be stored, if spectrum_rdir_url is not specified. Note that when using the mover CLI, db-facts may be used to provide a default.

--target.spectrum_rdir_url

S3 URL where a records directory with files will be stored; otherwise, use db-facts default if exists. If this is not specified, spectrum_base_url must be.

--target.existing_table

Possible choices: delete_and_overwrite, truncate_and_overwrite, drop_and_recreate, append

When loading into a database table, controls how any existing table found will be handled. This must be a records_mover.records.ExistingTableHandling object.

Default: “delete_and_overwrite”