airflow_dbt_python.operators

Airflow operators for all dbt commands.

class airflow_dbt_python.operators.dbt.DbtBaseOperator(project_dir=None, profiles_dir=None, profile=None, target=None, state=None, compiled_target=None, cache_selected_only=None, fail_fast=None, single_threaded=None, threads=None, use_experimental_parser=None, vars=None, warn_error=None, debug=None, log_path=None, log_level=None, log_level_file=None, log_format=LogFormat.DEFAULT, log_cache_events=False, quiet=None, no_print=None, record_timing_info=None, defer=None, no_defer=None, partial_parse=False, no_partial_parse=None, introspect=None, no_introspect=None, use_colors=None, no_use_colors=None, static_parser=None, no_static_parser=None, version_check=None, no_version_check=None, write_json=None, write_perf_info=None, send_anonymous_usage_stats=None, no_send_anonymous_usage_stats=None, dbt_conn_id='dbt_conn_id', profiles_conn_id=None, project_conn_id=None, do_xcom_push_artifacts=None, upload_dbt_project=False, delete_before_upload=False, replace_on_upload=False, env_vars=None, **kwargs)[source]

The basic Airflow dbt operator.

Defines how to build an argument list and execute a dbt command. Does not set a command itself, subclasses should set it.

Parameters:

  • project_dir (Optional[Union[str, Path]])

  • profiles_dir (Optional[Union[str, Path]])

  • profile (Optional[str])

  • target (Optional[str])

  • state (Optional[str])

  • compiled_target (Optional[Union[os.PathLike, str, bytes]])

  • cache_selected_only (Optional[bool])

  • fail_fast (Optional[bool])

  • single_threaded (Optional[bool])

  • threads (Optional[int])

  • use_experimental_parser (Optional[bool])

  • vars (Optional[dict[str, str]])

  • warn_error (Optional[bool])

  • debug (Optional[bool])

  • log_path (Optional[str])

  • log_level (Optional[str])

  • log_level_file (Optional[str])

  • log_format (LogFormat)

  • log_cache_events (Optional[bool])

  • quiet (Optional[bool])

  • no_print (Optional[bool])

  • record_timing_info (Optional[str])

  • defer (Optional[bool])

  • no_defer (Optional[bool])

  • partial_parse (Optional[bool])

  • no_partial_parse (Optional[bool])

  • introspect (Optional[bool])

  • no_introspect (Optional[bool])

  • use_colors (Optional[bool])

  • no_use_colors (Optional[bool])

  • static_parser (Optional[bool])

  • no_static_parser (Optional[bool])

  • version_check (Optional[bool])

  • no_version_check (Optional[bool])

  • write_json (Optional[bool])

  • write_perf_info (Optional[bool])

  • send_anonymous_usage_stats (Optional[bool])

  • no_send_anonymous_usage_stats (Optional[bool])

  • dbt_conn_id (Optional[str])

  • profiles_conn_id (Optional[str])

  • project_conn_id (Optional[str])

  • do_xcom_push_artifacts (Optional[list[str]])

  • upload_dbt_project (bool)

  • delete_before_upload (bool)

  • replace_on_upload (bool)

  • env_vars (Optional[Dict[str, Any]])

command

The dbt command to execute.

project_dir

Directory for dbt to look for dbt_profile.yml. Defaults to current directory.

profiles_dir

Directory for dbt to look for profiles.yml. Defaults to ~/.dbt.

profile

Which profile to load. Overrides dbt_profile.yml.

target

Which target to load for the given profile.

vars

Supply variables to the project. Should be a YAML string. Overrides variables defined in dbt_profile.yml.

log_cache_events

Flag to enable logging of cache events.

s3_conn_id

An s3 Airflow connection ID to use when pulling dbt files from s3.

do_xcom_push_artifacts

A list of dbt artifacts to XCom push.

property command: str

Return the current dbt command.

Each subclass of DbtBaseOperator should return its corresponding command.

property dbt_hook: DbtHook

Provides an existing DbtHook or creates one.

execute(context)[source]

Execute dbt command with prepared arguments.

Execution requires setting up a directory with the dbt project files and overriding the logging.

Parameters:

context – The Airflow’s task context

make_run_results_serializable(result)[source]

Makes dbt’s run result JSON-serializable.

Turn dbt’s RunResult into a dict of only JSON-serializable types Each subclas may implement this method to return a dictionary of JSON-serializable types, the default XCom backend. If implementing custom XCom backends, this method may be overriden.

Parameters:

result (Optional[RunResult])

Return type:

Optional[dict[Any, Any]]

xcom_push_dbt_results(context, dbt_results)[source]

Push any dbt results to XCom.

Parameters:

  • context – The Airflow task’s context.

  • dbt_results (DbtTaskResult) – A namedtuple the results of executing a dbt task, as returned by DbtHook.

Return type:

None

class airflow_dbt_python.operators.dbt.DbtBuildOperator(full_refresh=None, select=None, exclude=None, selector_name=None, singular=None, generic=None, show=None, indirect_selection=None, **kwargs)[source]

Executes a dbt build command.

The build command combines the run, test, seed, and snapshot commands into one. The full Documentation for the dbt build command can be found here: https://docs.getdbt.com/reference/commands/build.

Parameters:

  • full_refresh (Optional[bool])

  • select (Optional[list[str]])

  • exclude (Optional[list[str]])

  • selector_name (Optional[str])

  • singular (Optional[bool])

  • generic (Optional[bool])

  • show (Optional[bool])

  • indirect_selection (Optional[str])

property command: str

Return the build command.

class airflow_dbt_python.operators.dbt.DbtCleanOperator(upload_dbt_project=True, delete_before_upload=True, clean_project_files_only=True, **kwargs)[source]

Executes a dbt clean command.

The documentation for the dbt command can be found here: https://docs.getdbt.com/reference/commands/debug.

Parameters:

  • upload_dbt_project (bool)

  • delete_before_upload (bool)

  • clean_project_files_only (bool)

property command: str

Return the clean command.

class airflow_dbt_python.operators.dbt.DbtCompileOperator(parse_only=None, full_refresh=None, models=None, select=None, exclude=None, selector_name=None, upload_dbt_project=True, **kwargs)[source]

Executes a dbt compile command.

The compile command generates SQL files. The documentation for the dbt command can be found here: https://docs.getdbt.com/reference/commands/compile.

Parameters:

  • parse_only (Optional[bool])

  • full_refresh (Optional[bool])

  • models (Optional[list[str]])

  • select (Optional[list[str]])

  • exclude (Optional[list[str]])

  • selector_name (Optional[str])

  • upload_dbt_project (bool)

property command: str

Return the compile command.

class airflow_dbt_python.operators.dbt.DbtDebugOperator(config_dir=None, no_version_check=None, **kwargs)[source]

Executes a dbt debug command.

The documentation for the dbt command can be found here: https://docs.getdbt.com/reference/commands/debug.

Parameters:

  • config_dir (Optional[bool])

  • no_version_check (Optional[bool])

property command: str

Return the debug command.

class airflow_dbt_python.operators.dbt.DbtDepsOperator(upload_dbt_project=True, **kwargs)[source]

Executes a dbt deps command.

The documentation for the dbt command can be found here: https://docs.getdbt.com/reference/commands/deps.

Parameters:

upload_dbt_project (bool)

property command: str

Return the deps command.

class airflow_dbt_python.operators.dbt.DbtDocsGenerateOperator(compile=True, upload_dbt_project=True, **kwargs)[source]

Executes a dbt docs generate command.

The documentation for the dbt command can be found here: https://docs.getdbt.com/reference/commands/cmd-docs.

Parameters:

upload_dbt_project (bool)

property command: str

Return the generate command.

airflow_dbt_python.operators.dbt.DbtListOperator

alias of DbtLsOperator

class airflow_dbt_python.operators.dbt.DbtLsOperator(resource_types=None, select=None, exclude=None, selector_name=None, dbt_output=None, output_keys=None, indirect_selection=None, **kwargs)[source]

Executes a dbt list (or ls) command.

The documentation for the dbt command can be found here: https://docs.getdbt.com/reference/commands/list.

Parameters:

  • resource_types (Optional[list[str]])

  • select (Optional[list[str]])

  • exclude (Optional[list[str]])

  • selector_name (Optional[str])

  • dbt_output (Optional[Output])

  • output_keys (Optional[list[str]])

  • indirect_selection (Optional[str])

property command: str

Return the list command.

class airflow_dbt_python.operators.dbt.DbtParseOperator(upload_dbt_project=True, **kwargs)[source]

Executes a dbt parse command.

The documentation for the dbt command can be found here: https://docs.getdbt.com/reference/commands/parse.

Parameters:

upload_dbt_project (bool)

property command: str

Return the parse command.

class airflow_dbt_python.operators.dbt.DbtRunOperationOperator(macro, args=None, **kwargs)[source]

Executes a dbt run-operation command.

The documentation for the dbt command can be found here: https://docs.getdbt.com/reference/commands/run-operation.

Parameters:

  • macro (str)

  • args (Optional[dict[str, str]])

property command: str

Return the run-operation command.

class airflow_dbt_python.operators.dbt.DbtRunOperator(full_refresh=None, models=None, select=None, selector_name=None, exclude=None, **kwargs)[source]

Executes a dbt run command.

The run command executes SQL model files against the given target. The documentation for the dbt command can be found here: https://docs.getdbt.com/reference/commands/run.

Parameters:

  • full_refresh (Optional[bool])

  • models (Optional[list[str]])

  • select (Optional[list[str]])

  • selector_name (Optional[list[str]])

  • exclude (Optional[list[str]])

property command: str

Return the run command.

class airflow_dbt_python.operators.dbt.DbtSeedOperator(full_refresh=None, select=None, show=None, exclude=None, selector_name=None, **kwargs)[source]

Executes a dbt seed command.

The seed command loads csv files into the the given target. The documentation for the dbt command can be found here: https://docs.getdbt.com/reference/commands/seed.

Parameters:

  • full_refresh (Optional[bool])

  • select (Optional[list[str]])

  • show (Optional[bool])

  • exclude (Optional[list[str]])

  • selector_name (Optional[str])

property command: str

Return the seed command.

class airflow_dbt_python.operators.dbt.DbtSnapshotOperator(select=None, exclude=None, selector_name=None, **kwargs)[source]

Executes a dbt snapshot command.

The documentation for the dbt command can be found here: https://docs.getdbt.com/reference/commands/snapshot.

Parameters:

  • select (Optional[list[str]])

  • exclude (Optional[list[str]])

  • selector_name (Optional[str])

property command: str

Return the snapshot command.

class airflow_dbt_python.operators.dbt.DbtSourceFreshnessOperator(select=None, dbt_output=None, exclude=None, selector_name=None, upload_dbt_project=True, **kwargs)[source]

Executes a dbt source-freshness command.

The documentation for the dbt command can be found here: https://docs.getdbt.com/reference/commands/source.

Parameters:

  • select (Optional[list[str]])

  • dbt_output (Optional[Union[str, Path]])

  • exclude (Optional[list[str]])

  • selector_name (Optional[str])

  • upload_dbt_project (bool)

property command: str

Return the source command.

class airflow_dbt_python.operators.dbt.DbtTestOperator(singular=None, generic=None, models=None, select=None, exclude=None, selector_name=None, indirect_selection=None, **kwargs)[source]

Executes a dbt test command.

The test command runs data and/or schema tests. The documentation for the dbt command can be found here: https://docs.getdbt.com/reference/commands/test.

Parameters:

  • singular (Optional[bool])

  • generic (Optional[bool])

  • models (Optional[list[str]])

  • select (Optional[list[str]])

  • exclude (Optional[list[str]])

  • selector_name (Optional[str])

  • indirect_selection (Optional[str])

property command: str

Return the test command.

airflow_dbt_python.operators.dbt.run_result_factory(data)[source]

Dictionary factory for dbt’s run_result.

We need to handle dt.datetime and agate.table.Table. The rest of the types should already be JSON-serializable.

Parameters:

data (list[tuple[Any, Any]])