Client

tdworkflow.client module

class tdworkflow.client.AttemptAPI

Bases: object

attempt(attempt: Union[int, tdworkflow.attempt.Attempt], inplace: bool = False) → Optional[tdworkflow.attempt.Attempt, None]

Get an attempt

Parameters:
  • attempt (int) – Attempt ID or Attempt object
  • inplace – If True, do operation inplace and return None
Returns:

Attempt object
Return type:

Attempt
attempt_tasks(attempt: Union[int, tdworkflow.attempt.Attempt]) → List[tdworkflow.task.Task]

Get tasks of a session

Parameters:attempt – Attempt id or Attempt object
Returns:List of Task
attempts(project: Union[str, tdworkflow.project.Project, None] = None, workflow: Union[str, tdworkflow.workflow.Workflow, None] = None, include_retried: Optional[bool, None] = None, last_id: Optional[int, None] = None, page_size: Optional[int, None] = None) → List[tdworkflow.attempt.Attempt]

List attempts

Parameters:
  • project (Optional[Union[str, Project]]) – Project name or Project object, optional
  • workflow (Optional[Union[str, Workflow]]) – Workflow name or Workflow object, optional
  • include_retried (Optional[bool]) – List more than 1 attempts per session
  • last_id (Optional[int]) – List attempts whose id is grater than this id for pagination
  • page_size (Optional[int]) – Number of attempts to return
Returns:

List of Attempt object
Return type:

List[Attempt]
kill_attempt(attempt: Union[int, tdworkflow.attempt.Attempt], inplace: bool = False) → Optional[tdworkflow.attempt.Attempt, None]

Kill a session

Parameters:
  • attempt (Union[int, Attempt]) – Attempt ID or Attempt object
  • inplace – If True, do operation inplace and return None
Returns:

True if succeeded
Return type:

Attempt
retried_attempts(attempt: Union[int, tdworkflow.attempt.Attempt]) → List[tdworkflow.attempt.Attempt]

Get retried attempt list

Parameters:attempt – Attempt id or Attempt object
Returns:List of Attempt
start_attempt(workflow: Union[int, tdworkflow.workflow.Workflow], session_time: Optional[str, None] = None, retry_attempt_name: Optional[str, None] = None, workflow_params: Optional[Dict[str, Any], None] = None) → tdworkflow.attempt.Attempt

Start workflow session

Parameters:
  • workflow – Workflow id or Workflow object
  • session_time – Session time, optional Default: datetime.datetime.now()
  • retry_attempt_name – Retry attempt name, optional
  • workflow_params – Extra workflow parameters
Returns:
wait_attempt(attempt: Union[int, tdworkflow.attempt.Attempt], wait_interval: int = 5) → tdworkflow.attempt.Attempt

Wait until an attempt finished

Parameters:
  • attempt (Union[int, Attempt]) – Attempt ID or Attempt object
  • wait_interval (int) – Wait interval in second. Default 5 sec
Returns:

Latest status of Attempt
Return type:

Attempt
class tdworkflow.client.Client(site: str = 'us', endpoint: Optional[str, None] = None, apikey: Optional[str, None] = None, user_agent: Optional[str, None] = None, _session: Optional[requests.sessions.Session, None] = None, scheme: str = 'https')

Bases: tdworkflow.client.AttemptAPI, tdworkflow.client.WorkflowAPI, tdworkflow.client.ProjectAPI, tdworkflow.client.ScheduleAPI, tdworkflow.client.SessionAPI, tdworkflow.client.LogAPI

delete(path: str, params: Optional[Dict[str, str], None] = None) → Optional[Dict[str, Any], None]

DELETE operator for REST API

Parameters:
  • path (str) – Treasure Workflow API path
  • params (Optional[Dict[str, str]], optional) – Query parameters, defaults to None
Returns:

True if succeeded
Return type:

bool
get(path: str, params: Optional[Dict[str, Union[str, bool, int, None]], None] = None, content: bool = False) → Union[Dict[str, Any], bytes]

GET operator for REST API

Parameters:
  • path (str) – Treasure Workflow API path
  • params (Optional[Dict[str, Union[str, bool, int, None]]], optional) – Query parameters, defaults to None
  • content (bool) – Return content body without parsing JSON if True
Returns:

Response data in JSON or bytes
Return type:

Union[Dict[str, str], bytes]
http
Returns:Established session
Return type:requests.Session
post(path: str, body: Optional[Dict[str, Any], None] = None, content: bool = False) → Union[Dict[str, Any], bytes, None]

POST operator for REST API

Parameters:
  • path (str) – Treasure Workflow API path
  • body (Optional[Dict[str, Any]], optional) – Content body in dictionary to be passed in JSON
  • content (bool) – Return content body without parsing JSON if True
Returns:

True if succeeded
put(path: str, data: Union[str, Dict[str, Any], List[Tuple[Any]], BinaryIO, None] = None, _json: Optional[Dict[str, Any], None] = None, params: Optional[Dict[str, str], None] = None) → Optional[Dict[str, Any], None]

PUT operator for REST API

Parameters:
  • path (str) – Treasure Workflow API path
  • data (Optional[str, Union[Dict, List[Tuple], BinaryIO]], optional) – Content body
  • _json (Optional[Dict[str, Any]], optional) – Content body as JSON
  • params (Optional[Dict[str, str]], optional) – Query parameters
Returns:

Response content
Return type:

Dict[str,str]
class tdworkflow.client.LogAPI

Bases: object

log_file(attempt: Union[tdworkflow.attempt.Attempt, int], file: Union[tdworkflow.log.LogFile, str]) → Union[bytes, str]

Get a log string for an attempt

Parameters:
  • attempt – Target Attempt id or Attempt object
  • file – LogFile name or LogFile object
Returns:

Log string
log_files(attempt: Union[tdworkflow.attempt.Attempt, int], task: Optional[str, None] = None, direct_download: Optional[bool, None] = None) → List[tdworkflow.log.LogFile]

Get log files information

Parameters:
  • attempt – Target Attempt id or Attempt object
  • task – Task name
  • direct_download – Flag for direct download
Returns:

List of LogFile
logs(attempt: Union[tdworkflow.attempt.Attempt, int]) → List[Union[bytes, str]]

Get log string list for an attempt

Parameters:attempt – Attempt ID or Attempt object
Returns:A list of log 
>>> import tdworkflow
>>> client = tdworkflow.client.Client("us")
>>> attempts = client.attempts(project="pandas-df")
>>> logs = client.logs(attempts[0])
>>> print(logs)
['2019-10-30 08:34:51.672 +0000 [INFO] (0250@[1:pandas-df]+pandas-df+read_into_df) io.digdag.core.agent.OperatorManager: py>: py_scripts.examples.read_td_table\n',
'2019-10-30 08:34:59.879 +0000 [INFO] (0237@[1:pandas-df]+pandas-df+read_into_df) io.digdag.core.agent.OperatorManager: py>: py_scripts.examples.read_td_table\nWait running a command task: status provisioning',
...
class tdworkflow.client.ProjectAPI

Bases: object

create_project(project_name: str, target_dir: str, exclude_patterns: Optional[List[str], None] = None, revision: Optional[str, None] = None) → tdworkflow.project.Project

Create a new project

Parameters:
  • project_name – Project name
  • target_dir – Target directory name
  • exclude_patterns – Exclude file patterns. They are treated as regexp patterns. default: [“venv”, “.venv”, “__pycache__”, “.egg-info”, “.digdag”, “.pyc”] + dot files
  • revision – Revision name
Returns:
delete_project(project: Union[int, tdworkflow.project.Project]) → bool

Delete a project

Parameters:project – Project id or Project object
Returns:True if succeeded
delete_secret(project: Union[int, tdworkflow.project.Project], key: str) → bool

Delete secret key

Parameters:
  • project (Union[int, Project]) – Project ID or Project object
  • key (str) – Secret key to be deleted
Returns:

True if succeeded
Return type:

bool
delete_secrets(project: Union[int, tdworkflow.project.Project], keys: List[str]) → bool

Delete multiple secret keys at once

Parameters:
  • project (Union[int, Project]) – Project ID or Project object
  • keys (List[str]) – The list of secret keys to be deleted
Returns:

True if succeeded
Return type:

bool
download_project_archive(project: Union[int, tdworkflow.project.Project], file_path: str, revision: Optional[str, None] = None) → bool

Download a project and save as a file (tar.gz)

Parameters:
  • project – Project id or Project object
  • file_path – Target file path to be saved in tar.gz
  • revision – Revision name
Returns:

True if succeeded
project(project: Union[int, tdworkflow.project.Project]) → tdworkflow.project.Project

Get a project

Parameters:project (Union[int, Project]) – Project id or Project object
Returns:A Project
project_revisions(project: Union[int, tdworkflow.project.Project]) → List[tdworkflow.revision.Revision]

List revisions associated with Project

Parameters:project – Project id or Project object
Returns:List of Revision
project_schedules(project: Union[int, tdworkflow.project.Project], workflow: Union[str, tdworkflow.workflow.Workflow, None] = None, last_id: Optional[int, None] = None) → List[tdworkflow.schedule.Schedule]

List schedules associated with Project

Parameters:
  • project – Project ID or project object
  • workflow – Workflow name or Workflow object
  • last_id – List schedules whose id is grater than this id for pagination
Returns:

List of Schedule
project_sessions(project: Union[int, tdworkflow.project.Project], workflow: Union[str, tdworkflow.workflow.Workflow, None] = None, last_id: Optional[int, None] = None, page_size: Optional[int, None] = None) → List[tdworkflow.session.Session]

List sessions associated with a Project

Parameters:
  • project – Project ID or Project object
  • workflow – Workflow name or Workflow object
  • last_id – List sessions whose id is grater than this id for pagination
  • page_size – Number of sessions to return
Returns:

List of Session
project_workflows(project: Union[int, tdworkflow.project.Project], workflow: Union[str, tdworkflow.workflow.Workflow, None] = None, revision: Optional[str, None] = None) → List[tdworkflow.workflow.Workflow]

Get workflows associated with a project

Parameters:
  • project (Optional[Union[str, Workflow]], optional) – Project id or Project object
  • workflow – Workflow name or Workflow object
  • revision (Optional[str], optional) – Revision name
Returns:

List of Workflow
Return type:

List[Workflow]
project_workflows_by_name(project_name: str) → List[tdworkflow.workflow.Workflow]

List workflows associate with Project by project name

Parameters:project_name – Target project name
Returns:List of Workflow
projects(name: Optional[str, None] = None, name_pattern: Optional[str, None] = None, count: Optional[int, None] = None, last_id: Optional[int, None] = None) → List[tdworkflow.project.Project]

List projects

Parameters:
  • name (Optional[str], optional) – Project name
  • name_pattern (Optional[str], optional) – Name pattern to be partially matched
  • count (Optional[int], optional) – Number of projects to return
  • last_id (Optional[int], optional) – List projects whose id is grater than this id for pagination.
Returns:

List of Project
Return type:

List[Project]
secrets(project: Union[int, tdworkflow.project.Project]) → List[str]

Show secret keys

Parameters:project (Union[int, Project]) – Project ID or Project object
Returns:The list of secret keys
Return type:List[str]
set_secrets(project: Union[int, tdworkflow.project.Project], secrets: Dict[str, str]) → bool

Set project secrets

Parameters:
  • project (Union[int, Project]) – Project ID or Project object
  • secrets (Dict[str, str]) – Workflow secrets
Returns:

True if succeeded
Return type:

bool
class tdworkflow.client.ScheduleAPI

Bases: object

backfill_schedule(schedule: Union[int, tdworkflow.schedule.Schedule], attempt_name: str, from_time: Union[str, datetime.datetime], dry_run: bool = False, count: Optional[int, None] = None) → tdworkflow.schedule.ScheduleAttempt

Run or re-run past schedules

Parameters:
  • schedule – Target Schedule id or Schedule object
  • attempt_name – Attempt name
  • from_time – From time e.g “2019-11-01T06:20:07.000+00:00” in str or datetime.datetime.
  • dry_run – Flag for dry run
  • count – Count
Returns:

ScheduleAttempt
disable_schedule(schedule: Union[int, tdworkflow.schedule.Schedule]) → tdworkflow.schedule.Schedule

Disable a schedule

Parameters:schedule – Schedule ID or Schedule object
Returns:New Schedule
enable_schedule(schedule: Union[int, tdworkflow.schedule.Schedule]) → tdworkflow.schedule.Schedule

Enable a schedule

Parameters:schedule – Schedule ID or Schedule object
Returns:New Schedule
schedule(schedule: Union[int, tdworkflow.schedule.Schedule]) → tdworkflow.schedule.Schedule

Get a schedule

Parameters:schedule – Schedule id or Schedule object
Returns:Schedule
schedules(last_id: Optional[int, None] = None) → List[tdworkflow.schedule.Schedule]

List schedules

Parameters:last_id – List schedules whose id is grater than this id for pagination.
Returns:List of Schedule
skip_schedule(schedule: Union[int, tdworkflow.schedule.Schedule], from_time: Union[str, datetime.datetime, None] = None, next_time: Optional[str, None] = None, next_run_time: Union[str, datetime.datetime, None] = None, dry_run: Optional[bool, None] = False) → tdworkflow.schedule.Schedule

Skip schedules forward to a future time

Parameters:
  • schedule – Schedule ID or Schedule object
  • from_time – From time
  • next_time – Next time
  • next_run_time – Next run time
  • dry_run – Flag for dry run
Returns:

New Schedule
class tdworkflow.client.SessionAPI

Bases: object

session(session: Union[int, tdworkflow.session.Session]) → tdworkflow.session.Session

Get a session

Parameters:session – Sesion ID or Session object
Returns:New Session
session_attempts(session: Union[int, tdworkflow.session.Session], last_id: Optional[int, None] = None, page_size: Optional[int, None] = None) → List[tdworkflow.attempt.Attempt]

Get attempts of a session

Parameters:
  • session – Session ID or Session object
  • last_id – List attempts whose id is grater than this id for pagination
  • page_size – Number of attempts to return
Returns:

List of Attempt
sessions(last_id: Optional[int, None] = None, page_size: Optional[int, None] = None) → List[tdworkflow.session.Session]

List sessions

Parameters:
  • last_id – List sessions whose id is grater than this id for pagination
  • page_size – Number of sessions to return
Returns:

List of Session
class tdworkflow.client.WorkflowAPI

Bases: object

workflow(workflow: Union[int, tdworkflow.workflow.Workflow]) → tdworkflow.workflow.Workflow

Get a specific workflow

Parameters:workflow (Union[int, Workflow]) – Id for workflow or Workflow object
Returns:A workflow
Return type:Workflow
workflows(name_pattern: Optional[str, None] = None, search_project_name: bool = False, order: Optional[str, None] = None, count: Optional[int, None] = None, last_id: Optional[int, None] = None) → List[tdworkflow.workflow.Workflow]

List worlfows

Parameters:
  • name_pattern (Optional[str], optional) – Name pattern to be partially matched
  • search_project_name (bool) – Flag to use name_pattern to search partial project name. Default False
  • order (Optional[str]) – Sort order. ‘asc’ or ‘dsc’. Default ‘asc’
  • count (Optional[int], optional) – Number of workflows to return
  • last_id (Optional[int], optional) – List workflows whose id is grater than this id for pagination.
Returns:

List of Workflow
Return type:

List[Workflow]