DocsWorkflowsComponentsMachinemachine:extension:apply

machine:extension:apply

Turns one model decision into a step and performs it on the live page, then reports what the next turn needs to know. A step the loop may not write, a ref the page does not hold and a step the page refused are all ordinary outcomes here rather than component failures. Each one comes back as Error, which is the text the next prompt shows the model, because a loop that stopped on them would throw away the one thing that lets the model correct itself. A step that hands the page to the person in front of it is the one turn this does not hold to the turn timeout. It waits on them for as long as the step allows, and the turn is given back to the session's budget, so a loop is never shortened by the waiting it asked for.

Options

NameTypeDescription
decisionstringThe model's raw reply for this turn, which is one JSON object holding reasoning, done and step. An empty or unreadable reply is a turn that failed rather than a step that failed, so it is fed back rather than refused here
observationPageObservationThe page the decision was made against, whose refs the step's ref is resolved through
  ↳ urlstringIs the observed page's address.
  ↳ titlestringIs the document title.
  ↳ nodes[]ObservedNodeAre the interactive elements, in document order.
    ↳ refstringIs the short handle for this node ("e1", "e2"), unique inside one observation and meaningless outside it.
    ↳ rolestringIs the aria role, explicit or implied by the tag.
    ↳ namestringIs the accessible name, from aria-label, an associated label, the text, the placeholder, the title, or the alt text.
    ↳ tagstringIs the lowercase tag name.
    ↳ typestringIs the input type, for form controls that carry one.
    ↳ valuestringIs the current value, capped by the observer and never set for a password input.
    ↳ placeholderstringIs the control's placeholder text.
    ↳ disabledboolReports whether the control rejects interaction.
    ↳ checkedboolIs the state of a checkbox or radio.
    ↳ expandedboolMirrors aria-expanded.
    ↳ options[]stringAre a select's option values, capped by the observer.
    ↳ selectorstringIs the stable CSS selector the observer built for this node, the same one the element picker would produce.
    ↳ pathstringIs the structural path, for a node CSS cannot address.
    ↳ indexes[]intNarrows a multi-match selector to this node's positions.
    ↳ enter[]stringMirrors Selector.Enter, the shadow hosts and frames to descend through before Selector resolves. Absent for an element in the top document, which is most of them.
    ↳ sectionstringNames the nearest enclosing landmark or the preceding heading, so a reader can tell two identical controls apart by where they sit.
    ↳ contextstringIs the text of the row, card or list item the element sits in, filled in only when the element's own name does not already say which one this is. It is what tells twenty identical add-to-cart buttons apart.
    ↳ offscreenboolMarks an element laid out outside the current viewport. It is still real and still actionable, since the engine scrolls to it.
    ↳ obscuredboolMarks an element another element covers at its center point, which is what a cookie wall or an open dialog does to the page behind it.
    ↳ similarintCounts the further siblings left out because they repeat this one's shape. It sits on the last member of a collapsed run.
  ↳ headings[]ObservedHeadingAre the page's headings, in document order.
    ↳ levelintIs the heading level, 1 for h1.
    ↳ textstringIs the heading's text.
  ↳ textstringIs the readable page text, present only when the observation asked for it, capped by the observer.
  ↳ truncatedboolReports that the node or text cap clipped the page, so a reader knows the snapshot is partial.
  ↳ modalstringNames the dialog or consent wall standing over the page, when one is. Everything outside it is marked obscured, because clicking it is what a browser would refuse to do until the wall is dealt with.
  ↳ modal_frameboolSays the wall is a cross-origin frame, which is how most consent managers ship. Nothing inside one can be listed or clicked from the page it covers, so getting past it needs another way in rather than a ref from this snapshot.
  ↳ omittedintCounts the elements the caps and the repeat collapsing left out.
  ↳ framesintCounts the cross-origin iframes on the page, whose contents no observation can reach from here.
  ↳ lists[]ObservedListAre the repeated content blocks on the page. Nothing in Nodes describes a review card or a product tile that holds no control, so this is the only thing that says the page has records on it at all.
    ↳ refstringIs the short handle for this list inside one observation.
    ↳ selectorstringNames the item container.
    ↳ countintIs how many members the selector matched.
    ↳ sectionstringNames the region of the page the list sits in, when the observer could tell.
    ↳ fields[]ObservedListFieldAre the candidate columns, each relative to the item container.
      ↳ namestringIs the column name the observer proposed.
      ↳ cssstringIs relative to the item container.
      ↳ attributestringNames the attribute the column reads, when it reads one.
      ↳ samplestringIs what the column read on the first member, so a caller can tell a price from a rating without opening the page.
  ↳ paginationObservedPaginationNames the control that reaches the next batch of items, when the page has one.
    ↳ kindstringIs the shape the page uses to reach its next batch, one of link, numbered, more, or infinite.
    ↳ selectorstringTargets the control that advances the listing.
    ↳ labelstringIs the control's visible text.
    ↳ urlstringIs the next page's href when the control is a link.
    ↳ pagesintCounts the numbered page links a pager prints, when it prints any.
steps[]StepThe steps accepted before this turn, in order
  ↳ idstringUniquely names the step within its workflow, echoed on its StepResult.
  ↳ typestringIs the action to perform (click, fill, select, captureUrl, ...).
  ↳ labelstringIs a human-readable description shown in run progress UIs.
  ↳ selectorSelectorPicks the DOM node(s) the step acts on.
    ↳ cssstringIs a CSS selector.
    ↳ pathstringIs an explicit DOM path, used when CSS cannot express the target.
    ↳ indexes[]intNarrows a multi-match selector to these positions.
    ↳ enter[]stringNames the subtrees CSS has to resolve inside, outermost first. Each entry selects a shadow host or an iframe, resolved in the tree the previous entry opened, so a control inside a web component inside a same-origin frame is reachable by one Selector. Empty runs CSS against the top document.
    ↳ indexintNarrows a multi-match selector to a single position.
    ↳ allboolExpands the step to act on every match instead of the first.
  ↳ valuestringIs the input for value-taking steps such as fill and select. It may contain {{ variable }} placeholders.
  ↳ formatstringNames the shape extraction steps return their data in.
  ↳ attributestringNames the DOM attribute an extraction step reads.
  ↳ fieldsExtractFieldAre the named columns an extractList step reads out of each item its selector matched. Key order is the column order, and every returned item carries every key.
  ↳ limitintCaps how many items an extractList step returns. The extension applies its own bound when this is zero, so a listing that lazily grew to ten thousand rows truncates and says so.
  ↳ urlstringIs the address for navigation steps.
  ↳ fnstringNames the page function an eval step invokes.
  ↳ args[]anyAre the arguments passed to Fn.
  ↳ body_paramsmap[string]stringOverrides or adds form fields on the request a captureDownload re-issues, and carries the parameters a captureUrl sends, in the query for a GET and form-encoded otherwise.
  ↳ methodstringIs the HTTP method a captureUrl step fetches its URL with, defaulting to GET.
  ↳ filenamestringRenames a captured file (captureDownload, captureUrl and capturePageFile). It may contain {{ variable }} placeholders.
  ↳ outputstringChooses where a captured file goes, and the controlling client is its only destination.
  ↳ waitWaitConfigBlocks the step until a page condition holds.
    ↳ selectorstringWaits for a matching element to appear (or disappear, see Absent).
    ↳ textstringWaits for this text to be present on the page.
    ↳ url_includesstringWaits for the page URL to contain this substring.
    ↳ absentboolFlips the wait to expect the selector to disappear rather than appear.
    ↳ delay_msintIs a fixed pause in milliseconds, applied on top of any condition.
    ↳ timeout_msintBounds the wait, in milliseconds.
  ↳ promptstringIs the text shown at a user gate asking for manual input.
  ↳ continue_onContinueConditionHolds the step until a user gate is cleared or a page condition is met.
    ↳ manualboolPauses the run for an explicit user continue.
    ↳ selectorstringResumes when a matching element appears.
    ↳ textstringResumes when this text is present on the page.
    ↳ url_includesstringResumes when the page URL contains this substring.
    ↳ timeout_msintBounds the hold, in milliseconds.
  ↳ timeout_msintOverrides the workflow's default step timeout, in milliseconds.
  ↳ optionalboolLets the run continue when this step fails.
  ↳ conditionConditionCondition, when set, gates the step. If it does not hold, the step is skipped rather than failed.
    ↳ selectorstringHolds when a matching element exists on the page.
    ↳ textstringHolds when the page (or the selected element) contains this text.
    ↳ url_includesstringHolds when the page URL contains this substring.
    ↳ matchstringDecides how the set predicates combine, "any" or "all".
    ↳ negateboolInverts the combined result.
  ↳ storestringNames a slot the step's data is saved under, readable by later steps and returned on the WorkflowResult.
  ↳ resumeResumePointResume, when set, marks this step as a point a re-run may start at, and carries the predicate recognizing a browser already standing there.
    ↳ idstringNames this point so a later point's Rewind can reach it. It is unique within one workflow.
    ↳ whenConditionRecognizes the browser standing at this point.
      ↳ selectorstringHolds when a matching element exists on the page.
      ↳ textstringHolds when the page (or the selected element) contains this text.
      ↳ url_includesstringHolds when the page URL contains this substring.
      ↳ matchstringDecides how the set predicates combine, "any" or "all".
      ↳ negateboolInverts the combined result.
    ↳ rewindRewindRewind, when set, is how the browser leaves this point for an earlier one.
      ↳ tostringNames the resume point this reaches, which must sit earlier in the step list than the point declaring it.
      ↳ urlstringNavigates the tab back to the earlier point.
      ↳ selectorstringClicks a control on the current page that leads back, for a flow whose way back is a button rather than an address.
      ↳ waitWaitConfigHolds until the page has settled after the rewind, before the target point's own predicate is checked again.
        ↳ selectorstringWaits for a matching element to appear (or disappear, see Absent).
        ↳ textstringWaits for this text to be present on the page.
        ↳ url_includesstringWaits for the page URL to contain this substring.
        ↳ absentboolFlips the wait to expect the selector to disappear rather than appear.
        ↳ delay_msintIs a fixed pause in milliseconds, applied on top of any condition.
        ↳ timeout_msintBounds the wait, in milliseconds.
  ↳ alwaysboolRuns the step even when re-entry starts the run past it, for work the page cannot report as already done.
last_observationstringThe page as it read on the previous turn, which is how a step that changed nothing visible is told apart from one that moved the page on
last_urlstringWhere the page was on the previous turn, which is how the loop notices that the step it accepted then moved the page
trail[]ExtTurnThe turns taken before this one, so the record travels with the session
  ↳ turnintIs the position of this turn in the session, counting from one.
  ↳ reasoningstringIs the model's one-line justification.
  ↳ typestringIs the step type decided, empty when the model called the goal reached or when the answer could not be read.
  ↳ labelstringIs the step's own description.
  ↳ refstringIs the element the step named.
  ↳ valuestringIs what a fill or select step typed, as written, so a token stays a token and a real credential never reaches here.
  ↳ okboolReports whether the turn produced a step the page accepted.
  ↳ doneboolReports the model calling the goal reached.
  ↳ errorstringIs why the turn produced nothing, in the words the next turn read.
  ↳ urlstringIs where the page was when the turn was decided.
notes[]stringThe model's reasoning for each accepted step, in order
variablesmap[string]stringThe {{ }} placeholders the authored workflow may use, name to description
valuesmap[string]stringReal values for the declared variables, typed into the live page and never emitted
workflow_idstringThe id the session is authoring under, which names the inline run this step travels in
client_idstringThe extension client to drive, a hint the host may honour; empty leaves the choice of browser to the host
budgetintHow many turns the loop may still take, this one included. A turn that waited on a person is handed back, up to five of them
failuresintHow many turns in a row have already failed
max_failuresintHow many consecutive failures end the session (default 3)
timeout_msintHow long to wait for the browser before failing the step

Outputs

NameTypeDescription
steps[]StepEvery step accepted so far, this turn's included when it worked
  ↳ idstringUniquely names the step within its workflow, echoed on its StepResult.
  ↳ typestringIs the action to perform (click, fill, select, captureUrl, ...).
  ↳ labelstringIs a human-readable description shown in run progress UIs.
  ↳ selectorSelectorPicks the DOM node(s) the step acts on.
    ↳ cssstringIs a CSS selector.
    ↳ pathstringIs an explicit DOM path, used when CSS cannot express the target.
    ↳ indexes[]intNarrows a multi-match selector to these positions.
    ↳ enter[]stringNames the subtrees CSS has to resolve inside, outermost first. Each entry selects a shadow host or an iframe, resolved in the tree the previous entry opened, so a control inside a web component inside a same-origin frame is reachable by one Selector. Empty runs CSS against the top document.
    ↳ indexintNarrows a multi-match selector to a single position.
    ↳ allboolExpands the step to act on every match instead of the first.
  ↳ valuestringIs the input for value-taking steps such as fill and select. It may contain {{ variable }} placeholders.
  ↳ formatstringNames the shape extraction steps return their data in.
  ↳ attributestringNames the DOM attribute an extraction step reads.
  ↳ fieldsExtractFieldAre the named columns an extractList step reads out of each item its selector matched. Key order is the column order, and every returned item carries every key.
  ↳ limitintCaps how many items an extractList step returns. The extension applies its own bound when this is zero, so a listing that lazily grew to ten thousand rows truncates and says so.
  ↳ urlstringIs the address for navigation steps.
  ↳ fnstringNames the page function an eval step invokes.
  ↳ args[]anyAre the arguments passed to Fn.
  ↳ body_paramsmap[string]stringOverrides or adds form fields on the request a captureDownload re-issues, and carries the parameters a captureUrl sends, in the query for a GET and form-encoded otherwise.
  ↳ methodstringIs the HTTP method a captureUrl step fetches its URL with, defaulting to GET.
  ↳ filenamestringRenames a captured file (captureDownload, captureUrl and capturePageFile). It may contain {{ variable }} placeholders.
  ↳ outputstringChooses where a captured file goes, and the controlling client is its only destination.
  ↳ waitWaitConfigBlocks the step until a page condition holds.
    ↳ selectorstringWaits for a matching element to appear (or disappear, see Absent).
    ↳ textstringWaits for this text to be present on the page.
    ↳ url_includesstringWaits for the page URL to contain this substring.
    ↳ absentboolFlips the wait to expect the selector to disappear rather than appear.
    ↳ delay_msintIs a fixed pause in milliseconds, applied on top of any condition.
    ↳ timeout_msintBounds the wait, in milliseconds.
  ↳ promptstringIs the text shown at a user gate asking for manual input.
  ↳ continue_onContinueConditionHolds the step until a user gate is cleared or a page condition is met.
    ↳ manualboolPauses the run for an explicit user continue.
    ↳ selectorstringResumes when a matching element appears.
    ↳ textstringResumes when this text is present on the page.
    ↳ url_includesstringResumes when the page URL contains this substring.
    ↳ timeout_msintBounds the hold, in milliseconds.
  ↳ timeout_msintOverrides the workflow's default step timeout, in milliseconds.
  ↳ optionalboolLets the run continue when this step fails.
  ↳ conditionConditionCondition, when set, gates the step. If it does not hold, the step is skipped rather than failed.
    ↳ selectorstringHolds when a matching element exists on the page.
    ↳ textstringHolds when the page (or the selected element) contains this text.
    ↳ url_includesstringHolds when the page URL contains this substring.
    ↳ matchstringDecides how the set predicates combine, "any" or "all".
    ↳ negateboolInverts the combined result.
  ↳ storestringNames a slot the step's data is saved under, readable by later steps and returned on the WorkflowResult.
  ↳ resumeResumePointResume, when set, marks this step as a point a re-run may start at, and carries the predicate recognizing a browser already standing there.
    ↳ idstringNames this point so a later point's Rewind can reach it. It is unique within one workflow.
    ↳ whenConditionRecognizes the browser standing at this point.
      ↳ selectorstringHolds when a matching element exists on the page.
      ↳ textstringHolds when the page (or the selected element) contains this text.
      ↳ url_includesstringHolds when the page URL contains this substring.
      ↳ matchstringDecides how the set predicates combine, "any" or "all".
      ↳ negateboolInverts the combined result.
    ↳ rewindRewindRewind, when set, is how the browser leaves this point for an earlier one.
      ↳ tostringNames the resume point this reaches, which must sit earlier in the step list than the point declaring it.
      ↳ urlstringNavigates the tab back to the earlier point.
      ↳ selectorstringClicks a control on the current page that leads back, for a flow whose way back is a button rather than an address.
      ↳ waitWaitConfigHolds until the page has settled after the rewind, before the target point's own predicate is checked again.
        ↳ selectorstringWaits for a matching element to appear (or disappear, see Absent).
        ↳ textstringWaits for this text to be present on the page.
        ↳ url_includesstringWaits for the page URL to contain this substring.
        ↳ absentboolFlips the wait to expect the selector to disappear rather than appear.
        ↳ delay_msintIs a fixed pause in milliseconds, applied on top of any condition.
        ↳ timeout_msintBounds the wait, in milliseconds.
  ↳ alwaysboolRuns the step even when re-entry starts the run past it, for work the page cannot report as already done.
stepStepWhat this turn applied, absent when the turn was refused or the model called the goal reached
  ↳ idstringUniquely names the step within its workflow, echoed on its StepResult.
  ↳ typestringIs the action to perform (click, fill, select, captureUrl, ...).
  ↳ labelstringIs a human-readable description shown in run progress UIs.
  ↳ selectorSelectorPicks the DOM node(s) the step acts on.
    ↳ cssstringIs a CSS selector.
    ↳ pathstringIs an explicit DOM path, used when CSS cannot express the target.
    ↳ indexes[]intNarrows a multi-match selector to these positions.
    ↳ enter[]stringNames the subtrees CSS has to resolve inside, outermost first. Each entry selects a shadow host or an iframe, resolved in the tree the previous entry opened, so a control inside a web component inside a same-origin frame is reachable by one Selector. Empty runs CSS against the top document.
    ↳ indexintNarrows a multi-match selector to a single position.
    ↳ allboolExpands the step to act on every match instead of the first.
  ↳ valuestringIs the input for value-taking steps such as fill and select. It may contain {{ variable }} placeholders.
  ↳ formatstringNames the shape extraction steps return their data in.
  ↳ attributestringNames the DOM attribute an extraction step reads.
  ↳ fieldsExtractFieldAre the named columns an extractList step reads out of each item its selector matched. Key order is the column order, and every returned item carries every key.
  ↳ limitintCaps how many items an extractList step returns. The extension applies its own bound when this is zero, so a listing that lazily grew to ten thousand rows truncates and says so.
  ↳ urlstringIs the address for navigation steps.
  ↳ fnstringNames the page function an eval step invokes.
  ↳ args[]anyAre the arguments passed to Fn.
  ↳ body_paramsmap[string]stringOverrides or adds form fields on the request a captureDownload re-issues, and carries the parameters a captureUrl sends, in the query for a GET and form-encoded otherwise.
  ↳ methodstringIs the HTTP method a captureUrl step fetches its URL with, defaulting to GET.
  ↳ filenamestringRenames a captured file (captureDownload, captureUrl and capturePageFile). It may contain {{ variable }} placeholders.
  ↳ outputstringChooses where a captured file goes, and the controlling client is its only destination.
  ↳ waitWaitConfigBlocks the step until a page condition holds.
    ↳ selectorstringWaits for a matching element to appear (or disappear, see Absent).
    ↳ textstringWaits for this text to be present on the page.
    ↳ url_includesstringWaits for the page URL to contain this substring.
    ↳ absentboolFlips the wait to expect the selector to disappear rather than appear.
    ↳ delay_msintIs a fixed pause in milliseconds, applied on top of any condition.
    ↳ timeout_msintBounds the wait, in milliseconds.
  ↳ promptstringIs the text shown at a user gate asking for manual input.
  ↳ continue_onContinueConditionHolds the step until a user gate is cleared or a page condition is met.
    ↳ manualboolPauses the run for an explicit user continue.
    ↳ selectorstringResumes when a matching element appears.
    ↳ textstringResumes when this text is present on the page.
    ↳ url_includesstringResumes when the page URL contains this substring.
    ↳ timeout_msintBounds the hold, in milliseconds.
  ↳ timeout_msintOverrides the workflow's default step timeout, in milliseconds.
  ↳ optionalboolLets the run continue when this step fails.
  ↳ conditionConditionCondition, when set, gates the step. If it does not hold, the step is skipped rather than failed.
    ↳ selectorstringHolds when a matching element exists on the page.
    ↳ textstringHolds when the page (or the selected element) contains this text.
    ↳ url_includesstringHolds when the page URL contains this substring.
    ↳ matchstringDecides how the set predicates combine, "any" or "all".
    ↳ negateboolInverts the combined result.
  ↳ storestringNames a slot the step's data is saved under, readable by later steps and returned on the WorkflowResult.
  ↳ resumeResumePointResume, when set, marks this step as a point a re-run may start at, and carries the predicate recognizing a browser already standing there.
    ↳ idstringNames this point so a later point's Rewind can reach it. It is unique within one workflow.
    ↳ whenConditionRecognizes the browser standing at this point.
      ↳ selectorstringHolds when a matching element exists on the page.
      ↳ textstringHolds when the page (or the selected element) contains this text.
      ↳ url_includesstringHolds when the page URL contains this substring.
      ↳ matchstringDecides how the set predicates combine, "any" or "all".
      ↳ negateboolInverts the combined result.
    ↳ rewindRewindRewind, when set, is how the browser leaves this point for an earlier one.
      ↳ tostringNames the resume point this reaches, which must sit earlier in the step list than the point declaring it.
      ↳ urlstringNavigates the tab back to the earlier point.
      ↳ selectorstringClicks a control on the current page that leads back, for a flow whose way back is a button rather than an address.
      ↳ waitWaitConfigHolds until the page has settled after the rewind, before the target point's own predicate is checked again.
        ↳ selectorstringWaits for a matching element to appear (or disappear, see Absent).
        ↳ textstringWaits for this text to be present on the page.
        ↳ url_includesstringWaits for the page URL to contain this substring.
        ↳ absentboolFlips the wait to expect the selector to disappear rather than appear.
        ↳ delay_msintIs a fixed pause in milliseconds, applied on top of any condition.
        ↳ timeout_msintBounds the wait, in milliseconds.
  ↳ alwaysboolRuns the step even when re-entry starts the run past it, for work the page cannot report as already done.
notes[]stringThe reasoning behind each accepted step, in order
trail[]ExtTurnEvery turn taken so far, accepted or not, this one last. It is the only record of why a session went the way it did, since a refused turn leaves no trace in the workflow being written
  ↳ turnintIs the position of this turn in the session, counting from one.
  ↳ reasoningstringIs the model's one-line justification.
  ↳ typestringIs the step type decided, empty when the model called the goal reached or when the answer could not be read.
  ↳ labelstringIs the step's own description.
  ↳ refstringIs the element the step named.
  ↳ valuestringIs what a fill or select step typed, as written, so a token stays a token and a real credential never reaches here.
  ↳ okboolReports whether the turn produced a step the page accepted.
  ↳ doneboolReports the model calling the goal reached.
  ↳ errorstringIs why the turn produced nothing, in the words the next turn read.
  ↳ urlstringIs where the page was when the turn was decided.
countintHow many steps the session holds
okboolWhether this turn's step reached the page and succeeded
doneboolWhether the model called the goal reached
errorstringWhy this turn was refused or failed, in the words the next turn reads
datastringWhat the applied step returned, which is what an extract step read off the page
reasoningstringThe model's justification for this turn
failuresintHow many turns in a row have now failed, reset to zero by a turn that worked
budgetintHow many turns are left after this one
match_urlstringThe host and path the session's tab is now on, for the next turn to find it by
urlstringWhere the session is after this turn, which is where the page was observed
continueboolWhether another turn is worth taking, meaning the goal is not reached, the budget is not spent and the model is still recovering from its own failures

Example

Performs one model decision against the live page. It never fails on a step the page refused, so what a workflow reads afterwards is act.ok and act.error rather than a stopped run.

name: Perform one decided browser step
actions:
  - name: act
    component: machine:extension:apply
    vars:
      decision: "{{ decide.content }}"
      observation: "{{ look.observation }}"
      last_observation: "{{ var.last_observation }}"
      steps: "{{ var.steps }}"
      workflow_id: site-login
      budget: 12

outputs:
  steps: "{{ act.steps }}"
  keep_going: "{{ act.continue }}"
  why_not: "{{ act.error }}"