Datasets define what your automation returns. A well-designed dataset is stable, readable, and flexible enough to handle common page structure variations.

Dataset Structure

The top-level datasets field is a JSON object (map). Each key is the dataset name, and the value contains item_selector and properties directly. The dataset name itself is never stored inside the configuration object. Additional optional fields include required_fields, parent, and deduplicate_by.

{
  "datasets": {
    "posts": {
      "item_selector": "css=.social-post",
      "properties": [
        {
          "name": "author",
          "type": "text",
          "selector": "css=.author"
        },
        {
          "name": "content",
          "type": "text",
          "selector": "css=.post-content"
        }
      ]
    }
  }
}

Property Configuration

Each property requires name, type, and normally a selector. The exception is page_url and index, which do not need a selector. The list type uses a nested properties array rather than a simple selector.

Nested List Properties

Use nested lists when each item contains multiple sub-items. A forum thread has posts containing replies, and a review section has product cards containing individual ratings.

{
  "datasets": {
    "threads": {
      "item_selector": "css=.thread-item",
      "properties": [
        {
          "name": "topic",
          "type": "text",
          "selector": "css=.thread-title"
        },
        {
          "name": "replies",
          "type": "list",
          "selector": "css=.reply",
          "properties": [
            {
              "name": "reply_text",
              "type": "text",
              "selector": "css=self"
            }
          ]
        }
      ]
    }
  }
}

Transformations Within Properties

Properties can include a transformation object that runs after extraction but before the row is finalized. This keeps cleaning logic close to the source field.

{
  "name": "clean_price",
  "type": "text",
  "selector": "css=.price",
  "transformation": {
    "type": "extract_regex",
    "pattern": "\\$([0-9.]+)"
  }
}

Extraction Patterns

Flat Grid Pattern

One item_selector and several simple text/attribute properties. This is the fastest and most reliable pattern.

Parent-Child Pattern

One dataset for the outer repeating block and additional datasets for nested child groups. This matches tables, orders, and comment threads.

Multi-Page Pattern

Extract the list first, then use for_each to visit each detail URL. Store detail fields in a second dataset and merge them later.

API + DOM Pattern

Extract page-level data from the DOM, then call the site API for structured JSON. Merge the two output sets using a common ID.

Try It Yourself

Run Basic Navigation scenario 1 for the flat grid pattern. Then run Social Media scenario 4 for a forum thread example with nested replies.