Create an extraction task

POST

/302/crawler/task

任务提交

Create a web crawling task

Price: Charged according to the model called

Request

Authorization

Header Params

Authorization

string

required

302.AI API Key

Example:

Bearer {{YOUR_API_KEY}}

Model

string

optional

Default gpt-4o

Example:

gpt-4o-mini

Body Params application/x-msgpack

urls

array[string]

爬取的链接列表

required

Web page links that need to be crawled

target

string

任务描述

required

Detailed task descriptions are conducive to the accuracy of crawling data

schema

object

需要获取的数据结构

optional

The data format structure that needs to be returned

recursiveConfig

object

爬虫设置

required

Basic crawler settings

matchType

string

匹配模式

required

Automatically obtain information directly, and manually obtain the corresponding crawler code first.

searchModel

string

检索模式

required

By default, only incoming links are crawled, and the depth is to automatically retrieve related links.

outputMd

boolean

MD输出

required

Whether to output markdown content extracted by web pages

maxDepth

integer

最大深度

required

Turn on the depth limit of depth mode

maxUrls

integer

最大链接

required

Maximum search link restrictions

proxyConfig

object

代理设置

optional

The proxy service used when opening a web page

proxyUrl

string

optional

proxyUsername

string

optional

proxyPassword

string

optional

browserConfig

object

浏览器设置

optional

Browser detailed settings

headless

boolean

optional

browserType

string

optional

viewportWidth

integer

optional

viewportHeight

integer

optional

userAgent

string

optional

cacheEnabled

boolean

optional

jsEnabled

boolean

optional

lightMode

boolean

optional

textMode

boolean

optional

array[string]

optional

headers

object

optional

Example

{
  "urls": [
    "https://news.302.ai"
  ],
  "target": "AI model and detailed introduction",
  "schema": {
    "topic": {
      "type": "string",
      "description": "Topic of the article"
    },
    "view": {
      "type": "number",
      "description": "Article views"
    },
    "detail": {
      "type": "string",
      "description": "Article details summary"
    },
    "keywords": {
      "type": "array",
      "description": "Article keywords, up to 5",
      "items": {
        "type": "string"
      }
    },
    "hot": {
      "type": "boolean",
      "description": "Whether it is a popular article with more than 10 views"
    }
  },
  "recursiveConfig": {
    "matchType": "auto",
    "searchModel": "default",
    "outputMd": true,
    "maxDepth": 1,
    "maxUrls": 1
  },
  "proxyConfig": {
    "proxyUrl": "",
    "proxyUsername": "",
    "proxyPassword": ""
  },
  "browserConfig": {
    "headless": true,
    "browserType": "chromium",
    "viewportWidth": 1920,
    "viewportHeight": 1080,
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)",
    "cacheEnabled": false,
    "jsEnabled": false,
    "lightMode": false,
    "textMode": false,
    "cookies": [],
    "headers": {
      "Accept-Language": "zh-CN"
    }
  }
}

Request samples

Shell

JavaScript

Java

Swift

PHP

Python

HTTP

Objective-C

Ruby

OCaml

Dart

curl --location --request POST 'https://api.302.ai/302/crawler/task' \
--header 'Authorization: Bearer sk-jls4AaVBGoe1GwZD64qZA1qyKTN1MPHa4NmvH1cT68z7K1Zz' \
--header 'Model: gpt-4o-mini' \
--header 'Content-Type: application/x-msgpack' \
--data-raw '{
    "urls": [
        "https://news.302.ai"
    ],
    "target": "AI model and detailed introduction",
    "schema": {
        "topic": {
            "type": "string",
            "description": "Topic of the article"
        },
        "view": {
            "type": "number",
            "description": "Article views"
        },
        "detail": {
            "type": "string",
            "description": "Article details summary"
        },
        "keywords": {
            "type": "array",
            "description": "Article keywords, up to 5",
            "items": {
                "type": "string"
            }
        },
        "hot": {
            "type": "boolean",
            "description": "Whether it is a popular article with more than 10 views"
        }
    },
    "recursiveConfig": {
        "matchType": "auto",
        "searchModel": "default",
        "outputMd": true,
        "maxDepth": 1,
        "maxUrls": 1
    },
    "proxyConfig": {
        "proxyUrl": "",
        "proxyUsername": "",
        "proxyPassword": ""
    },
    "browserConfig": {
        "headless": true,
        "browserType": "chromium",
        "viewportWidth": 1920,
        "viewportHeight": 1080,
        "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)",
        "cacheEnabled": false,
        "jsEnabled": false,
        "lightMode": false,
        "textMode": false,
        "cookies": [],
        "headers": {
            "Accept-Language": "zh-CN"
        }
    }
}'

Responses

🟢201success

application/json

Body

taskId

string

任务ID

required

status

string

任务状态

required

message

string

任务描述

required

Example

{
  "taskId": "48c4efe2-f890-4b3a-a595-54fb2ad2623a",
  "status": "pending",
  "message": "Task started successfully"
}

Modified at 2025-02-19 02:44:25

Generate Schema

Query extraction progress