curl --request POST \
--url 'https://api-prod.usefini.com/v2/documents/public/deep-crawl/links' \
--header 'Authorization: Bearer fini_your_api_key' \
--header 'Content-Type: application/json' \
--data '{
"links": ["https://help.example.com"],
"limit": 50
}'
import requests
response = requests.post(
"https://api-prod.usefini.com/v2/documents/public/deep-crawl/links",
headers={
"Authorization": "Bearer fini_your_api_key",
"Content-Type": "application/json",
},
json={
"links": ["https://help.example.com"],
"limit": 50,
},
)
data = response.json()
const response = await fetch(
"https://api-prod.usefini.com/v2/documents/public/deep-crawl/links",
{
method: "POST",
headers: {
Authorization: "Bearer fini_your_api_key",
"Content-Type": "application/json",
},
body: JSON.stringify({
links: ["https://help.example.com"],
limit: 50,
}),
}
);
const data = await response.json();
{
"data": [
"https://help.example.com/articles/returns",
"https://help.example.com/articles/shipping"
],
"message": "Successfully crawled parent URLs"
}
Sources
Crawl links
Discover additional web URLs from one or more seed links before ingesting them as sources.
POST
/
v2
/
documents
/
public
/
deep-crawl
/
links
curl --request POST \
--url 'https://api-prod.usefini.com/v2/documents/public/deep-crawl/links' \
--header 'Authorization: Bearer fini_your_api_key' \
--header 'Content-Type: application/json' \
--data '{
"links": ["https://help.example.com"],
"limit": 50
}'
import requests
response = requests.post(
"https://api-prod.usefini.com/v2/documents/public/deep-crawl/links",
headers={
"Authorization": "Bearer fini_your_api_key",
"Content-Type": "application/json",
},
json={
"links": ["https://help.example.com"],
"limit": 50,
},
)
data = response.json()
const response = await fetch(
"https://api-prod.usefini.com/v2/documents/public/deep-crawl/links",
{
method: "POST",
headers: {
Authorization: "Bearer fini_your_api_key",
"Content-Type": "application/json",
},
body: JSON.stringify({
links: ["https://help.example.com"],
limit: 50,
}),
}
);
const data = await response.json();
{
"data": [
"https://help.example.com/articles/returns",
"https://help.example.com/articles/shipping"
],
"message": "Successfully crawled parent URLs"
}
Use this route when you have one or a few seed URLs and want Fini to discover more web pages before you call Ingest sources.
This route discovers URLs only. It does not create source records and it does not ingest content. After you pick the URLs you want, send them to Ingest sources with
source: "web".Headers
string
required
Bearer token containing your Fini workspace API key. Format:
Bearer fini_... The key needs write scope.string
required
application/jsonBody parameters
array
required
Seed URLs to crawl.
integer
default:"250"
Maximum number of discovered URLs to return.
curl --request POST \
--url 'https://api-prod.usefini.com/v2/documents/public/deep-crawl/links' \
--header 'Authorization: Bearer fini_your_api_key' \
--header 'Content-Type: application/json' \
--data '{
"links": ["https://help.example.com"],
"limit": 50
}'
import requests
response = requests.post(
"https://api-prod.usefini.com/v2/documents/public/deep-crawl/links",
headers={
"Authorization": "Bearer fini_your_api_key",
"Content-Type": "application/json",
},
json={
"links": ["https://help.example.com"],
"limit": 50,
},
)
data = response.json()
const response = await fetch(
"https://api-prod.usefini.com/v2/documents/public/deep-crawl/links",
{
method: "POST",
headers: {
Authorization: "Bearer fini_your_api_key",
"Content-Type": "application/json",
},
body: JSON.stringify({
links: ["https://help.example.com"],
limit: 50,
}),
}
);
const data = await response.json();
Response
array
Discovered child URLs.
string
Success message.
{
"data": [
"https://help.example.com/articles/returns",
"https://help.example.com/articles/shipping"
],
"message": "Successfully crawled parent URLs"
}
Next step
Take the URLs you want fromdata, then call Ingest sources with source: "web" and place those URLs into documentIdsToAdd.
Errors
400 Bad Request
400 Bad Request
The request body is malformed,
links is missing or empty, or one of the inputs is not a valid URL.401 Unauthorized
401 Unauthorized
The API key is missing, malformed, revoked, or invalid.
403 Forbidden
403 Forbidden
The API key does not include the
write scope.500 Internal Server Error
500 Internal Server Error
Fini failed while crawling one or more seed URLs. Retry once, then narrow the input set to isolate the failing URL.
Was this page helpful?

