DomainScope

Crawl Insights & Performance

Deep dive into our web crawling performance and business data extraction success rates.

Crawl Success Overview

73.9%
Success Rate
Successful Crawls
206,316
Failed Crawls
72,992
Business Data Extracted
121,507
Require JavaScript
29,592

Top HTTP Status Codes

1
200 - OK - Success
190,157 responses
68.1%
2
403 - Forbidden
10,950 responses
3.9%
3
502 - HTTP Status
5,752 responses
2.1%
4
404 - Not Found
4,471 responses
1.6%
5
503 - Service Unavailable
2,911 responses
1.0%
6
500 - Internal Server Error
1,718 responses
0.6%
7
402 - HTTP Status
911 responses
0.3%
8
407 - HTTP Status
726 responses
0.3%
9
522 - HTTP Status
562 responses
0.2%
10
401 - HTTP Status
429 responses
0.2%

Business Data Extraction

58.9%
Success Rate
Business data found
121,507 of 206,316 successful crawls

Crawl Performance

Total Crawls 279,308
Success Rate 73.9%
Business Extraction Rate 58.9%
JavaScript Required Rate 10.6%

Technical Details

Our web crawling system attempts to access each registered .PT domain to extract business information, detect technologies, and gather metadata. Crawl failures can occur due to various reasons including DNS resolution issues, timeouts, blocked access, or invalid SSL certificates.

Business data extraction uses a combination of structured data detection, HTML parsing, and machine learning to identify company information from website content. The success rate varies based on website structure and information availability.