TEST SITE INFORMATION
====================

Company: Test Site Inc.
Purpose: Web Crawler Testing
Version: 1.0
Date: 2024

OVERVIEW
--------
This test site is designed to validate web crawler functionality.
It includes various features to test:
- Crawl depth limiting
- Duplicate URL detection
- Robots.txt compliance
- File type handling
- Navigation patterns

STRUCTURE
---------
The site has multiple levels:
- Level 1: Home, main sections
- Level 2: Sub-pages like team, products
- Level 3: Deep documentation pages

TESTING FEATURES
----------------
1. Duplicate Links: Same URLs linked from multiple pages
2. Relative/Absolute URLs: Mixed URL formats
3. File Downloads: Text files for download testing
4. Restricted Areas: Private section blocked by robots.txt
5. Cross-linking: Pages reference each other

ROBOTS.TXT
----------
- General crawl delay: 2 seconds
- MunicipalCrawler delay: 1 second
- Disallowed: /private/

For more information, visit our website or contact support.