Nick.RobotsParser
2.0.5
There is a newer version of this package available.
See the version list below for details.
See the version list below for details.
dotnet add package Nick.RobotsParser --version 2.0.5
NuGet\Install-Package Nick.RobotsParser -Version 2.0.5
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="Nick.RobotsParser" Version="2.0.5" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add Nick.RobotsParser --version 2.0.5
The NuGet Team does not provide support for this client. Please contact its maintainers for support.
#r "nuget: Nick.RobotsParser, 2.0.5"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
// Install Nick.RobotsParser as a Cake Addin #addin nuget:?package=Nick.RobotsParser&version=2.0.5 // Install Nick.RobotsParser as a Cake Tool #tool nuget:?package=Nick.RobotsParser&version=2.0.5
The NuGet Team does not provide support for this client. Please contact its maintainers for support.
Library for working with robots.txt
- Parse robots.txt into Typed object.
- Lookup Allowed/Disallowed/Crawldelay based on User-Agent.
- Traverse sitemap in robots.txt for urls.
For More info see: https://github.com/nicholasbergesen/RobotsParser/blob/master/README.md
Product | Versions Compatible and additional computed target framework versions. |
---|---|
.NET | net6.0 is compatible. net6.0-android was computed. net6.0-ios was computed. net6.0-maccatalyst was computed. net6.0-macos was computed. net6.0-tvos was computed. net6.0-windows was computed. net7.0 was computed. net7.0-android was computed. net7.0-ios was computed. net7.0-maccatalyst was computed. net7.0-macos was computed. net7.0-tvos was computed. net7.0-windows was computed. net8.0 was computed. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed. |
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.
-
net6.0
- No dependencies.
NuGet packages
This package is not used by any NuGet packages.
GitHub repositories
This package is not used by any popular GitHub repositories.
- Improve sitemap parsing robustness.
- Ignore sitemap parsing errors.