Earl.Crawler
0.0.0-alpha.0.90
This is a prerelease version of Earl.Crawler.
There is a newer prerelease version of this package available.
See the version list below for details.
See the version list below for details.
dotnet add package Earl.Crawler --version 0.0.0-alpha.0.90
NuGet\Install-Package Earl.Crawler -Version 0.0.0-alpha.0.90
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="Earl.Crawler" Version="0.0.0-alpha.0.90" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
paket add Earl.Crawler --version 0.0.0-alpha.0.90
The NuGet Team does not provide support for this client. Please contact its maintainers for support.
#r "nuget: Earl.Crawler, 0.0.0-alpha.0.90"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
// Install Earl.Crawler as a Cake Addin #addin nuget:?package=Earl.Crawler&version=0.0.0-alpha.0.90&prerelease // Install Earl.Crawler as a Cake Tool #tool nuget:?package=Earl.Crawler&version=0.0.0-alpha.0.90&prerelease
The NuGet Team does not provide support for this client. Please contact its maintainers for support.
<h1 align="center">earl</h1>
<div align="center">
Looking for URLs in your area.
</div>
Earl is a suite of APIs for developing url crawlers & web scrapers driven by a middleware pattern similar to, and strongly influenced by, ASP.NET Core.
Basic Usage
var services = new ServiceCollection()
.AddEarlCrawler()
.AddEarlJsonPersistence()
.BuildServiceProvider();
var crawler = services.GetService<IEarlCrawler>();
var options = CrawlerOptionsBuilder.CreateDefault()
.BatchSize( 50 )
.MaxRequestCount( 500 )
.On<CrawlUrlResultEvent>(
( CrawlUrlResultEvent e, CancellationToken cancellation ) =>
{
Console.WriteLine( $"Crawled {e.Result.Url}" );
return ValueTask.CompletedTask;
}
)
.Timeout( TimeSpan.FromMinutes( 30 ) )
.Use(
( CrawlUrlContext context, CrawlUrlDelegate next ) =>
{
Console.WriteLine( $"Executing delegate middleware while crawling {context.Url}" );
return next( context );
}
)
.PersistTo( persist => persist.ToJson( json => json.Destination(...) ) )
.Build();
await crawler.CrawlAsync( new Uri(...), options );
Documentation
Documentation can be find within the READMEs of the sub-directories representing the conceptual components of Earl:
All public APIs should contain thorough XML (triple slash) comments.
Something missing, still have questions? Please open an Issue or submit a PR!
| Product | Versions Compatible and additional computed target framework versions. |
|---|---|
| .NET | net6.0 is compatible. net6.0-android was computed. net6.0-ios was computed. net6.0-maccatalyst was computed. net6.0-macos was computed. net6.0-tvos was computed. net6.0-windows was computed. net7.0 was computed. net7.0-android was computed. net7.0-ios was computed. net7.0-maccatalyst was computed. net7.0-macos was computed. net7.0-tvos was computed. net7.0-windows was computed. net8.0 was computed. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed. |
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.
-
net6.0
- Earl.Crawler.Abstractions (>= 0.0.0-alpha.0.90)
- Earl.Crawler.Events (>= 0.0.0-alpha.0.90)
- Earl.Crawler.Middleware (>= 0.0.0-alpha.0.90)
- Earl.Crawler.Middleware.Html (>= 0.0.0-alpha.0.90)
- Earl.Crawler.Middleware.Http (>= 0.0.0-alpha.0.90)
- Earl.Crawler.Middleware.UrlScraping (>= 0.0.0-alpha.0.90)
NuGet packages
This package is not used by any NuGet packages.
GitHub repositories
This package is not used by any popular GitHub repositories.