Stott.Optimizely.RobotsHandler 6.0.0

Stott.Optimizely.RobotsHandler

This is an admin extension for Optimizely CMS 12+ for managing robots content. Stott Robots Handler is a free to use module, however if you want to show your support, buy me a coffee on ko-fi:

Robots.Txt Management

Robots.txt content can be managed on a per site and host definition basis. A host of "default" applies to all unspecified hosts within a site, while specific host definitions will only apply to the specific host.

Environment Robots

Introduced within version 4.0.0

Environment Robots allows you to configure the meta robots tag and X-Robots-Tag header for all page requests within the current environment. This functionality provides the ability to prevent search engine robots from scanning indexing a site that is a lower level environment or a production environment that is not ready for general consumption.

Options will always exist for Integration, Preproduction, Production and the current environment name. This allows you to preconfigure a lower enviroment when cloning content from production to lower environments.

When a configuration is active, a Meta Tag Helper will look for and update the meta robots tag while a middleware will include the X-Robots-Tag header. Its best in this case that your solution always renders the meta robots element and allow the Meta Tag Helper to either override it or remove it where needed.

The meta tag helper will execute for any meta tag with a name attribute. The logic within the robots tag helper will only execute where the name attribute has a value of robots. In such a circumstance it will perform one of the following actions

• When name is robots
  • If the page has a robots definition and an environment configuration is not set, then the page robots value will be used.
  • If the page has a robots definition and an environment configuration is set, then the environment configuration replaces the page definition.
  • If the page does not have robots definition and an environment configuration is not set, then the meta tag will be removed.
  • If the page does not have a robots definition and an environment configuration is set, then the meta tag will use the environment configuration
• When name is not robots
  • Preserve existing state.

Examples: | Page Robots | Environment Robots | Result | |-------------|--------------------|--------| | noindex,nofollow | noindex,nofollow,noimageindex | noindex,nofollow,noimageindex | | noindex,nofollow | - | noindex,nofollow | | - | noindex,nofollow,noimageindex | noindex,nofollow,noimageindex | | - | - | meta robots tag is removed |

LLMS.txt Management

Introduced within version 5.0.0

Llms.txt content is a new standard in making website context and content easy to understand to Large Language Models (AI). This standard uses markdown as it is both easy to understand for both humans and AI. You can learn more about llms.txt here.

Llms.txt content can be managed on a per site and host definition basis. A host of "default" applies to all unspecified hosts within a site, while specific host definitions will only apply to the specific host. If Llms.txt content has not been defined, then a request to /llms.txt will result in a 404 response.

Opal Tools and Headless Support

Introduced within version 6.0.0

Opal is Optimizely's AI at the centre of it's content marketing suite known as Optimizely One. By providing tools for Opal, user's can access and manage robots.txt and llms.txt content directly from Opal's chat window. The APIs used for these Opal Tools also lend themselves to usage within headless solutions.

In order to ensure access to the endpoints of these tools requires a bearer token to be provided. A new tab has been added into the Robots Handler to allow a CMS Administrator to manage bearer tokens required to access all of the endpoints. This screen also has the ability to show API details to the user.

When creating a new API token, the token itself is randomly generated and visible to the user. This is the only time at which you can retrieve the token. Once it is created and the modal has closed, the token value cannot be retrieved again. The scope for robots.txt and llms.txt management can be managed individually, these scopes are:

• None: Grants no access to either the Get or Save endpoints.
• Read: Grants access to the Get endpoints.
• Write: Grants access to the Get and Save endpoints.

To grant access to Opal, navigate to the main opal interface and select the tools interface from the left hand menu. Switch to the Registries tab and then click on the "Add tool registry" button.

Copy the Discovery URL from the list of available endpoints in the Stott Robots Handler and enter it into the "Discovery URL" field. Place your newly generated token into the "Bearer Token (Optional)" field. Note that while the bearer token is optional in this interface, a bearer token is required for the tools in the Stott Robots Handler.

With your tools registered, you will then be able to instruct Opal to show you a list of robots.txt and llms.txt configurations and to save new configurations.

Installation

Install the Stott.Optimizely.RobotsHandler package into your website.

Startup.cs

You need to ensure the following lines are added to the startup class of your solution:

public void ConfigureServices(IServiceCollection services)
{
    services.AddRobotsHandler();
}

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    app.UseRobotsHandler();

    app.UseEndpoints(endpoints =>
    {
        endpoints.MapContent();
        endpoints.MapControllers();
    });
}

The call to services.AddRobotsHandler() sets up the dependency injection requirements for the RobotsHandler solution and is required to ensure the solution works as intended. This works by following the Services Extensions pattern defined by Microsoft.

The call to app.UseRobotsHandler() sets up the middleware required to create the X-Robots-Tag header

The call to endpoints.MapControllers(); ensures that the routing for the administration page, assets and robots.txt are correctly mapped.

Razor Files

In the _ViewImports.cshtml file you will need to add the following line to include meta robots tag helper.

@addTagHelper *, Stott.Optimizely.RobotsHandler

Program.cs

As this package includes static files such as JS and CSS files within the Razor Class Library, your solution must be configured to use Static Web Assets. This is done by adding webBuilder.UseStaticWebAssets(); to your Program.cs as follows:

Host.CreateDefaultBuilder(args)
    .ConfigureCmsDefaults()
    .ConfigureWebHostDefaults(webBuilder =>
    {
        webBuilder.UseStartup<Startup>();
        webBuilder.UseStaticWebAssets();
    });

You can read more about shared assets in Razor Class Libraries here: Create reusable UI using the Razor class library project in ASP.NET Core

Adding Robots Admin to the menus.

This solution also includes an implementation of IMenuProvider which ensures that the Robots Handler administration pages are included in the CMS Admin menu under the title of "Robots". You do not have to do anything to make this work as Optimizely CMS will scan and action all implementations of IMenuProvider.

Authorisation Configuration

The configuration of the module has some scope for modification by providing configuration in the service extension methods. The provision of authorizationOptions is optional in the following example.

Example:

services.AddRobotsHandler(authorizationOptions => 
{
    authorizationOptions.AddPolicy(RobotsConstants.AuthorizationPolicy, policy =>
    {
        policy.RequireRole("WebAdmins");
    });
});

If the authorizationOptions is not provided, then any of the following roles will be required by default:

• CmsAdmins
• Administrator
• WebAdmins

Authentication With Optimizely Opti ID

If you are using the new Optimizely Opti ID package for authentication into Optimizely CMS and the rest of the Optimizely One suite, then you will need to define the authorizationOptions for this module as part of your application start up. This should be a simple case of adding policy.AddAuthenticationSchemes(OptimizelyIdentityDefaults.SchemeName); to the authorizationOptions as per the example below.

services.AddRobotsHandler(authorizationOptions =>
{
    authorizationOptions.AddPolicy(RobotsConstants.AuthorizationPolicy, policy =>
    {
        policy.AddAuthenticationSchemes(OptimizelyIdentityDefaults.SchemeName);
        policy.RequireRole("WebAdmins");
    });
});

Contributing

I am open to contributions to the code base. The following rules should be followed:

1. Contributions should be made by Pull Requests.
2. All commits should have a meaningful message.
3. All commits should have a reference to your GitHub user.
4. Ideally, all new changes should include appropriate unit test coverage.

Contributors

Thank you for your feedback and contributions go to the following members of the community: