magic.data.cql 14.0.0

NoSQL based IO, caching, and logging adapters for Hyperlambda

This project contains alternative NoSQL file system services, implementing IFileService, IFolderService, and IStreamService, allowing you to use it as an interchangeable "virtual file system" for cases where you want to have 100% stateless magic instances. This is important if you're using Magic in a Kubernetes cluster or something similar, load balancing invocations, virtually resolving files and folders towards a virtual file system. If you take this path you'll have to configure your "appsettings.json" file such as illustrated further down in this document. The project also contains an ILogger implementation service you can use that will create log entries in a NoSQL based storage of your choice, in addition to an IMagicCache implementation service allowing you to use a NoSQL based out of process cache implementation. See further down in this document for details about how to configure these parts.

Configuration

The primary configuration for the project to apply for your "appsettings.json" file can be found below. Notice, although you can create as many NoSQL cluster connection settings as you wish, and use these in your own code, the IO, caching, and logging services requires you to use generic as your cluster name, and you cannot change this.

{
  "magic": {
    "cql": {
      "generic": {
        "host": "127.0.0.1",
        "credentials": {
          "username": "xxx",
          "password": "xxx"
        },
        "port": 12345
      }
    }
  }
}

Notice - The "credentials" parts above are optional, and can be ommitted if you don't require authentication to connect to your database. You can also provide multiple hosts as contact points, by separating multiple IP addresses or hosts by a comma (,). The above "port" parts is also optional.

The above configures the adapter to use 127.0.0.1 as the host for your contact point or cluster. To configure the adapter to store files and folders inside of its CQL based database, you can alternatively add something such as follows to your "appsettings.json" file.

{
  "magic": {
    "io": {
      "file-service": "magic.data.cql.io.CqlFileService",
      "folder-service": "magic.data.cql.io.CqlFolderService",
      "stream-service": "magic.data.cql.io.CqlStreamService"
    }
  }
}

Notice - This project also includes a "mixed" service implementation, storing system files and folders in the file system while storing dynamic files and folders in a NoSQL database of your choice. These can be found in the same namespace, but are named "MixedXXX" instead of using the above names.

If you want to use a CQL based log implementation, you'll have to configure Magic to use the NoSQL ILogger service such as follows.

{
  "magic": {
    "logging": {
      "service": "magic.data.cql.logging.Logger"
    }
  }
}

If you want to use a CQL based caching implementation, you'll have to configure Magic to use the NoSQL IMagicCache service such as follows.

{
  "magic": {
    "caching": {
      "service": "magic.data.cql.caching.Caching"
    }
  }
}

Schemas

To use the alternative CQL based file storage system you'll have to create your "magic_files" keyspace and its "files" table as follows.

create keyspace if not exists magic_files with replication = { 'class': 'NetworkTopologyStrategy', 'replication_factor': 5 };

use magic_files;

create table if not exists files(
   tenant text,
   cloudlet text,
   folder text,
   filename text,
   content blob,
   primary key((tenant, cloudlet), folder, filename));

To use the alternative CQL based logging implementation you'll have to create your "magic_log" keyspace and its "log" table as follows.

create keyspace if not exists magic_log with replication = { 'class': 'NetworkTopologyStrategy', 'replication_factor': 3 };

use magic_log;

create table if not exists log(
   tenant text,
   cloudlet text,
   created timeuuid,
   type text,
   content text,
   exception text,
   meta frozen<map<text, text>>,
   primary key((tenant, cloudlet), created)) with clustering order by (created desc);

alter table log with default_time_to_live = 1209600;

Notice - The above setting for TTL implies log items will be automatically evicted after 14 days, since 1,209,600 seconds implies 14 days. Depending upon your needs you might want to increase or decrease this setting. However, to avoid exhausting your hard drive space over time on your server, we suggest you do not entirely remove the TTL setting.

To use the alternative CQL based caching implementation you'll have to create your "magic_cache" keyspace and its "cache" table as follows.

create keyspace if not exists magic_cache with replication = { 'class': 'NetworkTopologyStrategy', 'replication_factor': 3 };

use magic_cache;

create table if not exists cache(
   tenant text,
   cloudlet text,
   key text,
   value text,
   primary key((tenant, cloudlet), key));

Adding existing files into NoSQL database

The following Hyperlambda will insert all your existing files and folders into your cluster keyspace, allowing you to play around with an existing CQL file system implementation. Notice, you'll have to change the [.tenant] and [.cloudlet] values to resemble the absolute root folder for your Magic backend. The values in the file below is obviously just an example of how it might look like if you've got the files on a Mac within your "Documents" folder. The "tenant" and the "cloudlet" parts are resolved by doing a string split operation on your magic:io:root-files root folder upon the last (/) found in your folder - Implying if you use the Docker images with the default configuration the "tenant" part would become "magic" and the "cloudlet" part would become "files", since the Docker images stores files within the "/magic/files/" folder.

/*
 * Inserts all dynamic files and folders into the magic CQL database.
 */
cql.connect:[generic|magic_files]

   /*
    * The root folder where your Magic backend is running.
    */
   .tenant:Users
   .cloudlet:"thomashansen/Documents/projects/magic/magic/backend"

   /*
    * Inserting root folder.
    */
   cql.execute:"insert into files (tenant, cloudlet, folder, filename) values (:tenant, :cloudlet, '/files/', '')"
      tenant:x:@.tenant
      cloudlet:x:@.cloudlet

   /*
    * Inserting appsettings.json and its folder.
    */
   config.load
   convert:x:-
      type:bytes
   cql.execute:"insert into files (tenant, cloudlet, folder, filename, content) values (:tenant, :cloudlet, '/config/', 'appsettings.json', :config)"
      tenant:x:@.tenant
      cloudlet:x:@.cloudlet
      config:x:@convert
   cql.execute:"insert into files (tenant, cloudlet, folder, filename) values (:tenant, :cloudlet, '/config/', '')"
      tenant:x:@.tenant
      cloudlet:x:@.cloudlet

   /*
    * Inserting folders.
    */
   io.folder.list-recursively:/
      display-hidden:true
   for-each:x:-/*

      strings.concat
         .:/files
         get-value:x:@.dp/#
      
      cql.execute:"insert into files (tenant, cloudlet, folder, filename) values (:tenant, :cloudlet, :folder, '')"
         tenant:x:@.tenant
         cloudlet:x:@.cloudlet
         folder:x:@strings.concat

   /*
    * Inserting files.
    */
   io.file.list-recursively:/
      display-hidden:true
   for-each:x:-/*

      io.file.load.binary:x:@.dp/#
   
      strings.split:x:@.dp/#
         .:/
      unwrap:x:+
      .filename:x:@strings.split/0/-
      remove-nodes:x:@strings.split/0/-
      strings.join:x:@strings.split/*
         .:/
      strings.concat
         .:/files/
         get-value:x:@strings.join
         .:/
      strings.replace:x:-
         .://
         .:/
      cql.execute:"insert into files (tenant, cloudlet, folder, filename, content) values (:tenant, :cloudlet, :folder, :filename, :content)"
         tenant:x:@.tenant
         cloudlet:x:@.cloudlet
         folder:x:@strings.replace
         filename:x:@.filename
         content:x:@io.file.load.binary

remove-nodes:x:../**/io.folder.list-recursively/*
remove-nodes:x:../**/io.file.list-recursively/*

Internals

This project is created to be a multi-tenant solution, implying multiple users, and/or cloudlets, can use the same physical database for both files, log entries, and cache entries. However, this implies you are able to correctly resolve the "tenant" parts and the "cloudlet" parts for all the above tables. How to achieve this is currently beyond the scope of this document, but our suggestion is to create some sort of IRootResolver service implementation that is dynamically created according to the Host HTTP header of requests.

Project website

The source code for this repository can be found at github.com/polterguy/magic.data.cql, and you can provide feedback, provide bug reports, etc at the same place.

Quality gates