Aiimi Insight Engine Florina
User GuidesAiimi
  • Introducing Aiimi Insight Engine
  • Architecture
    • Overview and Key Concepts
    • Search Flows
      • Search Flow Types
      • Smart Filtering
      • Query and Prompt Classification
      • Search Algorithms
      • Extractive and Generative Models
    • Hosting Options
    • Architecture and How It Works
      • Agent Servers
        • Security Agent
        • Source Agent
        • Content Agent
        • Enrichment Agent
        • Job Agent
        • OCR Agent
        • Migration Agent
        • Tika Agent
      • Repository
        • Data Node
        • Proxy Node
        • Kibana Node
      • Gateway and User Interface
      • Document and Data Sources
    • Deployment Options
    • Security
      • User Security
      • Data and Document Security
        • Progressive Access
        • Privileged Access
      • Source System Security
      • Firewalling
      • Agent Servers
        • Security Agent
        • Source Agent
        • Content Agent
        • Enrichment Agent
        • Job Agent
        • OCR Agent
        • Migration Agent
        • Tika Agent
      • Repository
      • Gateway (Web Server)
      • Tools & Utilities
  • Installation
    • Elastic and Kibana Install (Windows)
    • Aiimi Insight Engine Installation (Windows)
      • Installation Security
    • AI Services
      • Prerequisites
      • AI Enrichment Service
        • Installation and Setup
        • Enabling Enrichment Steps
        • Using AI Enrichment Steps
        • Performance and Concurrency
      • AI Model Service
        • Installation and Setup
        • Enabling Providers
        • Private Generative AI
        • Azure Open AI
      • Configuration of Logging
      • Offline Set-up of Models
      • Using SSL
      • Running as a Service (Windows)
      • Using GPUs
      • AI and Semantic Search Set Up
        • Open & Closed Book AI
        • Semantic Search
          • Vectors for Semantic Search
          • Source Configuration
          • Sentence Transformer Models
          • Enrichment
          • Kibana
          • Final Search Flow
    • HTML Cleaner Service
  • Control Hub
    • Configurations
      • Config Management
      • Security Configurations
        • Security - General
        • Security - Source
          • Active Directory
          • Atlassian
          • Azure Active Directory
          • Builtin Security
          • Miro Security
          • Google Directory
          • Slack Security
        • Security - Sync
        • Security - Agents
        • Security - Scheduling
      • Source Configurations
        • Source - General
        • Source - Source
          • Alfresco Kafka
          • Azure Blob Storage
          • BBC Monitoring
          • Big Query Cataloguer
          • BIM360
          • CSV Data Loader
          • Confluence
          • Content Server
          • Data File Cataloguer
          • Document Store
          • DocuSign
          • Dropbox
          • Exchange 365
          • Filesystem
          • Google Bucket
          • Google Drive
          • Google Vault
          • Jira
          • JSON Data Loader
          • Livelink
          • Microsoft Teams
          • Mimecast
          • Miro
          • ODBC Data Loader
          • PowerBi Cataloguer
          • Reuters Connect
          • ShareFile
          • SharePoint
          • SQL Server Cataloguer
          • Slack
          • Versioned Document Store
          • Websites
          • XML Data Loader
        • Source - Crawl
        • Source - Agents
        • Source - Schedule
        • Source - Advanced
      • Enrichment Configurations
        • Creating a Pipeline
          • General
          • Steps
            • AccessMiner
            • AI Classification
            • Anonymiser
            • CAD Extractor
            • Checksum
            • Content Retrieval
            • Copy
            • Data Rule Processor
            • Delete
            • Email Extractor
            • Entity Rule Processor
            • External Links
            • Geotag
            • Google NLP Extractor
            • Google Vision Extractor
            • Metrics Calculation
            • Microsoft Vision Extractor
            • OcrRest
            • Office Metadata
            • PCI Extractor
            • REST
            • Set Document Risk
            • Text Cleaner
            • Tika Text Extraction
            • Trie Entity Extractor
            • Update Metadata
          • Filters
          • Agents
          • Schedule
          • Advanced
      • OCR Engine
      • Job Configurations
        • General
        • Job
          • AutomatedSearchJob
          • Command Job
          • ElasticJob
          • Extended Metrics Job
          • GoogleVaultSAR
          • Google Drive Last Access Date
          • Nightly Events Processor Job
          • Notifications Processor Job
          • Portal Sync Job
          • Purge Job
          • Text Content Merge Job
        • Output
        • Agents
        • Scheduling
      • Migration Configuration
        • General
        • Filter
        • Metadata Mappings
        • Agents
        • Scheduling
        • Advanced
    • Credentials
      • Create a Credential
      • Find a Credential
      • Edit a Credential
      • Delete a Credential
    • Mappings
      • Entities
        • Managing Groups
        • Create an Entity
        • Managing Entities
      • Models
        • Create a New Model
        • Find a Model
        • Enable or Disable a Model
      • Vectors
      • Rank Features
    • Featured Links
    • AI Settings
      • Classifications
      • Class
      • Class Rules
      • AI Classification
    • Global Settings
      • General
        • Stackdriver
        • Document Recommendations
        • Searchable PDF Storage
        • Versioning
        • Results
        • Marking Useful Results
        • Folder Browsing
        • Cascading Search
        • Search Suggestions
        • Delve Settings
        • Miscellaneous
      • Authentication
      • Application Access
      • Search Relevancy
        • Core Settings
        • Makers Algorithm
        • Filename Boost Layer
        • Minimum Matching Terms Filter
        • Field Boost
        • Modified Date Boosting
        • Hit Highlighting
        • Why My Search Matched
        • Data Search Strategy
      • Search Performance
      • Filtering
      • Thumbnails
      • Presets
      • Code of Conduct
      • Metrics
      • Viewer
        • Redacting Information
        • Watermarking
      • SAR
        • Importing Data For A SAR
        • SAR Disclosure Document Storage
        • Getting SAR data from Google Vault
        • SAR Access
        • SAR File Status
      • Collections
      • Disclosure Portal
        • Disclosure Portal Set Up
        • SARs From The Portal
        • Email Delivery Settings
          • Delivery Settings
          • Brand Settings
          • Customise Emails
        • SMS Delivery Settings
        • Requestor Message Limit
        • Attachment Configuration
        • Password Configuration
        • File Scanner Configurator
      • Visualisations
        • Related Result Connections Diagram
        • Event Timeline
        • Timeline Lens Activity Chart
        • Relationship Map
      • Notifications
      • Map Lens
      • App
      • Theming
        • General
        • Layout
        • Colours
      • User Avatar
      • Related Results
      • OData API
      • Bulk Search
        • Managing a Bulk Search
      • Search Flows
        • Create a Search Flow
          • General
          • Query Classification Step
          • Search Steps
          • Model Steps
      • Uploads
      • Security
    • User Settings
    • Stats
      • Data Views
  • API Guides
    • Insight API Guide
      • Swagger Documentation
      • Trying Some Endpoints
      • Search Filter
      • Hits / Items
      • Inspecting REST Calls
    • Data Science API Guide
      • REST Interface
        • Login
        • Datasets
        • Fields
        • Field Statistics
        • Search
        • Scroll
        • Update
      • Python Wrapper
        • Login
        • Datasets
        • Fields
        • Field Statistics
        • Search
        • Query Builders
        • Scroll
        • Scroll Search
        • Update Single Document
        • Bulk Update
    • Creating a Native Enrichment Step
      • Creating an Enrichment Step
        • Creating the Core Classes
        • Extending our Enrichment Step
        • Adding a Configuration Template
        • Adding the Enrichment Step
        • Creating an Enrichment Pipeline
      • Other Tasks
        • Entities, Metadata and Data
        • Accessing the Repository
      • Example Code
      • Troubleshooting
    • Creating a Python Enrichment Step
      • Creating an Enrichment Step
        • Running the Example from Command Line
        • Running the Example
      • Creating Your Own Step
      • Adding or Changing Entities, Metadata
  • whitepapers and explainers
    • From a Billion To One – Mastering Relevancy
    • Methods for Text Summarization
      • Application
      • Technology Methods
      • Commercial Tools
      • Key Research Centres
      • Productionisation
      • Related Areas of Text Analytics
      • Conclusion
      • References
Powered by GitBook
On this page
  • Configuration
  • Enrichment Step
  • Enrichment Step Factory
  • Creating the Test Console App
  1. API Guides
  2. Creating a Native Enrichment Step
  3. Creating an Enrichment Step

Creating the Core Classes

Several classes need to be generated for configuration, the enrichment step itself, and the factory that creates it on demand.

Configuration

Create a class called HelloWorldESConfiguration for the configuration. This class will extend the Enrichment Step Configuration class, and also contain a class object that contains enrichment step specific configuration. Here we have a single ‘Text’ property that will contain text to append to the text content on an Aiimi Insight Engine item.

using InsightMaker.Core.Types.Config;

namespace InsightMaker.Enrichment.HelloWorldES
{
    public class HelloWorldESConfiguration : EnrichmentStepConfiguration
    {
        public HelloWorldESConfiguration()
        {
            base.Type = "HelloWorldES";
        }

        public HelloWorldESSpecificConfiguration HelloWorldES { get; set; } 
            = new HelloWorldESSpecificConfiguration();
    }

    public class HelloWorldESSpecificConfiguration
    {
        public string Text { get; set; } = "";
    }
}

Enrichment Step

Create a class called HelloWorldESStep for the actual enrichment step. This class will implement the Enrichment Step interface. For the minute the class will simply print ‘Hello World’ to the console window.

using InsightMaker.Core.Types;

namespace InsightMaker.Enrichment.HelloWorldES
{
    public class HelloWorldESStep : IEnrichmentStep
    {
        protected static readonly log4net.ILog log = 
            log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);

        private HelloWorldESConfiguration configuration;
        private CancellationToken cancellationToken;

        public HelloWorldESStep(HelloWorldESConfiguration configuration, 
                                CancellationToken token)
        {
            this.configuration = configuration;
            this.cancellationToken = token;
        }

        public void Run(EnrichmentWorkItem work)
        {
            log.Debug("Hello World");
        }
    }
}

Enrichment Step Factory

Create a class called HelloWorldESFactory that will be responsible for creating the enrichment step.

using InsightMaker.Core.Types.Plugin;
using InsightMaker.Core.Types;
using InsightMaker.Core.Types.Config;
using System.ComponentModel.Composition;

namespace InsightMaker.Enrichment.HelloWorldES
{
    [Export(typeof(IEnrichmentStepFactory))]
    [ExportMetadata("Type", "HelloWorldES")]
    [ExportMetadata("Description", "Hello World Enrichment Step")]
    [ExportMetadata("DisplayName", "HelloWorldES")]
    [ExportMetadata("Chargeable", false)]
    [ExportMetadata("ConfigurationType", typeof(HelloWorldESConfiguration))]
    [ExportMetadata("StepGroup", StepGroup.Entities)]
    [ExportMetadata("DisplayOrder", 40)]
    [ExportMetadata("Inputs", EnrichmentStepDataType.Text)]
    [ExportMetadata("Outputs", EnrichmentStepDataType.Text)]
    [ExportMetadata("ThirdParty", Dependency.None)]
    public class HelloWorldESFactory : IEnrichmentStepFactory
    {
        public IEnrichmentStep BuildEnrichmentStep(EnrichmentStepConfiguration       configuration, CancellationToken cancellationToken)
        {
            return new HelloWorldESStep((HelloWorldESConfiguration)configuration, 
                                        cancellationToken);
        }

        public object GetDefaultConfiguration()
        {
            return new HelloWorldESConfiguration();
        }
    }
}

Creating the Test Console App

We will now create the console app that we use to iterate development and test the enrichment step.

Edit the Program.cs file (or create a Program.cs file) in the console app project.

using InsightMaker.Core.Types;

[assembly: log4net.Config.XmlConfigurator(ConfigFile = "log4net.config")]

namespace InsightMaker.Enrichment.HelloWorldES.Test
{
    class Program
    {
        protected static readonly log4net.ILog log = 
            log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);

        static void Main(string[] args)
        {
            log.Info("InsightMaker.Enrichment.HelloWorldES.Test");
 
            var workItem = new EnrichmentWorkItem(new IndexedFile
            {
                Uri = "does not matter for testing",
                TextContent = "This is our documents text content."
            });

            var helloWorldESConfiguration = new HelloWorldESConfiguration();
            helloWorldESConfiguration.HelloWorldES.Text = "Hello World";

            var helloWorldESFactory = new HelloWorldESFactory();
            var helloWorldESStep = (HelloWorldESStep)helloWorldESFactory.BuildEnrichmentStep(helloWorldESConfiguration, 
                                                                                             new System.Threading.CancellationToken());

            try
            {
                helloWorldESStep.Run(workItem);
            }
            catch (Exception e)
            {
                Console.WriteLine(e.Message);
                Console.WriteLine(e.StackTrace);
            }
        }
    }
} 

Before attempting to run this, you will also need to add a log4net.config file. Add the following to the project, being sure to select ‘Copy to Output Directory’ on the files properties.

<?xml version="1.0" encoding="utf-8" ?>
<log4net>
  <appender name="RollingFile" type="log4net.Appender.RollingFileAppender">
    <file value="c:/temp/logs/InsightMaker.Enrichment.HelloWorldESStep.Test.log" />
    <appendToFile value="true" />
    <rollingStyle value="Size" />
    <maxSizeRollBackups value="10" />
    <maximumFileSize value="10240KB" />
    <staticLogFileName value="true" />
    <layout type="log4net.Layout.PatternLayout">
      <conversionPattern value="%date [%-2t] %-5level %logger - %message%newline" />
    </layout>
  </appender>
  <appender name="ConsoleAppender" type="log4net.Appender.ConsoleAppender">
    <param name="Threshold" value="DEBUG" />
    <layout type="log4net.Layout.PatternLayout">
      <param name="ConversionPattern" value="[%-2t] %message%newline" />
    </layout>
  </appender>
  <root>
    <level value="DEBUG" />
    <appender-ref ref="RollingFile" />
    <appender-ref ref="ConsoleAppender" />
  </root>
</log4net>

Now run the console app and you should see the following.

PreviousCreating an Enrichment StepNextExtending our Enrichment Step