Aiimi Insight Engine Disco
User GuidesAiimi
  • Introducing Aiimi Insight Engine
  • Architecture
    • Overview and Key Concepts
    • Hosting Options
    • Architecture and How It Works
      • Agent Servers
        • Security Agent
        • Source Agent
        • Content Agent
        • Enrichment Agent
        • Job Agent
        • OCR Agent
        • Migration Agent
        • Tika Agent
      • Repository
        • Data Node
        • Proxy Node
        • Kibana Node
      • Gateway and User Interface
      • Document and Data Sources
    • Deployment Options
      • Development Environments
      • Testing and QA Environments
      • Production Environments
      • Disaster Recovery Environments
    • Security
      • Source Systems
      • Firewalling
      • Agent Servers
        • Security Agent
        • Source Agent
        • Content Agent
        • Enrichment Agent
        • Job Agent
        • OCR Agent
        • Migration Agent
        • Tika Agent
      • Repository
      • Gateway (Web Server)
      • Tools & Utilities
  • Installation
    • Elasticsearch Installation (Windows)
    • Aiimi Insight Engine Installation (Windows)
    • Python REST Service Install and Configuration
      • Installation and Setup
        • Offline Huggingface Set-up
        • Configuration of Logging
        • Running as a Service (Windows)
        • Using SSL
        • Performance and Concurrency
        • Security
      • Communicating with the Python REST Service
      • Step Configuration
        • Classify
        • Entity Mapper
        • Huggingface NER
        • Language Detection
        • Phrases and Topics
        • Summary
        • Spacyner
    • HTML Cleaner Service
  • security
    • Users
      • User Authentication
      • Code of Conduct
      • App Access and Role-Based Access
    • Data & Documents
      • Document and Data Access Control Lists
      • Progressive Access and Privileged Access
        • Progressive Access
        • Privileged Access
      • Source Scope and Visibility within Apps
      • Classification
      • Advanced Security Controls - Security Classifications
      • Risk Ratings
      • Redaction and Anonymisation
      • Banned Words
      • Mark as Sensitive
      • Audit Controls
  • Control Hub
    • Configurations
      • Managing Configurations
        • Edit and Delete
        • Find a Configuration
        • Monitoring Stats
        • Stop a Configuration
        • Refreshing
      • Security Configurations
        • Configuring Security
        • Active Directory
        • Azure Active Directory
        • BuiltinSecurity
        • Security Sync
        • Security Agents
        • Security Scheduling
      • Source Configurations
        • General
        • Source
          • Azure Blob Storage
          • BBC Monitoring
          • Big Query Cataloguer
          • BIM360
          • CSV Data Loader
          • Confluence
          • Content Server
          • Data File Cataloguer
          • Document Store
          • Dropbox
          • Exchange 365
          • Filesystem
          • Google Bucket
          • JSON Data Loader
          • ODBC Data Loader
          • PowerBi Cataloguer
          • Reuters Connect
          • ShareFile
          • SharePoint
          • SharePoint Legacy
          • SQL Server Cataloguer
          • Websites
          • XML Data Loader
        • Crawl
        • Source Agents
        • Source Scheduling
        • Advanced
      • Enrichment Configurations
        • Creating a Pipeline
          • General
          • Steps
            • Tika Text Extraction
            • External Links
            • Delete
            • Copy
            • Text Cleaner
            • Data Rule Processor
            • Checksum
            • OcrRest
            • AccessMiner
            • CAD Extractor
            • Trie Entity Extractor
            • PCI Extractor
            • Email Extractor
            • Geotag
            • Google Vision Extractor
            • Google NLP Extractor
            • Metrics Calculation
            • Microsoft Vision Extractor
            • Entity Rule Processor
            • Anonymiser
            • Set Document Risk
            • Content Retrieval
            • REST
          • Filters
          • Agents
          • Schedule
          • Advanced
      • OCR Engine
      • Job Configurations
        • General
        • Job
          • AutomatedSearchJob
          • Command Job
          • ElasticJob
          • Extended Metrics Job
          • GoogleVaultSAR
          • Nightly Events Processor Job
          • Notifications Processor Job
          • Portal Sync Job
          • Purge Job
          • Text Content Merge Job
        • Output
        • Agents
        • Scheduling
      • Migration Configuration
        • General
        • Filter
        • Metadata Mappings
        • Agents
        • Scheduling
        • Advanced
      • Content Server
    • Credentials
      • Create a Credential
      • Find a Credential
      • Edit a Credential
      • Delete a Credential
    • Entities
      • Create a Group
      • Create an Entity
      • Managing Entities
      • Import or Export Entities
    • Models
      • Create a New Model
      • Find a Model
      • Enable or Disable a Model
    • Featured Links
    • Global Settings
      • General
        • Thumbnails
        • Windows Authentication
        • SAML2 Authentication
        • ADFS
        • Stackdriver
        • Document Recommendations
        • Searchable PDF Storage
        • Versioning
        • Results
        • Marking Useful Results
        • Folder Browsing
        • Cascading Search
        • Search Suggestions
        • Delve Settings
        • Collections
        • Miscellaneous
      • Application Access
      • Search Relevancy
        • Core Settings
        • Makers Algorithm
        • Filename Boost Layer
        • Minimum Matching Terms Filter
        • Field Boost
        • Modified Date Boosting
        • Hit Highlighting
        • Why My Search Matched
        • Data Search Strategy
      • Search Performance
        • Accurate Totals for Pagination
        • Source cards
        • Facets
      • Filtering
      • Presets
        • File Groups
        • Security Groups
      • Code of Conduct
      • Metrics
      • Viewer
      • SAR
        • Importing Data For A SAR
        • Getting SAR data from Google Vault
        • Redacting Information
        • SAR Access
      • Privacy Portal
        • Activate the Privacy Portal
        • Disclosure
        • Submit SARs From The Privacy Portal
        • Email Delivery Settings
          • Delivery Settings
          • Brand Settings
          • Customise Emails
        • SMS Delivery Settings
        • Requestor Message Limit
        • Attachment Configuration
        • Password Configuration
        • File Scanner Configurator
      • Visualisations
        • Related Result Connections Diagram
        • Event Timeline
        • Create and Modified Date Activity Chart
        • Relationship Map
      • Notifications
      • Map View
      • App
      • Theming
        • Details
        • Logo
        • Background Images
        • Colours
      • Related Results
      • OData API
      • Bulk Search
        • Managing a Bulk Search
    • User Settings
    • Stats
  • API Guides
    • Insight API Guide
      • Swagger Documentation
      • Trying Some Endpoints
      • Search Filter
      • Hits / Items
      • Inspecting REST Calls
    • Data Science API Guide
      • REST Interface
        • Login
        • Datasets
        • Fields
        • Field Statistics
        • Search
        • Scroll
        • Update
      • Python Wrapper
        • Login
        • Datasets
        • Fields
        • Field Statistics
        • Search
        • Query Builders
        • Scroll
        • Scroll Search
        • Update Single Document
        • Bulk Update
    • Creating a Native Enrichment Step
      • Creating an Enrichment Step
        • Creating the Core Classes
        • Extending our Enrichment Step
        • Adding a Configuration Template
        • Adding the Enrichment Step
        • Creating an Enrichment Pipeline
      • Other Tasks
        • Entities, Metadata and Data
        • Accessing the Repository
      • Example Code
      • Troubleshooting
    • Creating a Python Enrichment Step
      • Creating an Enrichment Step
        • Running the Example from Command Line
        • Running the Example
      • Creating Your Own Step
      • Adding or Changing Entities, Metadata
  • whitepapers and explainers
    • Methods for Text Summarization
      • Application
      • Technology Methods
      • Commercial Tools
      • Key Research Centres
      • Productionisation
      • Related Areas of Text Analytics
      • Conclusion
      • References
Powered by GitBook
On this page
  • Configuration
  • Enrichment Step
  • Enrichment Step Factory
  • Creating the Test Console App
  1. API Guides
  2. Creating a Native Enrichment Step
  3. Creating an Enrichment Step

Creating the Core Classes

Several classes need to be generated for configuration, the enrichment step itself, and the factory that creates it on demand.

Configuration

Create a class called HelloWorldESConfiguration for the configuration. This class will extend the Enrichment Step Configuration class, and also contain a class object that contains enrichment step specific configuration. Here we have a single ‘Text’ property that will contain text to append to the text content on an Aiimi Insight Engine item.

using InsightMaker.Core.Types.Config;

namespace InsightMaker.Enrichment.HelloWorldES
{
    public class HelloWorldESConfiguration : EnrichmentStepConfiguration
    {
        public HelloWorldESConfiguration()
        {
            base.Type = "HelloWorldES";
        }

        public HelloWorldESSpecificConfiguration HelloWorldES { get; set; } 
            = new HelloWorldESSpecificConfiguration();
    }

    public class HelloWorldESSpecificConfiguration
    {
        public string Text { get; set; } = "";
    }
}

Enrichment Step

Create a class called HelloWorldESStep for the actual enrichment step. This class will implement the Enrichment Step interface. For the minute the class will simply print ‘Hello World’ to the console window.

using InsightMaker.Core.Types;

namespace InsightMaker.Enrichment.HelloWorldES
{
    public class HelloWorldESStep : IEnrichmentStep
    {
        protected static readonly log4net.ILog log = 
            log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);

        private HelloWorldESConfiguration configuration;
        private CancellationToken cancellationToken;

        public HelloWorldESStep(HelloWorldESConfiguration configuration, 
                                CancellationToken token)
        {
            this.configuration = configuration;
            this.cancellationToken = token;
        }

        public void Run(EnrichmentWorkItem work)
        {
            log.Debug("Hello World");
        }
    }
}

Enrichment Step Factory

Create a class called HelloWorldESFactory that will be responsible for creating the enrichment step.

using InsightMaker.Core.Types.Plugin;
using InsightMaker.Core.Types;
using InsightMaker.Core.Types.Config;
using System.ComponentModel.Composition;

namespace InsightMaker.Enrichment.HelloWorldES
{
    [Export(typeof(IEnrichmentStepFactory))]
    [ExportMetadata("Type", "HelloWorldES")]
    [ExportMetadata("Description", "Hello World Enrichment Step")]
    [ExportMetadata("DisplayName", "HelloWorldES")]
    [ExportMetadata("Chargeable", false)]
    [ExportMetadata("ConfigurationType", typeof(HelloWorldESConfiguration))]
    [ExportMetadata("StepGroup", StepGroup.Entities)]
    [ExportMetadata("DisplayOrder", 40)]
    [ExportMetadata("Inputs", EnrichmentStepDataType.Text)]
    [ExportMetadata("Outputs", EnrichmentStepDataType.Text)]
    [ExportMetadata("ThirdParty", Dependency.None)]
    public class HelloWorldESFactory : IEnrichmentStepFactory
    {
        public IEnrichmentStep BuildEnrichmentStep(EnrichmentStepConfiguration       configuration, CancellationToken cancellationToken)
        {
            return new HelloWorldESStep((HelloWorldESConfiguration)configuration, 
                                        cancellationToken);
        }

        public object GetDefaultConfiguration()
        {
            return new HelloWorldESConfiguration();
        }
    }
}

Creating the Test Console App

We will now create the console app that we use to iterate development and test the enrichment step.

Edit the Program.cs file (or create a Program.cs file) in the console app project.

using InsightMaker.Core.Types;

[assembly: log4net.Config.XmlConfigurator(ConfigFile = "log4net.config")]

namespace InsightMaker.Enrichment.HelloWorldES.Test
{
    class Program
    {
        protected static readonly log4net.ILog log = 
            log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);

        static void Main(string[] args)
        {
            log.Info("InsightMaker.Enrichment.HelloWorldES.Test");
 
            var workItem = new EnrichmentWorkItem(new IndexedFile
            {
                Uri = "does not matter for testing",
                TextContent = "This is our documents text content."
            });

            var helloWorldESConfiguration = new HelloWorldESConfiguration();
            helloWorldESConfiguration.HelloWorldES.Text = "Hello World";

            var helloWorldESFactory = new HelloWorldESFactory();
            var helloWorldESStep = (HelloWorldESStep)helloWorldESFactory.BuildEnrichmentStep(helloWorldESConfiguration, 
                                                                                             new System.Threading.CancellationToken());

            try
            {
                helloWorldESStep.Run(workItem);
            }
            catch (Exception e)
            {
                Console.WriteLine(e.Message);
                Console.WriteLine(e.StackTrace);
            }
        }
    }
} 

Before attempting to run this, you will also need to add a log4net.config file. Add the following to the project, being sure to select ‘Copy to Output Directory’ on the files properties.

<?xml version="1.0" encoding="utf-8" ?>
<log4net>
  <appender name="RollingFile" type="log4net.Appender.RollingFileAppender">
    <file value="c:/temp/logs/InsightMaker.Enrichment.HelloWorldESStep.Test.log" />
    <appendToFile value="true" />
    <rollingStyle value="Size" />
    <maxSizeRollBackups value="10" />
    <maximumFileSize value="10240KB" />
    <staticLogFileName value="true" />
    <layout type="log4net.Layout.PatternLayout">
      <conversionPattern value="%date [%-2t] %-5level %logger - %message%newline" />
    </layout>
  </appender>
  <appender name="ConsoleAppender" type="log4net.Appender.ConsoleAppender">
    <param name="Threshold" value="DEBUG" />
    <layout type="log4net.Layout.PatternLayout">
      <param name="ConversionPattern" value="[%-2t] %message%newline" />
    </layout>
  </appender>
  <root>
    <level value="DEBUG" />
    <appender-ref ref="RollingFile" />
    <appender-ref ref="ConsoleAppender" />
  </root>
</log4net>

Now run the console app and you should see the following.

PreviousCreating an Enrichment StepNextExtending our Enrichment Step