Aiimi Insight Engine Disco
User GuidesAiimi
  • Introducing Aiimi Insight Engine
  • Architecture
    • Overview and Key Concepts
    • Hosting Options
    • Architecture and How It Works
      • Agent Servers
        • Security Agent
        • Source Agent
        • Content Agent
        • Enrichment Agent
        • Job Agent
        • OCR Agent
        • Migration Agent
        • Tika Agent
      • Repository
        • Data Node
        • Proxy Node
        • Kibana Node
      • Gateway and User Interface
      • Document and Data Sources
    • Deployment Options
      • Development Environments
      • Testing and QA Environments
      • Production Environments
      • Disaster Recovery Environments
    • Security
      • Source Systems
      • Firewalling
      • Agent Servers
        • Security Agent
        • Source Agent
        • Content Agent
        • Enrichment Agent
        • Job Agent
        • OCR Agent
        • Migration Agent
        • Tika Agent
      • Repository
      • Gateway (Web Server)
      • Tools & Utilities
  • Installation
    • Elasticsearch Installation (Windows)
    • Aiimi Insight Engine Installation (Windows)
    • Python REST Service Install and Configuration
      • Installation and Setup
        • Offline Huggingface Set-up
        • Configuration of Logging
        • Running as a Service (Windows)
        • Using SSL
        • Performance and Concurrency
        • Security
      • Communicating with the Python REST Service
      • Step Configuration
        • Classify
        • Entity Mapper
        • Huggingface NER
        • Language Detection
        • Phrases and Topics
        • Summary
        • Spacyner
    • HTML Cleaner Service
  • security
    • Users
      • User Authentication
      • Code of Conduct
      • App Access and Role-Based Access
    • Data & Documents
      • Document and Data Access Control Lists
      • Progressive Access and Privileged Access
        • Progressive Access
        • Privileged Access
      • Source Scope and Visibility within Apps
      • Classification
      • Advanced Security Controls - Security Classifications
      • Risk Ratings
      • Redaction and Anonymisation
      • Banned Words
      • Mark as Sensitive
      • Audit Controls
  • Control Hub
    • Configurations
      • Managing Configurations
        • Edit and Delete
        • Find a Configuration
        • Monitoring Stats
        • Stop a Configuration
        • Refreshing
      • Security Configurations
        • Configuring Security
        • Active Directory
        • Azure Active Directory
        • BuiltinSecurity
        • Security Sync
        • Security Agents
        • Security Scheduling
      • Source Configurations
        • General
        • Source
          • Azure Blob Storage
          • BBC Monitoring
          • Big Query Cataloguer
          • BIM360
          • CSV Data Loader
          • Confluence
          • Content Server
          • Data File Cataloguer
          • Document Store
          • Dropbox
          • Exchange 365
          • Filesystem
          • Google Bucket
          • JSON Data Loader
          • ODBC Data Loader
          • PowerBi Cataloguer
          • Reuters Connect
          • ShareFile
          • SharePoint
          • SharePoint Legacy
          • SQL Server Cataloguer
          • Websites
          • XML Data Loader
        • Crawl
        • Source Agents
        • Source Scheduling
        • Advanced
      • Enrichment Configurations
        • Creating a Pipeline
          • General
          • Steps
            • Tika Text Extraction
            • External Links
            • Delete
            • Copy
            • Text Cleaner
            • Data Rule Processor
            • Checksum
            • OcrRest
            • AccessMiner
            • CAD Extractor
            • Trie Entity Extractor
            • PCI Extractor
            • Email Extractor
            • Geotag
            • Google Vision Extractor
            • Google NLP Extractor
            • Metrics Calculation
            • Microsoft Vision Extractor
            • Entity Rule Processor
            • Anonymiser
            • Set Document Risk
            • Content Retrieval
            • REST
          • Filters
          • Agents
          • Schedule
          • Advanced
      • OCR Engine
      • Job Configurations
        • General
        • Job
          • AutomatedSearchJob
          • Command Job
          • ElasticJob
          • Extended Metrics Job
          • GoogleVaultSAR
          • Nightly Events Processor Job
          • Notifications Processor Job
          • Portal Sync Job
          • Purge Job
          • Text Content Merge Job
        • Output
        • Agents
        • Scheduling
      • Migration Configuration
        • General
        • Filter
        • Metadata Mappings
        • Agents
        • Scheduling
        • Advanced
      • Content Server
    • Credentials
      • Create a Credential
      • Find a Credential
      • Edit a Credential
      • Delete a Credential
    • Entities
      • Create a Group
      • Create an Entity
      • Managing Entities
      • Import or Export Entities
    • Models
      • Create a New Model
      • Find a Model
      • Enable or Disable a Model
    • Featured Links
    • Global Settings
      • General
        • Thumbnails
        • Windows Authentication
        • SAML2 Authentication
        • ADFS
        • Stackdriver
        • Document Recommendations
        • Searchable PDF Storage
        • Versioning
        • Results
        • Marking Useful Results
        • Folder Browsing
        • Cascading Search
        • Search Suggestions
        • Delve Settings
        • Collections
        • Miscellaneous
      • Application Access
      • Search Relevancy
        • Core Settings
        • Makers Algorithm
        • Filename Boost Layer
        • Minimum Matching Terms Filter
        • Field Boost
        • Modified Date Boosting
        • Hit Highlighting
        • Why My Search Matched
        • Data Search Strategy
      • Search Performance
        • Accurate Totals for Pagination
        • Source cards
        • Facets
      • Filtering
      • Presets
        • File Groups
        • Security Groups
      • Code of Conduct
      • Metrics
      • Viewer
      • SAR
        • Importing Data For A SAR
        • Getting SAR data from Google Vault
        • Redacting Information
        • SAR Access
      • Privacy Portal
        • Activate the Privacy Portal
        • Disclosure
        • Submit SARs From The Privacy Portal
        • Email Delivery Settings
          • Delivery Settings
          • Brand Settings
          • Customise Emails
        • SMS Delivery Settings
        • Requestor Message Limit
        • Attachment Configuration
        • Password Configuration
        • File Scanner Configurator
      • Visualisations
        • Related Result Connections Diagram
        • Event Timeline
        • Create and Modified Date Activity Chart
        • Relationship Map
      • Notifications
      • Map View
      • App
      • Theming
        • Details
        • Logo
        • Background Images
        • Colours
      • Related Results
      • OData API
      • Bulk Search
        • Managing a Bulk Search
    • User Settings
    • Stats
  • API Guides
    • Insight API Guide
      • Swagger Documentation
      • Trying Some Endpoints
      • Search Filter
      • Hits / Items
      • Inspecting REST Calls
    • Data Science API Guide
      • REST Interface
        • Login
        • Datasets
        • Fields
        • Field Statistics
        • Search
        • Scroll
        • Update
      • Python Wrapper
        • Login
        • Datasets
        • Fields
        • Field Statistics
        • Search
        • Query Builders
        • Scroll
        • Scroll Search
        • Update Single Document
        • Bulk Update
    • Creating a Native Enrichment Step
      • Creating an Enrichment Step
        • Creating the Core Classes
        • Extending our Enrichment Step
        • Adding a Configuration Template
        • Adding the Enrichment Step
        • Creating an Enrichment Pipeline
      • Other Tasks
        • Entities, Metadata and Data
        • Accessing the Repository
      • Example Code
      • Troubleshooting
    • Creating a Python Enrichment Step
      • Creating an Enrichment Step
        • Running the Example from Command Line
        • Running the Example
      • Creating Your Own Step
      • Adding or Changing Entities, Metadata
  • whitepapers and explainers
    • Methods for Text Summarization
      • Application
      • Technology Methods
      • Commercial Tools
      • Key Research Centres
      • Productionisation
      • Related Areas of Text Analytics
      • Conclusion
      • References
Powered by GitBook
On this page
  1. API Guides
  2. Creating a Python Enrichment Step

Creating an Enrichment Step

The installation and configuration guide for the Python REST Service provides an excellent insight into the folder structure of the service and how it works. This guide will therefore focus on the things specific to creating a Python enrichment step.

Example Enrichment Step

An example enrichment step ships with the Python REST Service, which provides a good starting point for a new enrichment step. This can be found alongside the other enrichment steps in the endpoints subfolder.

Walkthrough

First we import the libraries that we require for the base functionality of a REST step, which include flask, logging, json and some os calls.

import flask
from flask import request, jsonify
import logging
import logging.config, yaml,logging
import json
import os

Next, we work out where we are in the folder tree. We do this so we can run the file both directly for development and debugging purposes, and from within the actual service. More will become clear about this shortly.

module_location = os.path.dirname(os.path.abspath(__file__))

The example makes use of a config file for some settings. The convention is to store these in the config sub-folder, although you do not need to do this.

config_file = open(module_location + '/../config/endpoints/example.json')
example_config = json.load(config_file)["example"]
config_file.close()

If we are running the file directly we configure logging and pull in our example library. If we are running within the REST Service then logging will be configured for us already.

if __name__ == "__main__":
    logging.config.dictConfig(yaml.safe_load(open(module_location + '/../config/logging.conf')))
    from example_libs.example_lib import *
else:
    from .example_libs.example_lib import *

You can define an initialise function which will be called on start up once. You can also simply include this code inline if you wish.

def initialise():
    logging.info('Example initialise code goes here')

The end point file must contain a function with the same name. Here we just chain onto a do_<endpoint> method which does the work. We do this to make testing easy (this will become clear shortly).

The request will contain the JSON payload that is sent to the step by Aiimi Insight Engine. We call this the ‘work item’. Essentially, the step manipulates this structure and returns it.

# this is the function that will get called
def example():   
    return jsonify(do_example(request.get_json()))

This is the actual function that does the work. We print out the ‘work item’, call an example library function to get some text, and then append that to the text content of the work item.

# this is the function that will get called
def do_example(work):  
    print("BEFORE:")
    print(work)
    print()

    # add your code here to manipulate the EnrichmentWorkItem structure
    logging.info("Generating example for: %s", work['File']['Name'])
    work['File']['TextContent'] += ' - ' + example_lib_function(example_config["text"])
    print("AFTER:")
    print(work)
    return work

We include a main in the Python code so we can call the step directly from the command line. This makes development easy. You need to create a work item structure (there is a simple one here). You can print out what Aiimi Insight Engine sends you to and then code up something from this. For this type of testing you only need to include the fields that you are using an not the whole work item.

if __name__ == "__main__":
    text_content = "This is some test text"
    work = {}
    work["File"] = {}
    work["File"]["Name"] = "Test Document.txt"
    work["File"]["Extension"] = "txt"
    work["File"]["TextContent"] = text_content
    do_example(work)
PreviousCreating a Python Enrichment StepNextRunning the Example from Command Line