Aiimi Insight Engine Anaheim
User GuidesAiimi
  • Introducing Aiimi Insight Engine
  • Architecture
    • Overview and Key Concepts
    • Hosting Options
    • Architecture and How It Works
      • Agent Servers
        • Security Agent
        • Source Agent
        • Content Agent
        • Enrichment Agent
        • Job Agent
        • OCR Agent
        • Migration Agent
        • Tika Agent
      • Repository
        • Data Node
        • Proxy Node
        • Kibana Node
      • Gateway and User Interface
      • Document and Data Sources
    • Deployment Options
    • Security
      • Source System Security
      • Firewalling
      • Agent Servers
        • Security Agent
        • Source Agent
        • Content Agent
        • Enrichment Agent
        • Job Agent
        • OCR Agent
        • Migration Agent
        • Tika Agent
      • Repository
      • Gateway (Web Server)
      • Tools & Utilities
  • Installation
    • Elasticsearch Installation (Windows)
    • Aiimi Insight Engine Installation (Windows)
    • REST Enrichment Service Install and Configuration
      • Installation and Setup
        • Offline Set-up
        • Configuration of Logging
        • Running as a Service (Windows)
        • Using SSL
        • Performance and Concurrency
        • Security
      • Communicating with the Rest Enrichment Service
      • Step Configuration
    • HTML Cleaner Service
  • security
    • Users
    • Data and Documents
      • Progressive Access
      • Privileged Access
  • Control Hub
    • Configurations
      • Managing Configurations
        • Edit and Delete
        • Find a Configuration
        • Monitoring Stats
        • Stop a Configuration
        • Refreshing
      • Security Configurations
        • Configuring Security
          • Active Directory
          • Azure Active Directory
          • BuiltinSecurity
          • MiroSecurity
          • GoogleDirectory
        • Security Sync
        • Security Agents
        • Security Scheduling
      • Source Configurations
        • General
        • Source
          • Azure Blob Storage
          • BBC Monitoring
          • Big Query Cataloguer
          • BIM360
          • CSV Data Loader
          • Confluence
          • Content Server
          • Data File Cataloguer
          • Document Store
          • DocuSign
          • Dropbox
          • Exchange 365
          • Filesystem
          • Google Bucket
          • Google Drive
          • JSON Data Loader
          • Miro
          • ODBC Data Loader
          • PowerBi Cataloguer
          • Reuters Connect
          • ShareFile
          • SharePoint
          • SharePoint Legacy
          • SQL Server Cataloguer
          • Websites
          • XML Data Loader
        • Crawl
        • Source Agents
        • Source Scheduling
        • Advanced
      • Enrichment Configurations
        • Creating a Pipeline
          • General
          • Steps
            • AccessMiner
            • Anonymiser
            • CAD Extractor
            • Checksum
            • Content Retrieval
            • Copy
            • Data Rule Processor
            • Delete
            • Email Extractor
            • Entity Rule Processor
            • External Links
            • Geotag
            • Google NLP Extractor
            • Google Vision Extractor
            • Metrics Calculation
            • Microsoft Vision Extractor
            • OcrRest
            • Office Metadata
            • PCI Extractor
            • REST
            • Set Document Risk
            • Text Cleaner
            • Tika Text Extraction
            • Trie Entity Extractor
          • Filters
          • Agents
          • Schedule
          • Advanced
      • OCR Engine
      • Job Configurations
        • General
        • Job
          • AutomatedSearchJob
          • Command Job
          • ElasticJob
          • Extended Metrics Job
          • GoogleVaultSAR
          • Nightly Events Processor Job
          • Notifications Processor Job
          • Portal Sync Job
          • Purge Job
          • Text Content Merge Job
        • Output
        • Agents
        • Scheduling
      • Migration Configuration
        • General
        • Filter
        • Metadata Mappings
        • Agents
        • Scheduling
        • Advanced
      • Content Server
    • Credentials
      • Create a Credential
      • Find a Credential
      • Edit a Credential
      • Delete a Credential
    • Mappings
      • Entities
        • Managing Groups
        • Create an Entity
        • Managing Entities
      • Models
        • Create a New Model
        • Find a Model
        • Enable or Disable a Model
      • Vectors
      • Rank Features
    • Featured Links
    • Global Settings
      • General
        • Stackdriver
        • Document Recommendations
        • Searchable PDF Storage
        • Versioning
        • Results
        • Marking Useful Results
        • Folder Browsing
        • Cascading Search
        • Search Suggestions
        • Delve Settings
        • Collections
        • Miscellaneous
      • Authentication
      • Application Access
      • Search Relevancy
        • Core Settings
        • Makers Algorithm
        • Filename Boost Layer
        • Minimum Matching Terms Filter
        • Field Boost
        • Modified Date Boosting
        • Hit Highlighting
        • Why My Search Matched
        • Data Search Strategy
      • Search Performance
      • Filtering
      • Thumbnails
      • Presets
      • Code of Conduct
      • Metrics
      • Viewer
      • SAR
        • Importing Data For A SAR
        • Getting SAR data from Google Vault
        • Redacting Information
        • SAR Access
      • Privacy Portal
        • Activate the Privacy Portal
        • Disclosure
        • Submit SARs From The Privacy Portal
        • Email Delivery Settings
          • Delivery Settings
          • Brand Settings
          • Customise Emails
        • SMS Delivery Settings
        • Requestor Message Limit
        • Attachment Configuration
        • Password Configuration
        • File Scanner Configurator
      • Visualisations
        • Related Result Connections Diagram
        • Event Timeline
        • Create and Modified Date Activity Chart
        • Relationship Map
      • Notifications
      • Map Lens
      • App
      • Theming
      • Related Results
      • OData API
      • Bulk Search
        • Managing a Bulk Search
      • Search Flows
    • User Settings
    • Stats
      • Data Views
  • API Guides
    • Insight API Guide
      • Swagger Documentation
      • Trying Some Endpoints
      • Search Filter
      • Hits / Items
      • Inspecting REST Calls
    • Data Science API Guide
      • REST Interface
        • Login
        • Datasets
        • Fields
        • Field Statistics
        • Search
        • Scroll
        • Update
      • Python Wrapper
        • Login
        • Datasets
        • Fields
        • Field Statistics
        • Search
        • Query Builders
        • Scroll
        • Scroll Search
        • Update Single Document
        • Bulk Update
    • Creating a Native Enrichment Step
      • Creating an Enrichment Step
        • Creating the Core Classes
        • Extending our Enrichment Step
        • Adding a Configuration Template
        • Adding the Enrichment Step
        • Creating an Enrichment Pipeline
      • Other Tasks
        • Entities, Metadata and Data
        • Accessing the Repository
      • Example Code
      • Troubleshooting
    • Creating a Python Enrichment Step
      • Creating an Enrichment Step
        • Running the Example from Command Line
        • Running the Example
      • Creating Your Own Step
      • Adding or Changing Entities, Metadata
  • whitepapers and explainers
    • From a Billion To One – Mastering Relevancy
    • Methods for Text Summarization
      • Application
      • Technology Methods
      • Commercial Tools
      • Key Research Centres
      • Productionisation
      • Related Areas of Text Analytics
      • Conclusion
      • References
Powered by GitBook
On this page
  • Configure the Enrichment Steps
  • From The Command Line
  • No Python Version Exists - Install Python
  • Python versions exist - Install Python
  • Configure and Install Prerequisites
  1. Installation
  2. REST Enrichment Service Install and Configuration

Installation and Setup

PreviousREST Enrichment Service Install and ConfigurationNextOffline Set-up

Last updated 3 months ago

This guide assumes you are familiar with Python and JSON config files.

This section shows you how to run the service from the command line, useful for testing. It also shows you how to run it as a service, best suited to a production environment.

The REST Enrichment Service along with the enrichment steps can be found in the InsightMaker.Python distribution.

Configure the Enrichment Steps

  • You need to activate the enrichment steps by editing the configuration.

  • The REST Service configuration is based on custom overrides in the folder 'config/custom'.

  • You can copy configuration files from 'config' and 'config/providers' into the custom folder and override the configuration parameters required for your configuration.

From The Command Line

We recommend you get the service running from the command line first so you can easily spot and resolve issues.

The following steps are required to get an instance running on the command line.

  1. Check if you already have a version of Python running on the server. You can do this by checking the PATH and location of installed applications.

No Python Version Exists - Install Python

Install Python

  1. Install Python 3.12.8

    • To run Python REST services as a Window Service you need to install Python for ‘all users’.

Create a Python Virtual Environment

  1. Create a venv folder in the root of the RestEnrichmentService folder.

  2. Open a command prompt, navigate to the venv folder, and create the venv with this python command: python -m venv ./

Python versions exist - Install Python

Install Python

  1. Install Python 3.12.8

    • To run Python REST services as a Window Service you need to install Python for ‘all users’.

    • Do not add, select add to the system variables or path as this may interfere with existing Python applications running on the server.

  2. Open an administrator command prompt and install virtualenv: pip install virtualenv

Create a Python Virtual Environment

  1. Create a venv folder in the root of the RestEnrichmentService folder.

  2. Open a command prompt, navigate to the venv folder, and create the venv with this python command: python -m virtualenv ./ -p="C:\Program Files\Python312\python.exe"

  3. Replace the -p parameter with the path to the Python 3.12.8 executable.

Configure and Install Prerequisites

Ensure long path support is enabled

  1. Using Regedit:

    • HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem

    • LongPathsEnabled

    • DWORD = 1

Activate Python virtual machine

  1. Activate your virtual environment by navigating to the venv/Scripts folder and running: activate.bat

    • You should now see (venv) at the start of your command line.

  2. On the command line navigate back to InsightMaker.Python\RestEnrichmentService

Python Torch run-time prerequisites.

  1. Use ‘pip’ to install all the requirements in the latest requirements file:

    1. Online: pip install -r requirements_3.12.8.txt

    2. Offline: From an internet connected machine:

      1. Create a temporary folder.

      2. Navigate to the temporary folder.

      3. pip download -r requirements_3.12.8.txt

      4. Copy the resulting contents of the temporary folder to the target server.

      5. Run the following on the target server, replacing the path to the temporary folder as appropriate.

        1. pip install --no-index --find-links C:/tmp/libs -r requirements_3.12.8.txt

If you do not have an internet connection, run the following commands on a machine that does, and then copy the NLTK data to the target server. Remember to also recreate the environment variable on the target server.

  1. Install the NLTK data (models) to a location on the server:

    • python -m nltk.downloader all -d <DRIVE:/LOCATION>

  2. Set an environment variable to point at the NLTK data:

  3. Edit run.bat in the root of ‘InsightMaker.Python\RestEnrichmentService’ to point at the correct location for your virtual environment.

  4. Exit the virtual environment and open a new command window.

  5. Navigate to the ‘InsightMaker.Python\RestEnrichmentService’ folder.

If you are running on a server without an internet connection then you will need to disable all steps that use huggingface (hugginface NER, sentence transformers, spare vectors etc).

  1. Execute ‘run.bat’ on the command line. You should see the output as shown below.

    • This should activate your virtual machine and start the service.

  2. Make sure you are in the root folder of the RestEnrichmentService.

Once this runs correctly and you have tested it with a test enrichment pipeline then you can run the service as a Window service if you want to.

For Windows - Visual Studio runtime must be installed.

Other Operating Systems - Check requirements

NLTK_DATA=DRIVE:/LOCATION The NLTK guide:

This can be set up once the installation has been tested.

https://visualstudio.microsoft.com/visual-cpp-build-tools/
https://pypi.org/project/torch/
https://www.nltk.org/data.html
This is covered in the ‘Offline Set-up’ guide.
REST Enrichment Service
Custom Configuration
RestEnrichmentService