Aiimi Insight Engine Chipotle
User GuidesAiimi
  • Introducing Aiimi Insight Engine
  • Architecture
    • Overview and Key Concepts
    • Search Flows
      • Search Flow Types
      • Smart Filtering
      • Query and Prompt Classification
      • Search Algorithms
      • Extractive and Generative Models
    • Hosting Options
    • Architecture and How It Works
      • Agent Servers
        • Security Agent
        • Source Agent
        • Content Agent
        • Enrichment Agent
        • Job Agent
        • OCR Agent
        • Migration Agent
        • Tika Agent
      • Repository
        • Data Node
        • Proxy Node
        • Kibana Node
      • Gateway and User Interface
      • Document and Data Sources
    • Deployment Options
    • Security
      • Source System Security
      • Firewalling
      • Agent Servers
        • Security Agent
        • Source Agent
        • Content Agent
        • Enrichment Agent
        • Job Agent
        • OCR Agent
        • Migration Agent
        • Tika Agent
      • Repository
      • Gateway (Web Server)
      • Tools & Utilities
  • Installation
    • Elasticsearch Installation (Windows)
    • Aiimi Insight Engine Installation (Windows)
    • AI Services
      • Prerequisites
      • AI Enrichment Service
        • Installation and Setup
        • Enabling Enrichment Steps
        • Using AI Enrichment Steps
        • Performance and Concurrency
      • AI Model Service
        • Installation and Setup
        • Enabling Providers
        • Private Generative AI
        • Azure Open AI
      • Configuration of Logging
      • Offline Set-up of Models
      • Using SSL
      • Running as a Service (Windows)
      • Using GPUs
    • HTML Cleaner Service
  • security
    • Users
    • Data and Documents
      • Progressive Access
      • Privileged Access
  • Control Hub
    • Configurations
      • Config Management
      • Security Configurations
        • Security - General
        • Security - Source
          • Active Directory
          • Azure Active Directory
          • Builtin Security
          • Miro Security
          • Google Directory
        • Security - Sync
        • Security - Agents
        • Security - Scheduling
      • Source Configurations
        • Source - General
        • Source - Source
          • Azure Blob Storage
          • BBC Monitoring
          • Big Query Cataloguer
          • BIM360
          • CSV Data Loader
          • Confluence
          • Content Server
          • Data File Cataloguer
          • Document Store
          • DocuSign
          • Dropbox
          • Exchange 365
          • Filesystem
          • Google Bucket
          • Google Drive
          • Google Vault
          • JSON Data Loader
          • Miro
          • ODBC Data Loader
          • PowerBi Cataloguer
          • Reuters Connect
          • ShareFile
          • SharePoint
          • SharePoint Legacy
          • Slack
          • SQL Server Cataloguer
          • Websites
          • XML Data Loader
        • Source - Crawl
        • Source - Agents
        • Source - Schedule
        • Source - Advanced
      • Enrichment Configurations
        • Creating a Pipeline
          • General
          • Steps
            • AccessMiner
            • Anonymiser
            • CAD Extractor
            • Checksum
            • Content Retrieval
            • Copy
            • Data Rule Processor
            • Delete
            • Email Extractor
            • Entity Rule Processor
            • External Links
            • Geotag
            • Google NLP Extractor
            • Google Vision Extractor
            • Metrics Calculation
            • Microsoft Vision Extractor
            • OcrRest
            • Office Metadata
            • PCI Extractor
            • REST
            • Set Document Risk
            • Text Cleaner
            • Tika Text Extraction
            • Trie Entity Extractor
          • Filters
          • Agents
          • Schedule
          • Advanced
      • OCR Engine
      • Job Configurations
        • General
        • Job
          • AutomatedSearchJob
          • Command Job
          • ElasticJob
          • Extended Metrics Job
          • GoogleVaultSAR
          • Google Drive Last Access Date
          • Nightly Events Processor Job
          • Notifications Processor Job
          • Portal Sync Job
          • Purge Job
          • Text Content Merge Job
        • Output
        • Agents
        • Scheduling
      • Migration Configuration
        • General
        • Filter
        • Metadata Mappings
        • Agents
        • Scheduling
        • Advanced
      • Content Server
    • Credentials
      • Create a Credential
      • Find a Credential
      • Edit a Credential
      • Delete a Credential
    • Mappings
      • Entities
        • Managing Groups
        • Create an Entity
        • Managing Entities
      • Models
        • Create a New Model
        • Find a Model
        • Enable or Disable a Model
      • Vectors
      • Rank Features
    • Featured Links
    • Global Settings
      • General
        • Stackdriver
        • Document Recommendations
        • Searchable PDF Storage
        • Versioning
        • Results
        • Marking Useful Results
        • Folder Browsing
        • Cascading Search
        • Search Suggestions
        • Delve Settings
        • Collections
        • Miscellaneous
      • Authentication
      • Application Access
      • Search Relevancy
        • Core Settings
        • Makers Algorithm
        • Filename Boost Layer
        • Minimum Matching Terms Filter
        • Field Boost
        • Modified Date Boosting
        • Hit Highlighting
        • Why My Search Matched
        • Data Search Strategy
      • Search Performance
      • Filtering
      • Thumbnails
      • Presets
      • Code of Conduct
      • Metrics
      • Viewer
        • Redacting Information
      • SAR
        • Importing Data For A SAR
        • Getting SAR data from Google Vault
        • SAR Access
      • Privacy Portal
        • Activate the Privacy Portal
        • Disclosure
        • Submit SARs From The Privacy Portal
        • Email Delivery Settings
          • Delivery Settings
          • Brand Settings
          • Customise Emails
        • SMS Delivery Settings
        • Requestor Message Limit
        • Attachment Configuration
        • Password Configuration
        • File Scanner Configurator
      • Visualisations
        • Related Result Connections Diagram
        • Event Timeline
        • Create and Modified Date Activity Chart
        • Relationship Map
      • Notifications
      • Map Lens
      • App
      • Theming
        • Links
        • Layout
        • Colours
      • Related Results
      • OData API
      • Bulk Search
        • Managing a Bulk Search
      • Search Flows
        • Create a Search Flow
          • General
          • Query Classification Step
          • Search Steps
          • Model Steps
    • User Settings
    • Stats
      • Data Views
  • API Guides
    • Insight API Guide
      • Swagger Documentation
      • Trying Some Endpoints
      • Search Filter
      • Hits / Items
      • Inspecting REST Calls
    • Data Science API Guide
      • REST Interface
        • Login
        • Datasets
        • Fields
        • Field Statistics
        • Search
        • Scroll
        • Update
      • Python Wrapper
        • Login
        • Datasets
        • Fields
        • Field Statistics
        • Search
        • Query Builders
        • Scroll
        • Scroll Search
        • Update Single Document
        • Bulk Update
    • Creating a Native Enrichment Step
      • Creating an Enrichment Step
        • Creating the Core Classes
        • Extending our Enrichment Step
        • Adding a Configuration Template
        • Adding the Enrichment Step
        • Creating an Enrichment Pipeline
      • Other Tasks
        • Entities, Metadata and Data
        • Accessing the Repository
      • Example Code
      • Troubleshooting
    • Creating a Python Enrichment Step
      • Creating an Enrichment Step
        • Running the Example from Command Line
        • Running the Example
      • Creating Your Own Step
      • Adding or Changing Entities, Metadata
  • whitepapers and explainers
    • From a Billion To One – Mastering Relevancy
    • Methods for Text Summarization
      • Application
      • Technology Methods
      • Commercial Tools
      • Key Research Centres
      • Productionisation
      • Related Areas of Text Analytics
      • Conclusion
      • References
Powered by GitBook
On this page
  • From The Command Line
  • Activate Python Virtual Machine
  • Python Torch run-time prerequisites.
  • Install Requirements
  • Register the AI Enrichment Service
  1. Installation
  2. AI Services
  3. AI Enrichment Service

Installation and Setup

PreviousAI Enrichment ServiceNextEnabling Enrichment Steps

Last updated 3 months ago

This guide assumes you are familiar with Python and JSON config files. It also assumes that you have worked through the prerequisite sections.

This section shows you how to run the service from the command line, which is useful for testing. It also shows you how to run it as a service, which is best suited to a production environment.

The AI Enrichment Service should be found in the InsightMaker.Python folder that you created in the prerequisites step.

From The Command Line

We recommend you get the service running from the command line first so you can easily spot and resolve issues.

Activate Python Virtual Machine

  1. Activate your virtual environment by navigating to the AIEnrichmentService/venv/Scripts folder and running: activate.bat

    • You should now see (venv) at the start of your command line.

  2. On the command line navigate back to InsightMaker.Python\AIEnrichmentService

Python Torch run-time prerequisites.

For Windows - Visual Studio runtime must be installed. https://visualstudio.microsoft.com/visual-cpp-build-tools/

Other Operating Systems - Check requirements https://pypi.org/project/torch/

Install Requirements

  1. Use ‘pip’ to install all the requirements in the latest requirements file:

    1. Online: pip install -r requirements_3.12.8.txt

    2. Offline: From an internet connected machine:

      1. Create a temporary folder.

      2. Navigate to the temporary folder.

      3. pip download -r requirements_3.12.8.txt

      4. Copy the resulting contents of the temporary folder to the target server.

      5. Run the following on the target server, replacing the path to the temporary folder as appropriate.

        1. pip install --no-index --find-links C:/tmp/libs -r requirements_3.12.8.txt

  2. If you have a GPU ad you plan to use that to accelerate steps such as vectorisation:

    1. Uninstall torch with

      1. pip uninstall torch

    2. Reinstall torch using the pip command generated by this link: https://pytorch.org/get-started/locally/

      1. For example:

        1. pip install torch --index-url https://download.pytorch.org/whl/cu118

  3. Edit run.bat in the root of ‘InsightMaker.Python\AIEnrichmentService’ to point at the correct location for your virtual environment.

Register the AI Enrichment Service

Edit the main configuration to the service can register itself:

  1. Open the following in an editor config/config.json

  2. Edit the following fields:

    1. Scheme should remain http, unless you plan to proxy requests through a HTTPS reverse proxy (IIS ARR for example) - in his case see the using SSL section.

    2. Address should be the IP address or host name of the server (do not use localhost)

    3. Port should not need changing

    4. Threads should not need changing at this stage

    5. System secret should contain the AIE system secret

    6. Admin API should point at the Admin API URL

    7. Trusted IPs should contain the IP Address of all of your enrichment agents

Now you can start the service by running run.bat.

Note that the first time the service starts no enrichment steps will actually load, as they are disabled by default. In the next section we enable the steps that we want to use.

AI Enrichment Service
config.json
AI Enrichment Service