Aiimi Insight Engine
User GuidesAiimi
  • Introducing Aiimi Insight Engine
  • Architecture
    • Overview and Key Concepts
    • Search Flows
      • Search Flow Types
      • Smart Filtering
      • Query and Prompt Classification
      • Search Algorithms
      • Extractive and Generative Models
    • Hosting Options
    • Architecture and How It Works
      • Agent Servers
        • Security Agent
        • Source Agent
        • Content Agent
        • Enrichment Agent
        • Job Agent
        • OCR Agent
        • Migration Agent
        • Tika Agent
      • Repository
        • Data Node
        • Proxy Node
        • Kibana Node
      • Gateway and User Interface
      • Document and Data Sources
    • Deployment Options
    • Security
      • User Security
      • Data and Document Security
      • Source System Security
      • Firewalling
      • Agent Servers
      • Repository
      • Gateway (Web Server)
      • Tools & Utilities
  • Installation
    • Elastic and Kibana Install (Windows)
    • Aiimi Insight Engine Installation (Windows)
      • Installation Security
      • Certificates in a Key Vault
      • SAR Configuration
      • CSOM Bridge Set Up
      • AI Studio
    • AI Services
      • Prerequisites
      • AI Enrichment Service
        • Installation and Setup
        • Enabling Enrichment Steps
        • Using AI Enrichment Steps
        • Performance and Concurrency
      • AI Model Service
        • Installation and Setup
        • Enabling Providers
        • Private Generative AI
        • Azure Open AI
        • Enabling AI History
        • HTML Cleaner Service
      • Configuration of Logging
      • Offline Set-up of Models
      • Using SSL
      • Running as a Service (Windows)
      • Using GPUs
      • AI and Semantic Search Set Up
        • Open & Closed Book AI
        • Semantic Search
          • Vectors for Semantic Search
          • Source Configuration
          • Sentence Transformer Models
          • Enrichment
          • Kibana
          • Final Search Flow
    • Email Threading Upgrade
  • Run Books
    • SharePoint Online Connector
  • Control Hub
    • Configurations
      • Config Management
      • Security Configurations
        • Security - General
        • Security - Source
          • Active Directory
          • Atlassian
          • Azure Active Directory
          • Builtin Security
          • Miro Security
          • Google Directory
          • Slack Security
        • Security - Sync
        • Security - Agents
        • Security - Scheduling
      • Source Configurations
        • Source - General
        • Source - Source
          • Alfresco Kafka
          • Azure Blob Storage
          • BBC Monitoring
          • Big Query Cataloguer
          • BIM360
          • CSV Data Loader
          • Confluence
          • Content Server
          • Data File Cataloguer
          • Document Store
          • DocuSign
          • Dropbox
          • Exchange 365
          • Filesystem
          • Google Bucket
          • Google Drive
          • Google Vault
          • Jira
          • JSON Data Loader
          • Livelink
          • MFiles
          • Microsoft Teams
          • Mimecast
          • Miro
          • ODBC Data Loader
          • PowerBi Cataloguer
          • Reuters Connect
          • ShareFile
          • SharePoint
            • Azure Portal and Azure AD Authentication
            • Sensitivity Labels
          • SharePoint Legacy
          • SQL Server Cataloguer
          • Slack
          • Versioned Document Store
          • Websites
          • XML Data Loader
        • Source - Crawl
        • Source - Agents
        • Source - Schedule
        • Source - Advanced
      • Enrichment Configurations
        • Creating a Pipeline
          • General
          • Steps
            • AccessMiner
            • AI Classification
            • Anonymiser
            • CAD Extractor
            • Checksum
            • Content Retrieval
            • Copy
            • Data Rule Processor
            • Delete
            • Email Extractor
            • Entity Rule Processor
            • External Links
            • Geotag
            • Google NLP Extractor
            • Google Vision Extractor
            • Metrics Calculation
            • Microsoft Vision Extractor
            • OcrRest
            • Office Metadata
            • PCI Extractor
            • REST
            • Set Document Risk
            • Text Cleaner
            • Tika Text Extraction
            • Trie Entity Extractor
            • Update Metadata
          • Filters
          • Agents
          • Schedule
          • Advanced
      • OCR Engine
      • Job Configurations
        • General
        • Job
          • AutomatedSearchJob
          • Command Job
          • ElasticJob
          • Extended Metrics Job
          • File Extractor
          • GoogleVaultSAR
          • Google Drive Last Access Date
          • Nightly Events Processor Job
          • Notifications Processor Job
          • Portal Sync Job
          • Purge Job
          • Text Content Merge Job
        • Output
        • Agents
        • Scheduling
      • Migration Configuration
        • General
        • Filter
        • Metadata Mappings
        • Agents
        • Scheduling
        • Advanced
    • Credentials
    • Mappings
      • Entities
        • Manage Entity Groups
        • Create an Entity
        • Manage an Entity
      • Models
        • Create a New Model
        • Find a Model
        • Enable or Disable a Model
      • Vectors
      • Rank Features
    • Featured Links
    • AI Settings
      • Classifications
      • Class
      • Class Rules
      • AI Classification
    • User Settings
    • Stats
      • Data Views
    • Global Settings
      • General
      • Authentication
      • App Settings
      • Application Access
      • Thumbnails
      • Presets
      • Code of Conduct
      • Metrics
      • Viewer
      • SAR
        • Importing Data For A SAR
        • SAR Disclosure Document Storage
        • Getting SAR data from Google Vault
        • SAR Access
        • SAR File Status
      • Disclosure Portal
        • Disclosure Portal Set Up
        • SARs From The Portal
        • Email Delivery Settings
          • Delivery Settings
          • Brand Settings
          • Customise Emails
        • SMS Delivery Settings
        • Requestor Message Limit
        • Attachment Configuration
        • Password Configuration
        • File Scanner Configurator
      • Collections
      • Visualisations
        • Related Result Connections Diagram
        • Event Timeline
        • Timeline Lens Activity Chart
        • Relationship Map
      • Notifications
      • Map Lens
      • Theming
      • User Avatar
      • OData API
      • Uploads
      • Security
    • Search Settings
      • Search Relevancy
        • Core Settings
        • Makers Algorithm
        • Filename Boost Layer
        • Minimum Matching Terms Filter
        • Field Boost
        • Modified Date Boosting
        • Hit Highlighting
        • Why My Search Matched
        • Data Search Strategy
      • Bulk Search
        • Managing a Bulk Search
      • Search Flows
        • Create a Search Flow
          • General
          • Query Classification Step
          • Search Steps
          • Model Steps
      • Filtering
      • Search Performance
      • Related Results
  • AI Studio
    • Classifications
      • Classifications
      • Classification Rules
    • Jobs
  • Labels
  • API Guides
    • Insight API Guide
      • Swagger Documentation
      • Trying Some Endpoints
      • Search Filter
      • Hits / Items
      • Inspecting REST Calls
    • Data Science API Guide
      • REST Interface
        • Login
        • Datasets
        • Fields
        • Field Statistics
        • Search
        • Scroll
        • Update
      • Python Wrapper
        • Login
        • Datasets
        • Fields
        • Field Statistics
        • Search
        • Query Builders
        • Scroll
        • Scroll Search
        • Update Single Document
        • Bulk Update
    • Creating a Native Enrichment Step
      • Creating an Enrichment Step
        • Creating the Core Classes
        • Extending our Enrichment Step
        • Adding a Configuration Template
        • Adding the Enrichment Step
        • Creating an Enrichment Pipeline
      • Other Tasks
        • Entities, Metadata and Data
        • Accessing the Repository
      • Example Code
      • Troubleshooting
    • Creating a Python Enrichment Step
      • Creating an Enrichment Step
        • Running the Example from Command Line
        • Running the Example
      • Creating Your Own Step
      • Adding or Changing Entities, Metadata
  • Whitepapers and Explainers
    • From a Billion To One – Mastering Relevancy
    • Methods for Text Summarization
      • Application
      • Technology Methods
      • Commercial Tools
      • Key Research Centres
      • Productionisation
      • Related Areas of Text Analytics
      • Conclusion
      • References
Powered by GitBook
On this page
  1. Architecture

Deployment Options

The section looks at the various deployment options that you have for Aiimi Insight Engine. These are guides and are not representative of what your deployment will look like.

Development Environments

For a development environment we recommend a single server set up. This should contain all the Aiimi Insight Engine components.

Single Server Requirement

A laptop with the following requirements is ample for a single server installation of Aiimi Insight Engine.

  • 4 CPU Cores

  • 16GB RAM

  • 250GB Disk Space

You should be able to ingest a few hundred thousand documents and continue development and testing activities.

Two Server Set Up

Developers may set the whole platform up on a laptop or workstation computer using the below split.

  • One server should host the repository.

  • One server should host the agents and the gateway.

Testing and QA Environments

The key decision about your QA environment is driven do you want to do performance testing. If you do, how much should it match the production environment in terms of scale.

End-user performance testing of ‘search’ related transactions

  • The elastic cluster and web server should be scaled as production.

  • The QA system should contain the same volume of content.

  • If you have not scaled the QA platform to match production, you can extrapolate some estimations for performance. Please note this is not an exact science, and the only true test is to have the QA platform sized the same as production.

Strict pre-production testing environments:

  • If production has a separate web server and agent servers so should QA.

  • If production has multiple agent servers, QA should have at least 2 agent servers.

  • If production has separate proxy nodes on the elastic cluster so should QA.

  • If production uses a multi node elastic cluster than QA should.

Performance Testing

For performance testing your Aiimi contact can provide separate guidelines.

  • The most significant process for end user response times is search and aggregation requests.

    • For performance, test the Elasticsearch cluster and the web servers being scaled.

  • If you want to test ingestion throughput ensure the Elasticsearch cluster and Agent servers are scaled.

Production Environments

Production environments need to be designed and scaled with your specific requirements.

Below are small, medium, and large sizing guides for each type of server. This can help as a starting point when designing your platform.

For a medium or large platform you will have multiple servers for the agent and repository tiers.

Scaling Agent Servers

  • Small – 4 CPU cores, 16GB RAM, 100GB local disk

  • Medium – 8 CPU cores, 32GB RAM, 100GB local disk

  • Large – 16 CPU cores, 64GB RAM, 100GB local disk

Scaling Repository Servers

  • Small – 8 CPU cores, 16GB RAM, 200GB SSD local disk

  • Medium – 16 CPU cores, 32GB RAM, 400GB SSD local disk

  • Large – 32 CPU cores, 64GB RAM, 800GB SSD local disk

    • Proxy servers do not need the same amount of memory and local disk since they store no data. For this 16GB RAM will be sufficient and the CPU loading will increase with transactional concurrency.

Scaling Web Servers

  • Small – 4 CPU cores, 16GB RAM, 100GB local disk

  • Medium – 8 CPU cores, 32GB RAM, 100GB local disk

  • Large – 16 CPU cores, 32GB RAM, 100GB local disk

Production Platform Considerations

  • Separate agent and web servers should be used.

  • The number of agent servers required depends on:

    1. The number of source repositories

    2. Their respective size

    3. Pipeline complexity

    4. Content Ingestion rate

  • In most scenarios only a single web server will be required.

    • If you use two you will need to load balance the user requests using a standard load balancer.

      • Load balancing should be ‘sticky’ – i.e. send the same users requests to the same web server

  • If OCR is heavily used, then this should have it's own agent server.

  • Separate proxy nodes in the repository tier.

  • Multiple elastic servers and limit the maximum amount of memory per-node to 64GB.

Disaster Recovery Environments

Disaster recovery environments need to consider the service needed during a ‘disaster scenario’.

The business criticality of the system can be scaled accordingly. If you need the same level of users and ingestion rates, the recovery platform should replicate the production environment.

PreviousDocument and Data SourcesNextSecurity

Last updated 1 month ago