Aiimi Insight Engine Disco
User GuidesAiimi
  • Introducing Aiimi Insight Engine
  • Architecture
    • Overview and Key Concepts
    • Hosting Options
    • Architecture and How It Works
      • Agent Servers
        • Security Agent
        • Source Agent
        • Content Agent
        • Enrichment Agent
        • Job Agent
        • OCR Agent
        • Migration Agent
        • Tika Agent
      • Repository
        • Data Node
        • Proxy Node
        • Kibana Node
      • Gateway and User Interface
      • Document and Data Sources
    • Deployment Options
      • Development Environments
      • Testing and QA Environments
      • Production Environments
      • Disaster Recovery Environments
    • Security
      • Source Systems
      • Firewalling
      • Agent Servers
        • Security Agent
        • Source Agent
        • Content Agent
        • Enrichment Agent
        • Job Agent
        • OCR Agent
        • Migration Agent
        • Tika Agent
      • Repository
      • Gateway (Web Server)
      • Tools & Utilities
  • Installation
    • Elasticsearch Installation (Windows)
    • Aiimi Insight Engine Installation (Windows)
    • Python REST Service Install and Configuration
      • Installation and Setup
        • Offline Huggingface Set-up
        • Configuration of Logging
        • Running as a Service (Windows)
        • Using SSL
        • Performance and Concurrency
        • Security
      • Communicating with the Python REST Service
      • Step Configuration
        • Classify
        • Entity Mapper
        • Huggingface NER
        • Language Detection
        • Phrases and Topics
        • Summary
        • Spacyner
    • HTML Cleaner Service
  • security
    • Users
      • User Authentication
      • Code of Conduct
      • App Access and Role-Based Access
    • Data & Documents
      • Document and Data Access Control Lists
      • Progressive Access and Privileged Access
        • Progressive Access
        • Privileged Access
      • Source Scope and Visibility within Apps
      • Classification
      • Advanced Security Controls - Security Classifications
      • Risk Ratings
      • Redaction and Anonymisation
      • Banned Words
      • Mark as Sensitive
      • Audit Controls
  • Control Hub
    • Configurations
      • Managing Configurations
        • Edit and Delete
        • Find a Configuration
        • Monitoring Stats
        • Stop a Configuration
        • Refreshing
      • Security Configurations
        • Configuring Security
        • Active Directory
        • Azure Active Directory
        • BuiltinSecurity
        • Security Sync
        • Security Agents
        • Security Scheduling
      • Source Configurations
        • General
        • Source
          • Azure Blob Storage
          • BBC Monitoring
          • Big Query Cataloguer
          • BIM360
          • CSV Data Loader
          • Confluence
          • Content Server
          • Data File Cataloguer
          • Document Store
          • Dropbox
          • Exchange 365
          • Filesystem
          • Google Bucket
          • JSON Data Loader
          • ODBC Data Loader
          • PowerBi Cataloguer
          • Reuters Connect
          • ShareFile
          • SharePoint
          • SharePoint Legacy
          • SQL Server Cataloguer
          • Websites
          • XML Data Loader
        • Crawl
        • Source Agents
        • Source Scheduling
        • Advanced
      • Enrichment Configurations
        • Creating a Pipeline
          • General
          • Steps
            • Tika Text Extraction
            • External Links
            • Delete
            • Copy
            • Text Cleaner
            • Data Rule Processor
            • Checksum
            • OcrRest
            • AccessMiner
            • CAD Extractor
            • Trie Entity Extractor
            • PCI Extractor
            • Email Extractor
            • Geotag
            • Google Vision Extractor
            • Google NLP Extractor
            • Metrics Calculation
            • Microsoft Vision Extractor
            • Entity Rule Processor
            • Anonymiser
            • Set Document Risk
            • Content Retrieval
            • REST
          • Filters
          • Agents
          • Schedule
          • Advanced
      • OCR Engine
      • Job Configurations
        • General
        • Job
          • AutomatedSearchJob
          • Command Job
          • ElasticJob
          • Extended Metrics Job
          • GoogleVaultSAR
          • Nightly Events Processor Job
          • Notifications Processor Job
          • Portal Sync Job
          • Purge Job
          • Text Content Merge Job
        • Output
        • Agents
        • Scheduling
      • Migration Configuration
        • General
        • Filter
        • Metadata Mappings
        • Agents
        • Scheduling
        • Advanced
      • Content Server
    • Credentials
      • Create a Credential
      • Find a Credential
      • Edit a Credential
      • Delete a Credential
    • Entities
      • Create a Group
      • Create an Entity
      • Managing Entities
      • Import or Export Entities
    • Models
      • Create a New Model
      • Find a Model
      • Enable or Disable a Model
    • Featured Links
    • Global Settings
      • General
        • Thumbnails
        • Windows Authentication
        • SAML2 Authentication
        • ADFS
        • Stackdriver
        • Document Recommendations
        • Searchable PDF Storage
        • Versioning
        • Results
        • Marking Useful Results
        • Folder Browsing
        • Cascading Search
        • Search Suggestions
        • Delve Settings
        • Collections
        • Miscellaneous
      • Application Access
      • Search Relevancy
        • Core Settings
        • Makers Algorithm
        • Filename Boost Layer
        • Minimum Matching Terms Filter
        • Field Boost
        • Modified Date Boosting
        • Hit Highlighting
        • Why My Search Matched
        • Data Search Strategy
      • Search Performance
        • Accurate Totals for Pagination
        • Source cards
        • Facets
      • Filtering
      • Presets
        • File Groups
        • Security Groups
      • Code of Conduct
      • Metrics
      • Viewer
      • SAR
        • Importing Data For A SAR
        • Getting SAR data from Google Vault
        • Redacting Information
        • SAR Access
      • Privacy Portal
        • Activate the Privacy Portal
        • Disclosure
        • Submit SARs From The Privacy Portal
        • Email Delivery Settings
          • Delivery Settings
          • Brand Settings
          • Customise Emails
        • SMS Delivery Settings
        • Requestor Message Limit
        • Attachment Configuration
        • Password Configuration
        • File Scanner Configurator
      • Visualisations
        • Related Result Connections Diagram
        • Event Timeline
        • Create and Modified Date Activity Chart
        • Relationship Map
      • Notifications
      • Map View
      • App
      • Theming
        • Details
        • Logo
        • Background Images
        • Colours
      • Related Results
      • OData API
      • Bulk Search
        • Managing a Bulk Search
    • User Settings
    • Stats
  • API Guides
    • Insight API Guide
      • Swagger Documentation
      • Trying Some Endpoints
      • Search Filter
      • Hits / Items
      • Inspecting REST Calls
    • Data Science API Guide
      • REST Interface
        • Login
        • Datasets
        • Fields
        • Field Statistics
        • Search
        • Scroll
        • Update
      • Python Wrapper
        • Login
        • Datasets
        • Fields
        • Field Statistics
        • Search
        • Query Builders
        • Scroll
        • Scroll Search
        • Update Single Document
        • Bulk Update
    • Creating a Native Enrichment Step
      • Creating an Enrichment Step
        • Creating the Core Classes
        • Extending our Enrichment Step
        • Adding a Configuration Template
        • Adding the Enrichment Step
        • Creating an Enrichment Pipeline
      • Other Tasks
        • Entities, Metadata and Data
        • Accessing the Repository
      • Example Code
      • Troubleshooting
    • Creating a Python Enrichment Step
      • Creating an Enrichment Step
        • Running the Example from Command Line
        • Running the Example
      • Creating Your Own Step
      • Adding or Changing Entities, Metadata
  • whitepapers and explainers
    • Methods for Text Summarization
      • Application
      • Technology Methods
      • Commercial Tools
      • Key Research Centres
      • Productionisation
      • Related Areas of Text Analytics
      • Conclusion
      • References
Powered by GitBook
On this page
  1. Installation
  2. Python REST Service Install and Configuration

Installation and Setup

PreviousPython REST Service Install and ConfigurationNextOffline Huggingface Set-up

This section shows you how to run the service from the command line, which is useful for testing, and how to run as a service which is more suited to a production environment.

The Python Rest Service along with the Python enrichment steps can be found in the InsightMaker.Python distribution.

Note that this guide assumes you are familiar with Python.

First configure the enrichment steps that you want to activate by editing the config/endpoints.json file. Simply add and/or remove the ones that you want to use.

The following steps are required to get an instance running on the command line. It is recommended that you first get the service running on the command line so you can easily resolve any issues. Once this runs correctly and you have tested it with a test enrichment pipeline then you can optionally run the service as a Window service.

Before starting, check to see if there is already a version of Python running on the server. You can do this by checking the PATH and the location of installed applications.

Option 1 – Install Python where no other Python version exists on the server:

  1. Install Python 3.9.13

    1. You need to install Python for ‘all users’ if you plan to run the Python REST Service as a Window Service.

  2. Create a Python ‘virtual environment’:

    1. Create a venv folder in the root of the PythonRestService folder.

    2. Open a command prompt, navigate to the venv folder, and create the venv with the following python command:

      1. python -m venv ./

Option 2 – Install Python where other Python versions exist on the server:

  1. Install Python 3.9.13

    1. You need to install Python for ‘all users’ if you plan to run the Python REST Service as a Window Service.

    2. Do not add select add to the system variables or path (this may interfere with existing Python applications running on the server)

  2. Open an administrator command prompt and install virtualenv

    1. pip install virtualenv

  3. Create a Python ‘virtual environment’:

    1. Create a venv folder in the root of the PythonRestService folder.

    2. Open a command prompt, navigate to the venv folder, and create the venv with the following python command:

      1. python -m virtualenv ./ -p="C:\Program Files\Python39\python.exe"

      2. Replace the -p parameter with the path to the Python 3.9.13 executable

Configure and install prerequisites:

  1. Ensure long path support is enabled – use regedit to do this:

    1. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem

    2. LongPathsEnabled

    3. DWORD = 1

  2. Activate your Python virtual environment:

    1. Activate your virtual environment by navigating to the venv/Scripts folder and running activate.bat

    2. You should now see (venv) at the start of your command line.

  3. On the command line navigate back to InsightMaker.Python\PythonRestService

  4. Open the the requirements_<x.x.x>.txt file and check that the following line is not in the file, and if it is remove it:

    1. en_core_web_sm==xxx (where xxx is the version number)

  5. Python Torch has some run-time prerequisites.

    1. For Windows, the prerequisites for torch include the Visual Studio runtime – install this.

    2. For other OS please see: https://pypi.org/project/torch/

  6. Use ‘pip’ to install all the requirements in the latest requirements file:

    1. Online:

      1. pip install -r requirements_3.9.13.txt

    2. Offline:

      1. On an internet conenected machine:

        1. Create a temporary folder.

        2. Navigate to the temporary folder.

        3. pip download -r requirements_3.9.13.txt

        4. Copy the resulting contents of the temporary folder to the target server.

      2. Run the following on the target server, replacing the path to the temporary folder as appropriate.

        1. pip install --no-index --find-links C:/tmp/libs -r requirements_3.9.13.txt

  7. Run the following to install the Spacy models (if you are using this enrichment step):

    1. python -m spacy download en_core_web_sm

  8. Install the NLTK data (models) to a location on the server:

    1. Note, if you do not have an internet connection, run the following commands on a machine that does, and then copy the NLTK data to the target server. Remember to also recreate the environment variable on the target server.

    2. python -m nltk.downloader all -d <DRIVE:/LOCATION>

    3. Set an environment variable to point at the NLTK data:

      1. NLTK_DATA=DRIVE:/LOCATION

    4. Occasionally ‘omw-1.4’ is missing – run this to double check. It will install it if it is missing.

      1. python -m nltk.downloader omw-1.4

  9. Edit run.bat in the root of ‘InsightMaker.Python\PythonRestService’ to point at the correct location for your virtual environment and Python Rest Service

  10. Exit the virtual environment and open a new command window and navigate to the ‘InsightMaker.Python\PythonRestService’ folder.

  11. If you are running on a server without an internet connection, open the config/endpoints.json file and remove huggingfacener.

    1. We can get this working once we have tested the installation thus far. This is covered in the ‘Offline Huggingface Set-up’ section.

  12. Edit the ‘run.bat’ so that it contains an absolute path to the ‘waitress_server.py’ file and your virtual environment

  13. Execute ‘run.bat’ on the command line. You should see the output as shown below.

    1. This should activate your virtual machine and start the service

    2. Make sure you are in the root folder of the Python Rest Service

If you navigate to http://localhost :15005/ you should see this in the browser.

See this Python guide for creating a venv:

The guide is here for NLTK:

https://docs.python.org/3/tutorial/venv.html
https://visualstudio.microsoft.com/visual-cpp-build-tools/
https://www.nltk.org/data.html