Aiimi Insight Engine Installation (Windows)

This section presents a guide for setting up a simple single node Aiimi Insight Engine instance on Windows 2016 or later.

If you are designing and installing a production environment, then you will be installing individual agents and components on specific nodes. Further guidance on this can be provided on request.

Prerequisites

• Notepad++ (or your text editor of choice)

• Latest Java Development Kit and Java Runtime Environment - you can use something from https://adoptopenjdk.net if you wish. If you are installing Elastic on the same machine, use the one in the Elastic install folder.

• Search on Windows for 'Turn Windows Features On/Off' and enable Internet Information Services (IIS) in that dialog (used for managing websites and apis).

• .NET SDK (v8.0)

• .NET Core and Hosting bundle (.NET installs used to run InsightMaker agents and middleware as well as host web components) (v8.0)

Downloads

• Download NSSM to run the Tika Agent as a service

• Obtain your Aiimi Insight Engine distribution (Reach out to your contact at Aiimi)

• Obtain the Aiimi Aiimi Insight Engine Tika distribution

• Obtain your AIE licence from Aiimi

Download and Extract

The following guide will assume everything is being stored in the C: drive. You can use any drive but we recommend installing to the root of a drive.

1. Once you have your Aiimi Insight Engine distribution extract it to C:\InsightMaker.

1. Run the following scripts in an an administrator PowerShell window. These commands stop Windows silently blocking files.

   • gci c:\InsightMaker\Plugins\*.dll | Unblock-File

   • gci c:\InsightMaker\Scripts\*.ps1 | Unblock-File

   • Install the redistributables and .NET

2. Run the following Script in an admin PowerShell to update the Agent file names.

$Filename = "C:\InsightMaker\ContentAgent";
Copy-Item -Path "$filename\appsettings.default.json" -Destination "$filename\appsettings.json" -Force;
Copy-Item -Path "$filename\log4net.default.config" -Destination "$filename\log4net.config" -Force;
$Filename = "C:\InsightMaker\EnrichmentAgent";
Copy-Item -Path "$filename\appsettings.default.json" -Destination "$filename\appsettings.json" -Force;
Copy-Item -Path "$filename\log4net.default.config" -Destination "$filename\log4net.config" -Force;
$Filename = "C:\InsightMaker\JobAgent";
Copy-Item -Path "$filename\appsettings.default.json" -Destination "$filename\appsettings.json" -Force;
Copy-Item -Path "$filename\log4net.default.config" -Destination "$filename\log4net.config" -Force;
$Filename = "C:\InsightMaker\MigrationAgent";
Copy-Item -Path "$filename\appsettings.default.json" -Destination "$filename\appsettings.json" -Force;
Copy-Item -Path "$filename\log4net.default.config" -Destination "$filename\log4net.config" -Force;
$Filename = "C:\InsightMaker\OCRAgent";
Copy-Item -Path "$filename\appsettings.default.json" -Destination "$filename\appsettings.json" -Force;
Copy-Item -Path "$filename\log4net.default.config" -Destination "$filename\log4net.config" -Force;
$Filename = "C:\InsightMaker\SecurityAgent";
Copy-Item -Path "$filename\appsettings.default.json" -Destination "$filename\appsettings.json" -Force;
Copy-Item -Path "$filename\log4net.default.config" -Destination "$filename\log4net.config" -Force;
$Filename = "C:\InsightMaker\SourceAgent";
Copy-Item -Path "$filename\appsettings.default.json" -Destination "$filename\appsettings.json" -Force;
Copy-Item -Path "$filename\log4net.default.config" -Destination "$filename\log4net.config" -Force;
.\InsightMaker.SettingsUtilities.exe patch -i C:\InsightMaker\ -a "jwtConfiguration.Key" -v "ADD THE KEY HERE"; 

1. Run the following script in an admin PowerShell to update the app file names.

$Filename = "C:\InsightMaker\Apps\Admin\Api";
Copy-Item -Path "$filename\appsettings.default.json" -Destination "$filename\appsettings.json" -Force;
Copy-Item -Path "$filename\log4net.default.config" -Destination "$filename\log4net.config" -Force;
Copy-Item -Path "$filename\web.default.config" -Destination "$filename\web.config" -Force
$Filename = "C:\InsightMaker\Apps\Admin\app";
Copy-Item -Path "$filename\web.default.config" -Destination "$filename\web.config" -Force;
$Filename = "C:\InsightMaker\Apps\search\api";
Copy-Item -Path "$filename\appsettings.default.json" -Destination "$filename\appsettings.json" -Force;
Copy-Item -Path "$filename\log4net.default.config" -Destination "$filename\log4net.config" -Force;
Copy-Item -Path "$filename\web.default.config" -Destination "$filename\web.config" -Force;
$Filename = "C:\InsightMaker\Apps\web\api";
Copy-Item -Path "$filename\web.default.config" -Destination "$filename\web.config" -Force;
$Filename = "C:\InsightMaker\Utils\InsightMaker.IndexUtilities";
Copy-Item -Path "$filename\appsettings.default.json" -Destination "$filename\appsettings.json" -Force;
Copy-Item -Path "$filename\log4net.default.config" -Destination "$filename\log4net.config" -Force;
$Filename = "C:\InsightMaker\Utils\InsightMaker.Security.BuiltinSecurityUtilities";
Copy-Item -Path "$filename\appsettings.default.json" -Destination "$filename\appsettings.json" -Force;
Copy-Item -Path "$filename\log4net.default.config" -Destination "$filename\log4net.config" -Force;

Aiimi Insight Engine Configurations

Some values need to be encrypted to ensure security. .\InsightMaker.SettingsUtility encryptstring <value>

1. Open appsettings.default.json in Notepad++.

2. Update the Plugins path points to the plugins folder (likely C:\InsightMaker\Plugins).

3. Run the following script in an admin PowerShell to update your app settings Json file.

   1. Update where the script below says "Text input" with the relevant information.

   2. Ensure the correct Elastic password is set.

      • This password must be encrypted.

   3. The Elasticsearch credentials are set to the elastic login previously created.

      • The password must be encrypted.

   4. Your license signature should be a unique value for this system.

   5. Your license secret must be encrypted.

   6. The jwt configuration key must be encrypted and at least 32 characters.

      • For assistance creating a random 32 character key see https://www.random.org.

.\InsightMaker.SettingsUtilities.exe patch -i C:\InsightMaker\ -a "elastic.certificate.path" -v "C:\Apps\certs\elastic-stack-ca.p12";
.\InsightMaker.SettingsUtilities.exe patch -i C:\InsightMaker\ -a "elastic.certificate.password" -v "Text input";
.\InsightMaker.SettingsUtilities.exe patch -i C:\InsightMaker\ -a "elastic.password" -v "Text input";
.\InsightMaker.SettingsUtilities.exe patch -i C:\InsightMaker\ -a "elastic.prefix" -v "Text input";
.\InsightMaker.SettingsUtilities.exe patch -i C:\InsightMaker\ -a "plugins.locations" -l "C:\InsightMaker\Plugins"
.\InsightMaker.SettingsUtilities.exe patch -i C:\InsightMaker\ -a "remoteApi.certificate.path" -v "C:\Apps\certs\elastic-certificates.p12";
.\InsightMaker.SettingsUtilities.exe patch -i C:\InsightMaker\ -a "remoteApi.certificate.password" -v "Text input";
.\InsightMaker.SettingsUtilities.exe patch -c -i C:\InsightMaker\ -a "licenseKey" -v "Text input";
.\InsightMaker.SettingsUtilities.exe patch -c -i C:\InsightMaker\ -a "licenseSig" -v "Text input";

Configure Log File Storage

Each agent has a log4net.default.config file. This file defines the path for the log directoy. This is C:\tmp\logs by default.

1. Run the following script to update the logs into their own subfolders.

$Filename = "C:\InsightMaker\ContentAgent\log4net.config";
((Get-Content -path $filename -Raw) -replace 'c:/tmp/logs/InsightMaker.ContentAgent.log','C:/tmp/logs/agents/InsightMaker.ContentAgent.log') | Set-Content -Path $filename;
$Filename = "C:\InsightMaker\EnrichmentAgent\log4net.config";((Get-Content -path $filename -Raw) -replace 'c:/tmp/logs/InsightMaker.EnrichmentAgent.log','C:/tmp/logs/agents/InsightMaker.EnrichmentAgent.log') | Set-Content -Path $filename;
$Filename = "C:\InsightMaker\JobAgent\log4net.config";
((Get-Content -path $filename -Raw) -replace 'c:/tmp/logs/InsightMaker.JobAgent.log','C:/tmp/logs/agents/InsightMaker.JobAgent.log') | Set-Content -Path $filename;
$Filename = "C:\InsightMaker\MigrationAgent\log4net.config";
((Get-Content -path $filename -Raw) -replace 'c:/tmp/logs/InsightMaker.MigrationAgent.log','C:/tmp/logs/agents/InsightMaker.MigrationAgent.log') | Set-Content -Path $filename;
$Filename = "C:\InsightMaker\OCRAgent\log4net.config";
((Get-Content -path $filename -Raw) -replace 'c:/tmp/logs/InsightMaker.OCRAgent.log','C:/tmp/logs/agents/InsightMaker.OCRAgent.log') | Set-Content -Path $filename;
$Filename = "C:\InsightMaker\SecurityAgent\log4net.config";
((Get-Content -path $filename -Raw) -replace 'c:/tmp/logs/InsightMaker.SecurityAgent.log','C:/tmp/logs/agents/InsightMaker.SecurityAgent.log') | Set-Content -Path $filename;
$Filename = "C:\InsightMaker\SourceAgent\log4net.config";
((Get-Content -path $filename -Raw) -replace 'c:/tmp/logs/InsightMaker.SourceAgent.log','C:/tmp/logs/agents/InsightMaker.SourceAgent.log') | Set-Content -Path $filename;
$Filename = "C:\InsightMaker\Apps\Search\api\log4net.config";
((Get-Content -path $filename -Raw) -replace 'c:/tmp/logs/InsightMaker.Middleware.search.log','C:/tmp/logs/agents/InsightMaker.Middleware.search.log') | Set-Content -Path $filename;
$Filename = "C:\InsightMaker\Apps\Admin\api\log4net.config";
((Get-Content -path $filename -Raw) -replace 'c:/tmp/logs/InsightMaker.Middleware.Admin.log','C:/tmp/logs/InsightMaker.Middleware.Admin.log') | Set-Content -Path $filename;
$Filename = "C:\InsightMaker\Utils\InsightMaker.IndexUtilities\log4net.config";
((Get-Content -path $filename -Raw) -replace 'c:/tmp/logs/InsightMaker.IndexUtilities.log','C:/tmp/logs/InsightMaker.IndexUtilities.log') | Set-Content -Path $filename;
$Filename = "C:\InsightMaker\Utils\InsightMaker.Security.BuiltinSecurityUtilities\log4net.config";
((Get-Content -path $filename -Raw) -replace 'c:/tmp/logs/InsightMaker.Security.BuiltinSecurityUtilities','C:/tmp/logs/InsightMaker.Security.BuiltinSecurity.log') | Set-Content -Path $filename;

Initialise Control Hub Parameter

In Utils\InsightMaker.IndexUtilities, run the following from an admin command prompt. These commands configure the default elastic mappings for InsightMaker indices and create some useful default entities etc.

cd C:\InsightMaker\Utils\InsightMaker.IndexUtilities;
.\InsightMaker.IndexUtilities.exe map
.\InsightMaker.IndexUtilities.exe initialise -fmsci
.\InsightMaker.IndexUtilities.exe initialise --collection-types;
.\InsightMaker.IndexUtilities.exe initialise --theme;
.\InsightMaker.IndexUtilities.exe upgrade --generate-entity-groups;

Install Services

1. Run the following script in an Admin PowerShall to update the logs into their own subfolders.

sc.exe create InsightMakerContentAgent binPath= c:\InsightMaker\ContentAgent\InsightMaker.ContentAgent.exe DisplayName= "InsightMaker Content Agent" start= auto
sc.exe create InsightMakerEnrichmentAgent binPath= c:\InsightMaker\EnrichmentAgent\InsightMaker.EnrichmentAgent.exe DisplayName= "InsightMaker Enrichment Agent" start= auto
sc.exe create InsightMakerJobAgent binPath= c:\InsightMaker\JobAgent\InsightMaker.JobAgent.exe DisplayName= "InsightMaker Job Agent" start= auto
sc.exe create InsightMakerMigrationAgent binPath= c:\InsightMaker\MigrationAgent\InsightMaker.MigrationAgent.exe DisplayName= "InsightMaker Migration Agent" start= auto
sc.exe create InsightMakerOcrAgent binPath= c:\InsightMaker\OcrAgent\InsightMaker.OcrAgent.exe DisplayName= "InsightMaker Ocr Agent" start= auto
sc.exe create InsightMakerSecurityAgent binPath= c:\InsightMaker\SecurityAgent\InsightMaker.SecurityAgent.exe DisplayName= "InsightMaker Security Agent" start= auto
sc.exe create InsightMakerSourceAgent binPath= c:\InsightMaker\SourceAgent\InsightMaker.SourceAgent.exe DisplayName= "InsightMaker Source Agent" start= auto
sc.exe description InsightMakerContentAgent "InsightMaker Content Agent"
sc.exe description InsightMakerEnrichmentAgent "InsightMaker Enrichment Agent"
sc.exe description InsightMakerJobAgent "InsightMaker Job Agent"
sc.exe description InsightMakerMigrationAgent "InsightMaker Migration Agent"
sc.exe description InsightMakerOcrAgent "InsightMaker Ocr Agent"
sc.exe description InsightMakerSecurityAgent "InsightMaker Security Agent"
sc.exe description InsightMakerSourceAgent "InsightMaker Source Agent"

IIS Configuration

IIS Setup

Initially: Ensure that '.NET Core and HostingBundle' has been installed, otherwise middleware will not be able to run in IIS.

1. Run the following Commands in Admin PowerShell to enable IIS.

Enable-WindowsOptionalFeature -Online -FeatureName IIS-WebServerRole, IIS-WebServer, IIS-CommonHttpFeatures, IIS-ManagementConsole, IIS-HttpErrors, IIS-HttpRedirect, IIS-WindowsAuthentication, IIS-StaticContent, IIS-DefaultDocument, IIS-HttpCompressionStatic, IIS-DirectoryBrowsing 

1. Create a self-signed certificate using the following commands in Admin PowerShell.

   • This is used later and can be updated later for a production build.

$newcert = New-SelfSignedCertificate -dnsname "InsightEngine20" -KeyLength 2048 -CertStoreLocation cert:\LocalMachine\My -NotAfter (Get-Date).AddYears(20) 

1. Create new https (443) bindings and remove http (80) bindings using the following commands in Admin PowerShell.

New-WebBinding -Name "Default Web site" -IP "*" -Port 443 -Protocol https
Get-WebBinding -Port 80 -Name "Default Web Site" | Remove-WebBinding

IIS Application Pools Setup

1. Create a new application pool using the following commands in Admin PowerShell.

$newAppPool = New-WebAppPool -Name "Admin";
$newAppPool.autoStart = "true";
$newAppPool | Set-Item;

1. There are a number of application that need to be added using the following commands in Admin PowerShell.

Import-Module WebAdministration;
Set-ItemProperty "IIS:\Sites\Default Web site" -name physicalPath -value "C:\InsightMaker\Apps\Search\app";
New-WebApplication -Name "API" -Site "Default Web Site" -PhysicalPath "C:\InsightMaker\Apps\Search\api" -ApplicationPool "DefaultAppPool";
New-WebApplication -Name "Admin" -Site "Default Web Site" -PhysicalPath "C:\InsightMaker\Apps\Admin\app" -ApplicationPool "Admin";
New-WebApplication -Name "Admin\api" -Site "Default Web Site" -PhysicalPath "C:\InsightMaker\Apps\Admin\api" -ApplicationPool "Admin";

IIS Assign Certificate to Binding

This sets up the default website on the server. It means it can be accessed using 'localhost' in any browser and the other apps using their alias. For example admin is accessed using localhost/admin.

1. Ensure the account that IIS runs under has read access to your Elasticsearch certificates.

2. Open IIS and navigate to the Default web site.

1. Select Bindings from Edit Site within the actions bar.

2. Select port 443 and Edit.

1. SSL certificate: Select the new certificate you created.

2. Access the following link to check this is set up correctly. https://localhost/#/login

If you see a blank screen it's possible that IIS didn't set itself up correctly you may need to restart IIS.

Tika

Installing Tika

1. Unzip the Tika distribution.

2. Create a directory to store IM utilities, outside of the InsightMaker directory to avoid adding any complication to upgrades (eg. C:\Utils).

3. Copy the Tika server using the following commands in Admin PowerShell.

mkdir C:\Utils\Tika; 
copy-item "C:\insightmaker\Tika\tika-server-standard-2.9.2.jar" -Destination "C:\Utils\Tika"; 

1. Create a run object using the following commands in Admin PowerShell.

   • If your Elasticsearch is stored in a folder with spaces in the name, the Java path will need to be in quotation marks.

'"C:\Apps\elasticsearch-8.14.3\jdk\bin\Java" -jar C:\\Utils\Tika\tika-server-standard-2.9.2.jar' | Out-File -FilePath C:\Utils\Tika\run-tika.bat -encoding ascii; 

1. Create the Tika service using the following commands in Admin PowerShell.

   1. This will open an NSSM service installer modal.

C:\Utils\nssm-2.24\win64\nssm.exe install InsightMakerTika; 

1. Path: Point this field to C:Utils\Tika\run-tika.bat.

   1. We recommend browsing the wizard to find this location instead of copy and pasting the file path.

2. Startup Directory: Point this field to C:Utils\Tika

   1. We recommend browsing the wizard to find this location instead of copy and pasting the file path.

If you want to run Tika without installing then from a command prompt navigate to the folder where you have placed Tika. Then run .\run-tika.bat serve. This will allow you to identify any issues which are less obvious then going straight for the install. (Ctrl + C to close down once tested).

Check Tika Services

1. Access Tika in your web browser on http://localhost:9998.

Password Encryption

1. We recommend you encrypt your build. For help encrypting your build see our Installation Security area.

Restart Agent Services

1. Restart all the agent services using the following commands in Admin PowerShell.

restart-service InsightMakerContentAgent;
restart-service InsightMakerEnrichmentAgent;
restart-service InsightMakerJobAgent;
restart-service InsightMakerMigrationAgent;
restart-service InsightMakerOcrAgent;
restart-service InsightMakerSecurityAgent;
restart-service InsightMakerSourceAgent;
iisreset;

Accessing Control Hub

1. Navigate to http://localhost/admin and login with your elastic credentials. Use the elastic username and password generated or the one you changed it to.

2. Assuming you can login correctly, then you should see your agents on the configuration page. They will all be empty. To get started, please see the Quick Start Guide in the Administration Guide.

Installing Odata API (optional)

1. Navigate to C:/Insightmaker/Apps/OData.

2. Open appsettings.default.json in a source code editor like NotePad++.

3. Check the details are correct:

   • The Plugins path points to the plugins folder. Likely C:\InsightMaker\Plugins

   • The elastic certificate path points to the elastic-stack-ca.p12 file. This was created when you installed Elasticsearch.

   • The correct password is set.

   • The Elasticsearch credentials are set to the elastic login you created.

   • The prefix is set to the correct value and consistent across all components.

4. Add your two Aiimi Insight Engine licences to the root of the json:

   • "licenseKey": "your key"

   • "licenseSig": "your sig"

5. Check the remoteAddress is set to ‘http://localhost/analytics/‘.

6. Save this file, do not change the name or location.

7. Within C:/Insightmaker/Apps/OData open log4net.default.config.

8. Change the file paths to where you’d like log files to save.

   • This can not be the in InsightMaker folder.

9. Save this file, do not change the name or location.

10. Remove the '.default' from each of the following file names:

    • 'appsettings.default.json'

    • 'log4net.default.config'

    • 'web.default.config'

11. Open the web.config file.

    • Check the module is modules="AspNetCoreModuleV2".

12. Open Internet Information Services (IIS)

13. Open the dropdown on the left and Right click on default website.

14. Click add new application, call the alias “analytics”.

15. Set the physical path to the ‘API’ folder in InsightMaker/Apps/OData.

    • You should be able to view the OData metadata document using localhost/analytics/odata/$metadata.