Aiimi Insight Engine Florina
User GuidesAiimi
  • Introducing Aiimi Insight Engine
  • Architecture
    • Overview and Key Concepts
    • Search Flows
      • Search Flow Types
      • Smart Filtering
      • Query and Prompt Classification
      • Search Algorithms
      • Extractive and Generative Models
    • Hosting Options
    • Architecture and How It Works
      • Agent Servers
        • Security Agent
        • Source Agent
        • Content Agent
        • Enrichment Agent
        • Job Agent
        • OCR Agent
        • Migration Agent
        • Tika Agent
      • Repository
        • Data Node
        • Proxy Node
        • Kibana Node
      • Gateway and User Interface
      • Document and Data Sources
    • Deployment Options
    • Security
      • User Security
      • Data and Document Security
        • Progressive Access
        • Privileged Access
      • Source System Security
      • Firewalling
      • Agent Servers
        • Security Agent
        • Source Agent
        • Content Agent
        • Enrichment Agent
        • Job Agent
        • OCR Agent
        • Migration Agent
        • Tika Agent
      • Repository
      • Gateway (Web Server)
      • Tools & Utilities
  • Installation
    • Elastic and Kibana Install (Windows)
    • Aiimi Insight Engine Installation (Windows)
      • Installation Security
    • AI Services
      • Prerequisites
      • AI Enrichment Service
        • Installation and Setup
        • Enabling Enrichment Steps
        • Using AI Enrichment Steps
        • Performance and Concurrency
      • AI Model Service
        • Installation and Setup
        • Enabling Providers
        • Private Generative AI
        • Azure Open AI
      • Configuration of Logging
      • Offline Set-up of Models
      • Using SSL
      • Running as a Service (Windows)
      • Using GPUs
      • AI and Semantic Search Set Up
        • Open & Closed Book AI
        • Semantic Search
          • Vectors for Semantic Search
          • Source Configuration
          • Sentence Transformer Models
          • Enrichment
          • Kibana
          • Final Search Flow
    • HTML Cleaner Service
  • Control Hub
    • Configurations
      • Config Management
      • Security Configurations
        • Security - General
        • Security - Source
          • Active Directory
          • Atlassian
          • Azure Active Directory
          • Builtin Security
          • Miro Security
          • Google Directory
          • Slack Security
        • Security - Sync
        • Security - Agents
        • Security - Scheduling
      • Source Configurations
        • Source - General
        • Source - Source
          • Alfresco Kafka
          • Azure Blob Storage
          • BBC Monitoring
          • Big Query Cataloguer
          • BIM360
          • CSV Data Loader
          • Confluence
          • Content Server
          • Data File Cataloguer
          • Document Store
          • DocuSign
          • Dropbox
          • Exchange 365
          • Filesystem
          • Google Bucket
          • Google Drive
          • Google Vault
          • Jira
          • JSON Data Loader
          • Livelink
          • Microsoft Teams
          • Mimecast
          • Miro
          • ODBC Data Loader
          • PowerBi Cataloguer
          • Reuters Connect
          • ShareFile
          • SharePoint
          • SQL Server Cataloguer
          • Slack
          • Versioned Document Store
          • Websites
          • XML Data Loader
        • Source - Crawl
        • Source - Agents
        • Source - Schedule
        • Source - Advanced
      • Enrichment Configurations
        • Creating a Pipeline
          • General
          • Steps
            • AccessMiner
            • AI Classification
            • Anonymiser
            • CAD Extractor
            • Checksum
            • Content Retrieval
            • Copy
            • Data Rule Processor
            • Delete
            • Email Extractor
            • Entity Rule Processor
            • External Links
            • Geotag
            • Google NLP Extractor
            • Google Vision Extractor
            • Metrics Calculation
            • Microsoft Vision Extractor
            • OcrRest
            • Office Metadata
            • PCI Extractor
            • REST
            • Set Document Risk
            • Text Cleaner
            • Tika Text Extraction
            • Trie Entity Extractor
            • Update Metadata
          • Filters
          • Agents
          • Schedule
          • Advanced
      • OCR Engine
      • Job Configurations
        • General
        • Job
          • AutomatedSearchJob
          • Command Job
          • ElasticJob
          • Extended Metrics Job
          • GoogleVaultSAR
          • Google Drive Last Access Date
          • Nightly Events Processor Job
          • Notifications Processor Job
          • Portal Sync Job
          • Purge Job
          • Text Content Merge Job
        • Output
        • Agents
        • Scheduling
      • Migration Configuration
        • General
        • Filter
        • Metadata Mappings
        • Agents
        • Scheduling
        • Advanced
    • Credentials
      • Create a Credential
      • Find a Credential
      • Edit a Credential
      • Delete a Credential
    • Mappings
      • Entities
        • Managing Groups
        • Create an Entity
        • Managing Entities
      • Models
        • Create a New Model
        • Find a Model
        • Enable or Disable a Model
      • Vectors
      • Rank Features
    • Featured Links
    • AI Settings
      • Classifications
      • Class
      • Class Rules
      • AI Classification
    • Global Settings
      • General
        • Stackdriver
        • Document Recommendations
        • Searchable PDF Storage
        • Versioning
        • Results
        • Marking Useful Results
        • Folder Browsing
        • Cascading Search
        • Search Suggestions
        • Delve Settings
        • Miscellaneous
      • Authentication
      • Application Access
      • Search Relevancy
        • Core Settings
        • Makers Algorithm
        • Filename Boost Layer
        • Minimum Matching Terms Filter
        • Field Boost
        • Modified Date Boosting
        • Hit Highlighting
        • Why My Search Matched
        • Data Search Strategy
      • Search Performance
      • Filtering
      • Thumbnails
      • Presets
      • Code of Conduct
      • Metrics
      • Viewer
        • Redacting Information
        • Watermarking
      • SAR
        • Importing Data For A SAR
        • SAR Disclosure Document Storage
        • Getting SAR data from Google Vault
        • SAR Access
        • SAR File Status
      • Collections
      • Disclosure Portal
        • Disclosure Portal Set Up
        • SARs From The Portal
        • Email Delivery Settings
          • Delivery Settings
          • Brand Settings
          • Customise Emails
        • SMS Delivery Settings
        • Requestor Message Limit
        • Attachment Configuration
        • Password Configuration
        • File Scanner Configurator
      • Visualisations
        • Related Result Connections Diagram
        • Event Timeline
        • Timeline Lens Activity Chart
        • Relationship Map
      • Notifications
      • Map Lens
      • App
      • Theming
        • General
        • Layout
        • Colours
      • User Avatar
      • Related Results
      • OData API
      • Bulk Search
        • Managing a Bulk Search
      • Search Flows
        • Create a Search Flow
          • General
          • Query Classification Step
          • Search Steps
          • Model Steps
      • Uploads
      • Security
    • User Settings
    • Stats
      • Data Views
  • API Guides
    • Insight API Guide
      • Swagger Documentation
      • Trying Some Endpoints
      • Search Filter
      • Hits / Items
      • Inspecting REST Calls
    • Data Science API Guide
      • REST Interface
        • Login
        • Datasets
        • Fields
        • Field Statistics
        • Search
        • Scroll
        • Update
      • Python Wrapper
        • Login
        • Datasets
        • Fields
        • Field Statistics
        • Search
        • Query Builders
        • Scroll
        • Scroll Search
        • Update Single Document
        • Bulk Update
    • Creating a Native Enrichment Step
      • Creating an Enrichment Step
        • Creating the Core Classes
        • Extending our Enrichment Step
        • Adding a Configuration Template
        • Adding the Enrichment Step
        • Creating an Enrichment Pipeline
      • Other Tasks
        • Entities, Metadata and Data
        • Accessing the Repository
      • Example Code
      • Troubleshooting
    • Creating a Python Enrichment Step
      • Creating an Enrichment Step
        • Running the Example from Command Line
        • Running the Example
      • Creating Your Own Step
      • Adding or Changing Entities, Metadata
  • whitepapers and explainers
    • From a Billion To One – Mastering Relevancy
    • Methods for Text Summarization
      • Application
      • Technology Methods
      • Commercial Tools
      • Key Research Centres
      • Productionisation
      • Related Areas of Text Analytics
      • Conclusion
      • References
Powered by GitBook
On this page
  • Introduction
  • General
  • Linux Specific Options
  • Permissions Lookup
  • Crawl Options
  • Retrieval Options
  • Local Database
  • Folder Permissions
  • Content Management
  1. Control Hub
  2. Configurations
  3. Source Configurations
  4. Source - Source

Filesystem

PreviousExchange 365NextGoogle Bucket

Last updated 7 months ago

Introduction

Connect your filesystem source to Aiimi Insight Engine to make the most of the data on your machines. Once you have selected a Source System type more detail will expand to customise this. Source System Settings.

General

  1. You need to enter a starting point for the crawl within File System Root. This should be a path rooted on locally mounted drive or a UNC path.

  2. Discovery and Content agents use an alternative path within Service Root. This allows SAN/NAS/NFS backup partitions to be crawled but still display the address.

  3. If left empty the File SystemRoot will be used.

  4. Select a username from the Select Credential dropdown. This will allow you to connect to the network share.

    • If left black the share will not be mounted.

    • For support setting up credentials use

  5. A free drive letter will be used to mount the drive if credentials are added.

  6. If the network share password changes a system restart is needed on the Network Share system. This ensures all new configurations are persistent.

  7. You can choose to crawl specific folders within a filesystems root. Enter any folder names into the Root Folders field.

  8. If left blank all folders in the root will be crawled.

Linux Specific Options

These settings will be used when mounting to a Linux environment and will be ignored for a windows connection.

  1. Select the Mount Filesystem Type from the dropdown.

    • At the moment CIFS is the only available option.

  2. Enter the Filesystem type version to use when mounting in a Linux environment.

    • If blank, the latest supported version will be used.

    • Format versions 0.0.0 or 0.0.

Permissions Lookup

  1. If the hosting agent servers are not members of or trusted by the domain uncheck "Use direct domain lookup for security identifiers." For all other servers this should remain checked.

  2. You can adjust the number of permission threads that are run along side the indexes. By default we recommend 4.

Crawl Options

  • Is the file system case sensitive - Check this if your filesystem is case sensitive.

  • Folder only crawl - Check to only crawl folders within the source.

  • Enable FileSystem watcher - Check to enable a smart Crawl. Achieve an optimal crawl time by driving crawls directly from changes to target.

    • This is dependent on your file system, network and if it is supported and enabled.

    • The performance of a large NAS/SAN systems may be impacted if enabled.

Retrieval Options

  • Preserve last access date - Where possible, the last access date when they are accessed by system functions will be kept. The service account must have sufficient permissions to update file attributes.

    • This may not be possible with all file systems and they may not allow this.

    • You must also set the enable the content retrieval, preserve last access date for this to function.

Local Database

  1. To enable a local database check used local database.

  2. Enter the location within local database location. This location must not be used by another local database.

Folder Permissions

  1. Check Store Folder Permissions to store NTFS ACLs.

    • This will store them for each folder in the Elastic index. This is used for ACL monitoring solutions and increases the Elastic document size.

Content Management

Allow Aiimi Insight Engine to change a read-only flag on an item before deleting it. The read-only field will be changed as part of a delete action.

If this it not enabled, items flagged as read-only will remain on the course system and will not be deleted.

The crawlers service account must have permission to change file attributes and delete.

  1. Check Delete read-only files.

    • The service account must have permission to change file attributes for this to work.

our guide on managing credentials.