Aiimi Insight Engine Disco
User GuidesAiimi
  • Introducing Aiimi Insight Engine
  • Architecture
    • Overview and Key Concepts
    • Hosting Options
    • Architecture and How It Works
      • Agent Servers
        • Security Agent
        • Source Agent
        • Content Agent
        • Enrichment Agent
        • Job Agent
        • OCR Agent
        • Migration Agent
        • Tika Agent
      • Repository
        • Data Node
        • Proxy Node
        • Kibana Node
      • Gateway and User Interface
      • Document and Data Sources
    • Deployment Options
      • Development Environments
      • Testing and QA Environments
      • Production Environments
      • Disaster Recovery Environments
    • Security
      • Source Systems
      • Firewalling
      • Agent Servers
        • Security Agent
        • Source Agent
        • Content Agent
        • Enrichment Agent
        • Job Agent
        • OCR Agent
        • Migration Agent
        • Tika Agent
      • Repository
      • Gateway (Web Server)
      • Tools & Utilities
  • Installation
    • Elasticsearch Installation (Windows)
    • Aiimi Insight Engine Installation (Windows)
    • Python REST Service Install and Configuration
      • Installation and Setup
        • Offline Huggingface Set-up
        • Configuration of Logging
        • Running as a Service (Windows)
        • Using SSL
        • Performance and Concurrency
        • Security
      • Communicating with the Python REST Service
      • Step Configuration
        • Classify
        • Entity Mapper
        • Huggingface NER
        • Language Detection
        • Phrases and Topics
        • Summary
        • Spacyner
    • HTML Cleaner Service
  • security
    • Users
      • User Authentication
      • Code of Conduct
      • App Access and Role-Based Access
    • Data & Documents
      • Document and Data Access Control Lists
      • Progressive Access and Privileged Access
        • Progressive Access
        • Privileged Access
      • Source Scope and Visibility within Apps
      • Classification
      • Advanced Security Controls - Security Classifications
      • Risk Ratings
      • Redaction and Anonymisation
      • Banned Words
      • Mark as Sensitive
      • Audit Controls
  • Control Hub
    • Configurations
      • Managing Configurations
        • Edit and Delete
        • Find a Configuration
        • Monitoring Stats
        • Stop a Configuration
        • Refreshing
      • Security Configurations
        • Configuring Security
        • Active Directory
        • Azure Active Directory
        • BuiltinSecurity
        • Security Sync
        • Security Agents
        • Security Scheduling
      • Source Configurations
        • General
        • Source
          • Azure Blob Storage
          • BBC Monitoring
          • Big Query Cataloguer
          • BIM360
          • CSV Data Loader
          • Confluence
          • Content Server
          • Data File Cataloguer
          • Document Store
          • Dropbox
          • Exchange 365
          • Filesystem
          • Google Bucket
          • JSON Data Loader
          • ODBC Data Loader
          • PowerBi Cataloguer
          • Reuters Connect
          • ShareFile
          • SharePoint
          • SharePoint Legacy
          • SQL Server Cataloguer
          • Websites
          • XML Data Loader
        • Crawl
        • Source Agents
        • Source Scheduling
        • Advanced
      • Enrichment Configurations
        • Creating a Pipeline
          • General
          • Steps
            • Tika Text Extraction
            • External Links
            • Delete
            • Copy
            • Text Cleaner
            • Data Rule Processor
            • Checksum
            • OcrRest
            • AccessMiner
            • CAD Extractor
            • Trie Entity Extractor
            • PCI Extractor
            • Email Extractor
            • Geotag
            • Google Vision Extractor
            • Google NLP Extractor
            • Metrics Calculation
            • Microsoft Vision Extractor
            • Entity Rule Processor
            • Anonymiser
            • Set Document Risk
            • Content Retrieval
            • REST
          • Filters
          • Agents
          • Schedule
          • Advanced
      • OCR Engine
      • Job Configurations
        • General
        • Job
          • AutomatedSearchJob
          • Command Job
          • ElasticJob
          • Extended Metrics Job
          • GoogleVaultSAR
          • Nightly Events Processor Job
          • Notifications Processor Job
          • Portal Sync Job
          • Purge Job
          • Text Content Merge Job
        • Output
        • Agents
        • Scheduling
      • Migration Configuration
        • General
        • Filter
        • Metadata Mappings
        • Agents
        • Scheduling
        • Advanced
      • Content Server
    • Credentials
      • Create a Credential
      • Find a Credential
      • Edit a Credential
      • Delete a Credential
    • Entities
      • Create a Group
      • Create an Entity
      • Managing Entities
      • Import or Export Entities
    • Models
      • Create a New Model
      • Find a Model
      • Enable or Disable a Model
    • Featured Links
    • Global Settings
      • General
        • Thumbnails
        • Windows Authentication
        • SAML2 Authentication
        • ADFS
        • Stackdriver
        • Document Recommendations
        • Searchable PDF Storage
        • Versioning
        • Results
        • Marking Useful Results
        • Folder Browsing
        • Cascading Search
        • Search Suggestions
        • Delve Settings
        • Collections
        • Miscellaneous
      • Application Access
      • Search Relevancy
        • Core Settings
        • Makers Algorithm
        • Filename Boost Layer
        • Minimum Matching Terms Filter
        • Field Boost
        • Modified Date Boosting
        • Hit Highlighting
        • Why My Search Matched
        • Data Search Strategy
      • Search Performance
        • Accurate Totals for Pagination
        • Source cards
        • Facets
      • Filtering
      • Presets
        • File Groups
        • Security Groups
      • Code of Conduct
      • Metrics
      • Viewer
      • SAR
        • Importing Data For A SAR
        • Getting SAR data from Google Vault
        • Redacting Information
        • SAR Access
      • Privacy Portal
        • Activate the Privacy Portal
        • Disclosure
        • Submit SARs From The Privacy Portal
        • Email Delivery Settings
          • Delivery Settings
          • Brand Settings
          • Customise Emails
        • SMS Delivery Settings
        • Requestor Message Limit
        • Attachment Configuration
        • Password Configuration
        • File Scanner Configurator
      • Visualisations
        • Related Result Connections Diagram
        • Event Timeline
        • Create and Modified Date Activity Chart
        • Relationship Map
      • Notifications
      • Map View
      • App
      • Theming
        • Details
        • Logo
        • Background Images
        • Colours
      • Related Results
      • OData API
      • Bulk Search
        • Managing a Bulk Search
    • User Settings
    • Stats
  • API Guides
    • Insight API Guide
      • Swagger Documentation
      • Trying Some Endpoints
      • Search Filter
      • Hits / Items
      • Inspecting REST Calls
    • Data Science API Guide
      • REST Interface
        • Login
        • Datasets
        • Fields
        • Field Statistics
        • Search
        • Scroll
        • Update
      • Python Wrapper
        • Login
        • Datasets
        • Fields
        • Field Statistics
        • Search
        • Query Builders
        • Scroll
        • Scroll Search
        • Update Single Document
        • Bulk Update
    • Creating a Native Enrichment Step
      • Creating an Enrichment Step
        • Creating the Core Classes
        • Extending our Enrichment Step
        • Adding a Configuration Template
        • Adding the Enrichment Step
        • Creating an Enrichment Pipeline
      • Other Tasks
        • Entities, Metadata and Data
        • Accessing the Repository
      • Example Code
      • Troubleshooting
    • Creating a Python Enrichment Step
      • Creating an Enrichment Step
        • Running the Example from Command Line
        • Running the Example
      • Creating Your Own Step
      • Adding or Changing Entities, Metadata
  • whitepapers and explainers
    • Methods for Text Summarization
      • Application
      • Technology Methods
      • Commercial Tools
      • Key Research Centres
      • Productionisation
      • Related Areas of Text Analytics
      • Conclusion
      • References
Powered by GitBook
On this page
  • General
  • Inclusions and Exclusions
  • Crawl Details
  • Advanced Options
  • Meta tag mappings
  1. Control Hub
  2. Configurations
  3. Source Configurations
  4. Source

SharePoint

PreviousShareFileNextSharePoint Legacy

Connect your SharePoint source to Aiimi Insight Engine to make the most of your data. Once you have selected a Source System type more detail will expand to customise this.

General

  1. Select the type of authentication to use when connecting to SharePoint from Authentication Mode.

  2. Choose the credentials to use, this will contain both the Username and Password used to connect to SharePoint.

  3. If your SharePoint Online does not have a Username and Password you must use a secret. From Select Credential (Secret Only) select the matching secret credential for your SharePoint.

    • You will need to create a credential "SecretOnly" for it to appear in this list.

  4. Enter the SharePoint online Client ID to use.

  5. Enter the root URL of the site collection in Site Collection URL that needs crawling.

    • Formatted https://[site]-admin.sharepoint.com

  6. If your site collections do not match the Site Collect Lists, make sure you update these.

    • Do not include O365

Inclusions and Exclusions

  1. Choose the sites within the root collection to crawl.

    • Enter in specific Sites or leave blank to crawl all sites.

  2. If sites need to excluding from the Crawl add them to Sites to Exclude.

  3. Libraries can be excluded from the Crawl by adding them to the Excluded Libraries list.

    • The default excluded libraries can be removed or edited as needed.

  4. Anything added to Included Libraries will crawled.

    • This list overrides any exclusions if there is an overlap.

  5. Add extra Permission Groups to those from the security sync. Allows SharePoint groups in content permissions.

  6. Exclude Permission groups to ignore them from every file.

    • The default Excluded Permission Groups can be removed or edited as needed.

Crawl Details

  1. To reduce timeouts you can change the Page Load Throttling.

    • By changing this you can increase the wait time between Chromium Page loads.

    • Enter the time in MS (2000 ms = 2 seconds).

  2. To get the owner of a file enter the Owner Content Type Mapping field.

    • Formatted like SharePoints internal naming conventions. Content_x0020_Owner

  3. Limit the content per request to reduce timeouts.

    • By lowering the Content Limit Per Request more requests will run but each run will be quicker.

  4. Set a timeout value for all requests made to Sharepoint.

    • Enter a CSOM value into the SharePoint Client Context Timeout.

    • By default this is set to -1.5.

  5. If Chromium has been manually deployed enter the full path including chrome.exe into Path to Chromium.

    • If this field is left black it will be downloaded once.

  6. Within Path to SharePoint Online enter the path to your preferred cookie authentication utility.

    • This will handle authentication when processing ASPX pages.

    • You must explicitly enter the full path including the .exe.

    • If this field is left blank Default utils install locations will be used.

Advanced Options

  1. Process without Delta Tokens - Check this to NOT use the library delta tokens.

    • This will do a reprocess of the full library at every crawl and can be enabled to sync an index. All documents will be retrieved from the library, regardless of changes. However, only necessary changes will be made to your index.

      • If this is not checked, only the changes since the last change token will be retrieved from the library. Likewise only the necessary changes will be made to your index.

    • To use SharePoint change tokens to enable delta crawls uncheck this option.

  2. Permission-less Crawl - Check this to NOT retrieve document permissions from SharePoint.

    • All documents will be set to the defined permission groups.

  3. Do Permissions Check - Check to test all the document permissions.

  4. Get Permissions In Bulk - Check to gather all document permissions for everything in a library at once.

    • The higher the document count the more memory intensive it will be.

    • Not retrieving the permissions in bulk will mean a permission request for every file indexed. This can get you rate limited by SharePoint if many requests are sent in quick succession.

  5. Build Site Caches - Check to build a cache of all users with access to a site.

    • If unchecked validation will be processed with each document.

  6. Include OneDrives - Check this to process OneDrives and sites.

  7. Do Metadata/Entity Update - Check to reprocess all entity or metadata mappings at crawl.

    • This should only be enabled when a field of column has been deleted.

    • Note files updated will need to go through enrichment again after the crawl.

  8. Generate Edit in Browser Link - Check to generate an Open In browser app as default (SP Online only).

  9. Generate Open in App Link (MS Office docs only) - Check to generate an Open In link as SharePoint does.

    • For example, Open in word on the Users Machine (SP Online).

    • This will override the open in browser option.

Meta tag mappings

  1. You can map the entity fields to meta tags from the web page.

    • Enter the full entity field in the left column. For example, entities.Websites.category.

    • Enter the meta tag name in the right column

  2. You can map the metadata fields to meta tags from the web page.

    • Enter the full metadata field in the left column. For example, metadata.webtype.

    • Enter the meta tag name in the right column