Oct 9 2019 incident: some D7 sites may have lost Acquia Search indexing configuration

    See more
    • Updated

    Issue

    In or around October 9th, 2019, Drupal sites powered by the Drupal 7 Acquia Search and Apache Solr Search Integration modules may have lost indexing settings and tracking information (which keeps track of items to index), as those settings were reset to a default state.

    Symptoms could include:

    • indexing different content types than those intended
    • biasing/scoring may have changed
    • the fields to search within may have changed
    • any other settings kept as variables within apachesolr.module as shown with a drush solr-vget command.

    Note: Other symptoms are discussed in our other KB article: https://support.acquia.com/hc/en-us/articles/360036638154-Oct-9-2019-incident-Search-pages-may-stop-working-on-some-D7-sites

    Resolution

    Note: Acquia has taken steps to protect your October 9th database backups from the automated purge process and keep them available in the Acquia Cloud UI for you to restore, should you want to do so according to the options listed below. 

    This needs Drupal site administrator permissions to diagnose and fix.

    Before you begin, you can first manually log into your site(s) and inspect your Search and indexing settings at /admin/config/search/apachesolr/settings/acquia_search_server_1 to determine if any repairs are needed. Some settings that could have been changed include:

    • Content types to be indexed
    • Biasing options
    • Fields to search within

    We urge you to take caution and:

    • create DB backups before you enact any changes to your sites
    • test thoroughly to ensure the changes represent what you require

    You will need to repeat the chosen steps on any affected sites.

    Option A) On the affected site, perform a restore of a full DB backup from or before Wednesday, 9 October. 

    This will erase all content and configuration created after the backup was created. ONLY choose this option if you are OK with fully replacing ALL content and configuration changes created AFTER October 9, 2019. We strongly recommend creating an on-demand backup immediately beforehand, should you need to recover any lost content changes.

    OPTION B) On a separate environment, restore the database backup and compare what's different. 

    You could for example, restore this DB backup on your Dev environment, then:

    • Log into the your Drupal application and navigate to /admin/config/search/apachesolr/settings/acquia_search_server_1 on both production and non production. 
    • Only copy the needed settings from non production manually to your production environment. 

    Cause

    On Oct 9th, 19:30 UTC, Acquia released a change to the Acquia Network Site Profile Information system (NSPI) which was then reverted at 21:21 UTC. This caused some sites' Acquia Search functionality to become unusable. 

    The change inadvertently caused issues with some Drupal sites using the Drupal 7 version of the Acquia Search module. The module uses information obtained from the above NSPI service; during the event, that information was not available, causing the module to change configuration within the apachesolr.module.

    The Solr backends were not affected; only the connection information used by Drupal was changed so that the backend could not be reachable.

    The problem may only present itself if a site ran a Drupal cron job during the event. If it did, the effect will be present until fixed manually.

    The problem lies in that, when the D7 Acquia Search module could not determine the available backend Solr indexes that were available (because of the temporary NSPI change), then the module would have switched the "Default" connection in the Apache Solr Search Integration module to use a Solr index located at "http://localhost:8983/" and deleted the existing "Acquia Search" connection information, including the indexing settings.

    After the event, the removed connection was later automatically re-created with default settings. This means that information about content types (bundles) to index, bias settings, and faceting settings could have been removed. Acquia is currently investigating methods to recover these settings for any affected customers, and will issue a communication to those affected customers as soon as a method is available.

    Note: A different side effect of the same NSPI service outage are discussed in our other KB article: https://support.acquia.com/hc/en-us/articles/360036638154-Oct-9-2019-incident-Search-pages-may-stop-working-on-some-D7-sites

    Related articles