The purpose of this Root Cause Analysis (RCA) is to determine the causes that contributed to the empty search results for the Healthwise Search API from November 07, 2023 to November 09, 2023.
At 6:00 PM MST, on Tuesday, November 07, 2023, Healthwise deployed a new content database to the production environment. At 1:20 PM MST on Wednesday, November 08, 2023, we were alerted that Learning Center pages were not responding for a client. At 9:15 AM MST, on Thursday, November 09, 2023, development teams investigated the incident and found that no search results were being returned for the search endpoint used by API implementations and escalated the incident as urgent. At 12:13 PM, teams had identified the cause of the issue, and a fix was moved to production at 12:58 PM MST. The total time of the incident was 1 day, 18 hours, and 58 minutes.
Based on the investigation conducted, the team determined the following findings regarding this event:
An incomplete database overwrote the initial database that had passed all validation and quality tests. This incomplete database was deployed to the production deployment on Tuesday, November 07, 2023. This resulted in the search index being incomplete.
Results for search requests were restored when Healthwise engineers deployed a database that could index the articles missing in search results. We are actively working on automated tests to ensure the index used by Healthwise Search API clients to search content aligns with the content the client can access and to add tools to improve manual testing. We have also made sure the new deployment solution is pointed away from production resources until it is completed.This incident is still under investigation and the RCA will be updated when the full investigation is complete. If you are subscribed to the status page unfortunately you will not receive a notice when the RCA is updated to the final version. However, at any time you may come back to this page to see the latest updates.