Data Grief – The Start of the Journey to Data Health

Elisabeth Kübler-Ross, a pioneer in the study of  grief identified these five emotional responses: denial, anger, bargaining, depression, and acceptance.  These stages have been used in many other contexts as steps in a journey of discovery.  The journey to optimize data management can bounce around between these stages until an organization sees and accepts their predicament and is ready to begin a journey to comprehensive data management.  Sometimes, well-intentioned actions can move in unintended ways, resulting in these stages of grief.   

Scenario 1: One of company X1’s data analysts just brought in a new set of customer data.   

Scenario 2: One of company X2’s departments just purchased and started to use a SAAS solution to process a portion of the department’s transactions.   

Scenario 3: Company X3’s data warehouse has been constructed with the newest technology and an innovative data architecture. 

Scenario 4: An analyst working for company X4’s finance department with extensive skills in spreadsheets and desktop data bases built a system that extracts data from core transaction systems and stores it his own data lake.  The system automates key corporate calculations that feed quarterly SEC financial disclosures.  

Scenario 5: Company X5’s legal department is charged with managing the protection of HIPPA.   The IT Security Department is also responsible for making sure that data in corporate systems is adequately secured, including HIPPA data.

Let’s follow these scenarios through to their likely conclusion.  I will let the reader fill in the reactions of the various participants based on four of the five emotional responses: denial, anger, bargaining, and depression.  We’ll discuss acceptance at the end of the blog.

Scenario 1: The new dataset duplicates other data already integrated into the organization’s data analytics, except for one new piece of information.  However, some of the overlapping information is not consistent. One group of data analysts adopts the new dataset because of the new piece of information, ignoring the fact that there is overlapping data with existing datasets.  The other set of data analysts use the old datasets.   All is well until an executive looks at two sets of reports and realizes that there are inconsistencies in the same piece of information.  The executive begins interviewing staff to uncover what went wrong. 

Scenario 2: The department manager with the new SAAS system was unhappy that IT priorities put his legacy system enhancements in a large backlog, and he got tired of waiting.  The same department manager did not want to recognize that new SAAS system did not feed downstream applications like the legacy system.  IT moved around priorities with other departments to build an interface to the SAAS system and negotiated with quality assurance to compress the test plan into one business cycle.  In the interim, accounting had to reenter the data into the legacy system, requiring significant overtime.  The interface built by IT was delayed and had numerous bugs.  In the meantime, the accounting data entry effort had numerous errors.  Executives got their reports late.  Two months later, errors were discovered in the financial statements, requiring a restatement. 

Scenario 3: The elegant new data warehouse sat unused because no-one took the time to switch user generated reports to the new system.  When someone did make the switch, two of the key fields were not consistent with the old system.  Analysts went back to using their old data sources.  The senior manager in charge of the project saw a report on the final costs of the project, including significant cost overruns.  This manager then reviewed the usage report for the new data warehouse.  He immediately added the data warehouse project as his main discussion item for the next team meeting.

Scenario 4: The financial analyst who built the new application gave his two week notice when he landed a new job which he got by hyping the work done on his spreadsheet and data lake.  The person assigned to take over the new system was still doing his prior job during the two week transition period.  The application ran fine for 6 months until there was a change in one of the accounts feeding the system.  The new analyst has no idea how the system worked or how to fix it.  The department resorted to making the calculations in a new spreadsheet with a great deal of manual input.  Reports were delayed and management started to question what had happened.

Scenario 5: A senior manager begins to question why he has two different people from two different departments asking him the same questions.

When does the wheel of denial, anger, bargaining and depression end?  Usually it requires that more than one of these scenarios are exposed at the same time, and / or when auditors and examiners discover the issues and findings ensue.   Management then takes a deep breath and accepts that they have a bigger problem.

Why do these issues arise in the first place?  As management begins to investigate these scenarios they often realize that data is the poor step child of their organization.  Different departments are responsible for different aspects of the organization’s data.  In some cases, it is hard to find someone who claims responsibility at all.  Data related initiatives are often not prioritized until problems arise. Then the project objectives are targeted at the most obvious issues, without looking for root causes or broader organizational inefficiencies.

 The organization that sees the context of how these problems have evolved has now moved beyond denial, anger, bargaining, and depression.   Acceptance is the first step of their new journey. 

However, the journey is fraught with dangers and pitfalls.  Our next blog will discuss some of the pitfalls and roadblocks and suggest some alternative paths for effective and comprehensive data management.