Subject: Re: Looking for a lot of NGP DownTime
From: Manisha Patel <[email-masked]>
To: "Johnson, Matt" <[email-masked]>
Cc: Alan Reed <[email-masked]>, "Greeson, Katja" <[email-masked]>, "Parrish, Daniel" <[email-masked]>, "Hoffman, Alex" <[email-masked]>, Jessica TeSelle <[email-masked]>, Andrew Brown <[email-masked]>, "Wilson, Jackie K" <[email-masked]>, Yared Tamene <[email-masked]>, "Ellis, Lizzie" <[email-masked]>
Date: Tue, 24 May 2016 12:49:38 -0700

All good with downtime on our end.

On May 24, 2016, at 3:48 PM, Johnson, Matt <[email-masked]<mailto:[email-masked]>> wrote:

About the downtime:
Does this work for departments?

About Nicknames:
We definitely should, but it's hard to find some of those odd differences on a large-scale fashion.  Happy to take a look after this round is done.

About these duplicates:
Some common issues with these duplicates:
First Name/last name is swamped between two accounts.
Last Name " Tibbetts-Cape" in one account, "Tibbetts" in the other.

I should have better counts on them later today.

I'm happy to send around a sample of the "problem merges" to anyone who is interested in looking into it.

-Matt

From: Alan Reed
Sent: Tuesday, May 24, 2016 3:04 PM
To: Greeson, Katja; Johnson, Matt; Manisha Patel; Parrish, Daniel; Hoffman, Alex; Jessica TeSelle
Cc: Andrew Brown; Wilson, Jackie K; Yared Tamene; Ellis, Lizzie
Subject: RE: Looking for a lot of NGP DownTime

Just curious, would alternate spellings of names be considered in a second wave if they have other matching points?  Just trying to figure out why we wouldn’t merge “Matt” and “mat” in the example below or a Rob, Bob, Robert scenario.

From: Greeson, Katja
Sent: Tuesday, May 24, 2016 3:01 PM
To: Alan Reed; Johnson, Matt; Manisha Patel; Parrish, Daniel; Hoffman, Alex; Jessica TeSelle
Cc: Andrew Brown; Wilson, Jackie K; Yared Tamene; Ellis, Lizzie
Subject: RE: Looking for a lot of NGP DownTime

Full address and full name match.

From: Alan Reed
Sent: Tuesday, May 24, 2016 3:00 PM
To: Johnson, Matt; Greeson, Katja; Manisha Patel; Parrish, Daniel; Hoffman, Alex; Jessica TeSelle
Cc: Andrew Brown; Wilson, Jackie K; Yared Tamene; Ellis, Lizzie
Subject: RE: Looking for a lot of NGP DownTime

What is the criteria for a potential merge?

From: Johnson, Matt
Sent: Tuesday, May 24, 2016 2:50 PM
To: Greeson, Katja; Alan Reed; Manisha Patel; Parrish, Daniel; Hoffman, Alex; Jessica TeSelle
Cc: Andrew Brown; Wilson, Jackie K; Yared Tamene; Ellis, Lizzie
Subject: Looking for a lot of NGP DownTime

Hey Team,
  Direct Marketing recently sent all of the NGP records through a data-hygiene process, which highlighted over 320,000 duplicate records in NGP. I would love to merge these duplicates in NGP, as they cause a lot of problems.
There's two concerns with this: making sure we should merge these duplicates, and getting time that NGP can be slow to process them.

Short version:
Most of the duplicates look like we should merge them (more of that below), which means we need 160 hours of slow NGP time to process them. This time can be broken up and separated, as we can do a few a night.
I was hoping to process them after 8pm on weekdays and over weekends for the next 2-3 weeks. During these times, NGP would be unavailable or extremely slow. If we could process everything straight through this holiday day weekend, we could get over half of them done by next Tuesday.

Before I email all NGP users, I wanted to double-check: does NGP slow time after 8pm and during weekends work for your department? Is there a change we can make that would be fine?

Longer Version
As I said above, there's two concerns with duplicates from NGP:

1)      We need to double-check these duplicates ARE duplicates

2)      We need to schedule time to merge them.

About the Duplicates
We are researching the full impact of these duplicates on the file right now, but 47% of them are low dollar donors who only given once. I have a few select counts below:

Returned Records         :  328758
Unique Records             :  157505 (ie, number of record we should have at the end)
Last Gift 2007                 :  7101
Last Gift 2008                 :  31109
Last Gift 2009                 :  16413
Last Gift 2010                 :  31915
Last Gift 2011                 :  14594
Last Gift 2012                 :  37788
Last Gift 2013                 :  24888
Last Gift 2014                 :  46178
Last Gift 2015                 :  27341
Last Gift 2016                 :  19524

Running counts of EXACT differences (ie, "Matt" and "Mat" would count as a different name).
Merges with different names    :  52849         (25%)
Merges with different Address :   42102        (13%)
Merges with different City         :   6815          (2%)
Merges with different States(!) :   275           (less than a 1%)
Dups with 3+ merges                  : 11,297       (3%)
Dups with 4+ merges                  : 1,986         (less than a percent)

Most of these donations would NOT impact FEC reports we have already made, as they are low-dollar donors well under the FEC report. I'm still getting an exact number, but I have over 75000 we should be fine with right now.

As always, I would love everyone's opinion on this about things we should look out for.

About the DownTime
Merging duplicates takes time. We can merge a lot of an hour, but we're still looking at 160 hours of processing time. In order to get this done quickly (pre-primary, pre-next FEC report, pre-next mail list, so on and so on), I want an aggressive period of downtime. I was hoping to run them overnight and weekends, thus allowing NGP to be up during business hours.

It seems most activity on NGP is done after 8pm every night, which means if we run after 8pm and over the weekends, we could process this in 2-3 weeks.

As we work to pindown the duplicates, I want to double-check: do these hours work with your teams?

I'm also happy to discuss this or anything related to this in a meeting.

Matt Johnson
Technical Financial Manager
Democratic National Committee
Office: 202-572-5478
[email-masked]<mailto:[email-masked]>