Resolution + a 14-day credit

Issue: For 50% of our users who had 2-way integration turned on, during the time period Oct 2nd - Oct. 6th (as shown in your logs), some of the profiles were removed from the people groups within CCB). No CCB records were ever deleted. We don’t have any scripts that will delete your records from your account.

Dilemma: We immediately rolled back the code on Oct. 6th and our systems were back to normal. However, to be able to avoid further issues, we took action to disable syncing temporarily. After trying to unsuccessfully revert the data, we assumed we needed the involvement of the CCB Support Team.

Additional Research: After further investigation, we realized the segment of users affected was smaller than we thought, so we re-enabled syncing for the accounts that weren’t affected. We also began to re-enable accounts that may have been affected but had given us the ok to turn the switch back on. While we were waiting for the CCB Support Team to access the situation, we decided to create a plan B option—a way to revert the data using the information we have. We looked at several possible solutions and finally found a plausible option.

Solution: We are finalizing the coding and testing as I type, to be able to revert all accounts affected by our errant code. The way we are doing this is by using the API call (how we communicate with CCB or MC), finding the specific API calls we did that caused the issue and using that metadata to reverse the process—stuff that gets nerds like us excited. For you, we are just happy to be able to move forward. This reversion will be done by tomorrow after C.O.B unless you tell us not to. Reply if you want us NOT to proceed.

Caveats?
Not much at all. Technically, we are doing the opposite of what our errant code did. Even if you started adding back the names to that group, if our code finds a matching existing profile, we will not do anything, assuming it’s already been done.

The only issue is more administrative. Let’s say that between the time we stopped syncing and now, you decided to swap out a group's members and repurpose that group for something else. Our revert is going to add back the names that we initially removed. I don't see this as something a church would do. Did you? If so, please be in touch.

14-day Extension

We have prorated all accounts affected by our coding issue (2-way syncing enabled before Oct. 6th) to delay payments by 14 days. So, your next typical renewal date will be 14 days later but you still have continued access to our system. It’s our small way to say sorry for the issues caused and thank you for working with us.

The Silver Lining
You got us where you want us. We are so eager to continue to be loved that we are going to fast-track tons of requests which you guys have made over the last few months. We have already developed a version of the “undo” option that we will roll out in the coming weeks. The staging server is done and being connected to our workflow. We are storing your logs in a format that ensures we can revert faster in the future.

Other projects include logs via email on each sync, a better way to handle unsubscribes from MC, options for how we should overwrite your data if conflicts are found, and so much more.

Questions?
Join us at our weekly office hour to get other questions answered. https://ccbsync.com/