Lessons From The Give Local America Day Kimbia SNAFU Are A Work In Progress

In the wake of events following the Give Local America Day calamity that saw the entire program grind to a halt at mid-day and leave thousands of donors and nonprofits stranded, there is no shortage of commentary. Details surrounding why the web based payment platform collapsed are scant and the online fundraising and crowdfunding platform at the center of the storm, Kimbia, continues to attract a considerable amount of ire.

Adaptistration People 08282As a technology provider that specializes in performing arts organizations, a worst-case scenario like this is always a concern. Working with clients via many of the same considerations surrounding the Give Local America campaign is a routine aspect of that work; as such, I wanted to offer up a bit of insider perspective to the chorus of non-tech based discussion surrounding this incident.

Although Kimbia has yet to release a detailed post-mortem, they have provided enough information to get a good idea of what happened. Keep in mind, the views presented here are not in any way definitive but they should help provide a better understanding of how these issues are connected.

Per Kimbia’s press release, here are the three major points from their technical overview of the shutdown in service:

  1. Kimbia: Removed the affected hardware from service.
    In English: One or more components in the data center that houses the machines responsible for handling traffic failed. It is unknown if this was simply bad timing or the result of something triggered by a software based action.
  2. Kimbia: Reduced leaderboard functionality and focused solely on the ability to serve donation forms
    In English: The leaderboard was likely causing a larger than anticipated server resource strain, which could have contributed to the hardware failure. In turn, disabling that functionality would reduce server load and free up those resources for critical payment processing action (which is what they mean by serving up donation forms). Why that leaderboard may have caused so much strain is as of yet, unknown.
  3. Kimbia: Implemented measures to reduce other potential risks.
  4. In English: In all likelihood, they likely threw a combination of additional software and hardware related solutions at the problem. Likely suspects here include reducing as many functions as possible contributing to the overall server load but weren’t necessary for payment processing and moving the entire platform to a server capable of handling the increased server load.

I am particularly anxious to see if Kimbia will release any technical details surrounding the issues, especially those related to server load and apache requests. Moreover, it will be fascinating to see if any new site functionality introduced for the 2016 campaign tied to donation processing but not included in typical load testing may have produced higher than anticipated server loads.

It Isn’t Likely A Problem With Scaling, Capacity, Or Nonprofits Not Being Tall Enough To Ride The Giving Day Coaster

The part you want to be prepared to wrap your head around is even though capacity (i.e. web traffic) likely triggered the failure, it wasn’t the cause.

As such, much of the online discussion I’ve come across on this topic, including a thoughtful post on 5/8/16 from Beth Kanter, tend to head down rabbit holes that may inadvertently lead away from cause-centric solutions.

Simply put, the problems experienced during the May 5th episode are probably easier to avoid than not. My hunch is the trouble stems not from a lack of necessary hardware, or inadequate load testing, but more with inefficient platform design.

And when compared to something like the discussions going on about ecommerce scaling, this is a much simpler nut to crack.

Time, and Kimbia’s transparency, will tell.

But until we cross that threshold, the notion that a large, nation-wide giving event is by its very nature prohibitive or out of reach for the nonprofit sector is ultimately self-defeating. Likewise, considering alternatives to a shortcoming that has too many unknowns may be therapeutic, but it could inadvertently do more harm than good if it goes too far.

How Did You Handle The Shutdown?

Many groups, including most of my clients participating in the event, did a good job at leveraging social media and direct email marketing to redirect donors to their existing donation platforms. And yes, I will be the first to admit feeling a twinge of anxiety driven tightness in my chest but thankfully, everything operated as expected.

Did your organization participate in Give Local America Day? If so, what did you do as a result of the shutdown (other than grow a few extra gray hairs)?

About Drew McManus

"I hear that every time you show up to work with an orchestra, people get fired." Those were the first words out of an executive's mouth after her board chair introduced us. That executive is now a dear colleague and friend but the day that consulting contract began with her orchestra, she was convinced I was a hatchet-man brought in by the board to clean house.

I understand where the trepidation comes from as a great deal of my consulting and technology provider work for arts organizations involves due diligence, separating fact from fiction, interpreting spin, as well as performance review and oversight. So yes, sometimes that work results in one or two individuals "aggressively embracing career change" but far more often than not, it reinforces and clarifies exactly what works and why.

In short, it doesn't matter if you know where all the bodies are buried if you can't keep your own clients out of the ground, and I'm fortunate enough to say that for more than 15 years, I've done exactly that for groups of all budget size from Qatar to Kathmandu.

For fun, I write a daily blog about the orchestra business, provide a platform for arts insiders to speak their mind, keep track of what people in this business get paid, help write a satirical cartoon about orchestra life, hack the arts, and love a good coffee drink.

Related Posts

  • Not unlike many across the country, I became absorbed with current events yesterday so today's original topic will have to wait. It's been an…

    Adaptistration People 143
  • Whether you're home all day enjoying the holiday with family and friends or your orchestra is putting on a performance today, try to take…

    Adaptistration People 118
  • Whether you're home all day enjoying the holiday with family and friends or your orchestra is putting on a performance today, try to take…

    Adaptistration People 118

2 thoughts on “Lessons From The Give Local America Day Kimbia SNAFU Are A Work In Progress”

    • Hi Beth, that’s where I’m in the “work in progress” camp. Assuming Kimbia releases enough technical details about the shutdown, those solutions should be fairly straightforward.

      My go-to hypothesis at this point still focuses on the “inefficient platform design” direction (over-designed/under-engineered). If that ends up being the case, the good news is it removes the hardware capacity variables related to scaling up for spike level traffic from the list of concerns. Consequently, the focus would shift toward a tighter development process geared toward keeping processes that contribute to aggregated server load under control.

Leave a Comment