Yesterday, this blog post documented how a bug accidentally cost Carlo $6500 in a few hours because his AWS keys were compromised. Please read the whole post.
Synopsis: He was using the GitHub Extension for Visual Studio. He published a new repository that he *thought* would be private, but due to a bug, he created a public repository. That repository contained his Amazon Access Key. Scanners found it and created lots of resources.
First, some good news and a task for all of you: This issue was reported on the GitHub project for that extension (https://github.com/github/VisualStudio/issues/62). That page shows the quick response by the team involved for that extension, and a fix has already been implemented, and deployed to the Visual Studio Extension Gallery: https://visualstudiogallery.msdn.microsoft.com/75be44fb-0794-4391-8865-c3279527e97d
A task for all of you: If you are using the GitHub for Visual Studio Extension, update it now. You’ll get a notification from Visual Studio that there is an update available. Install it. Now. (This extension does not auto-update, so you will need to perform this task.)
A Bit of Background On How This Happened So Fast
One fact I learned yesterday was that there are organizations that constantly monitor Github commits to see if any contain credentials to AWS or other cloud providers. That is the price of success. A tremendous amount of code is added or modified on github.com every day. It’s an attractive target.
As soon as a public repository gets created, or updated, that information will be captured. Be careful.
What Habits Could Have Prevented This?
Clearly, hindsight is 20:20, I’m writing this to share a few tips that might help you. I’m also interested in hearing what techniques you use to manage these kinds of secrets. My experience is with the open source projects for Humanitarian Toolbox. Those projects are deployed on Azure, and are all Open Source projects on GitHub. That means we must manage to keep secrets out of the repository. Here are the recommendations we use:
No Credentials in *.config
The default web.config files that are in the open source repositories are set for a localDB on the developer’s drive. There are no keys for our deployed resources.
Instead, we use the Azure Portal to configure any and all secrets for our sites. (click on the image for a larger view...) This does mean that we, as an organization, must protect our Azure portal settings. And, folks like me must make sure to obscure the secrets in images like this.
Deploy Directly from Source Control
We deploy our applications directly from GitHub source control. The staging site deploys from the master branch, whenever we merge pull requests. If you are one of our contributors, you can see your changes live as soon as we merge the pull request.
For release builds, we use the Release branch in GitHub. One of the commit teams must merge changes from master to the release branch for those to be live. We do that after our partners have had the chance to validate the changes on Master.
Consider Visual Studio Online
For my projects that are not planned to be open source, but will remain private, I use Visual Studio Online. (https://www.visualstudio.com/en-us/products/what-is-visual-studio-online-vs.aspx) It supports Git, and has some pull request support for teams. (I’d like a model where I could use Pull Requests even with my shared repositories, but I haven’t found that feature yet.)
VSO is designed around a team model, not an Open Source model. It’s less likely (well, almost impossible) that I accidentally make a public repository on VSO.
What Next?
The key takeaway for me is that the era of Open Source requires all of us to learn some new skills and take care of new potential risks. These security breaches can and do happen to experienced people. Teamwork, care, and multiple levels of defense are needed.
Most importantly, professionals did their job: I’m truly impressed with the speed and professionalism of the response to this issue by team members from both Microsoft and GitHub.
