Don’t put secrets into your repositories!
October 5, 2020 10:06 PM Subscribe
I developed shhgit to raise awareness and bring to life the prevalence of [secrets committed to GitHub repositories].
Finding secrets in GitHub is nothing new. There are many great tools available to help with this depending on which side of the fence you sit.
On the adversarial side, popular tools such as gitrob and truffleHog focus on digging in to commit history to find secret tokens from specific repositories, users or organizations.
On the defensive side, GitHub themselves are actively scanning for secrets through their token scanning project.
Finding secrets in GitHub is nothing new. There are many great tools available to help with this depending on which side of the fence you sit.
On the adversarial side, popular tools such as gitrob and truffleHog focus on digging in to commit history to find secret tokens from specific repositories, users or organizations.
On the defensive side, GitHub themselves are actively scanning for secrets through their token scanning project.
Just to confirm I understand: This is a collection of scripts that scan github repositories for things that look like sensitive data (eg API keys) and then posts them(shhhgit) or stores them for review (gitrob)?
There's not a huge amount of context to the links, but wouldn't it be better to run this kind of script and notify the people affected before publishing it publicly? I can see that at least one of the most recent users is attached to a university email so it wouldn't be hard to get it touch.
It looks like this is what GitHub's own program does, but I'm curious as to if there are non-nefarious uses here.
posted by Braeburn at 3:13 AM on October 6, 2020
There's not a huge amount of context to the links, but wouldn't it be better to run this kind of script and notify the people affected before publishing it publicly? I can see that at least one of the most recent users is attached to a university email so it wouldn't be hard to get it touch.
It looks like this is what GitHub's own program does, but I'm curious as to if there are non-nefarious uses here.
posted by Braeburn at 3:13 AM on October 6, 2020
I know that GitHub work with AWS to react to this, but I hear that you can expect to be pwned in significantly less than a minute if you commit valid keys to a public repo. Basically, there are already plenty of bad actors running, essentially, this without the web frontend, and maybe this kind of awareness will actually make people stop and think before running
I think the author might want to screen out the string EXAMPLEKEY and ABCD....
It is amazing how much there is though... practical people, focused on the problem at hand, trying to make things work, finally getting the light to go "beep", pushing the code and hitting the road. Even for technical people, thinking about what shouldn't work is a cognitive blindspot.
posted by Wrinkled Stumpskin at 3:39 AM on October 6, 2020 [7 favorites]
git add .indiscriminately.
I think the author might want to screen out the string EXAMPLEKEY and ABCD....
It is amazing how much there is though... practical people, focused on the problem at hand, trying to make things work, finally getting the light to go "beep", pushing the code and hitting the road. Even for technical people, thinking about what shouldn't work is a cognitive blindspot.
posted by Wrinkled Stumpskin at 3:39 AM on October 6, 2020 [7 favorites]
There's not a huge amount of context to the links, but wouldn't it be better to run this kind of script and notify the people affected before publishing it publicly? I can see that at least one of the most recent users is attached to a university email so it wouldn't be hard to get it touch.
There are generally two ways to deal with security issues that aren't yours to fix. Contact and name-and-shame. There's probably a FPP to be made on the choice between the two. I haven't thought about it "objectively" in a while, but in the context of what's standard operating procedure in the field, this type of project is not particularly surprising. I'd at least mask some characters in the secrets and the locations.
There's a long history of name-and-shame in the security community. See the Wall of Sheep.
It looks like this is what GitHub's own program does, but I'm curious as to if there are non-nefarious uses here.
Running a tool like this is a great way to catch an "oops" on something that is yours to fix.
posted by bfranklin at 4:32 AM on October 6, 2020 [2 favorites]
There are generally two ways to deal with security issues that aren't yours to fix. Contact and name-and-shame. There's probably a FPP to be made on the choice between the two. I haven't thought about it "objectively" in a while, but in the context of what's standard operating procedure in the field, this type of project is not particularly surprising. I'd at least mask some characters in the secrets and the locations.
There's a long history of name-and-shame in the security community. See the Wall of Sheep.
It looks like this is what GitHub's own program does, but I'm curious as to if there are non-nefarious uses here.
Running a tool like this is a great way to catch an "oops" on something that is yours to fix.
posted by bfranklin at 4:32 AM on October 6, 2020 [2 favorites]
Even for technical people, thinking about what shouldn't work is a cognitive blindspot.
There aren’t many other fields where doing the wrong thing with a tool immediately ruins your life, where that tool doesn’t have some sort of warnings or safety guards on it. The fact that version control tools do this without hesitation is an indictment of the industry and its shameful standards, not a blind spot.
posted by mhoye at 5:02 AM on October 6, 2020 [11 favorites]
There aren’t many other fields where doing the wrong thing with a tool immediately ruins your life, where that tool doesn’t have some sort of warnings or safety guards on it. The fact that version control tools do this without hesitation is an indictment of the industry and its shameful standards, not a blind spot.
posted by mhoye at 5:02 AM on October 6, 2020 [11 favorites]
Shhgit can run against your own local git repository. So a sensible plan would be to run it before you sync your local repo up to the public one on github, so you spot secrets before they get published. There are a number of tools such as automated tests that are run before code gets published already, so this could easily slot in there and stop you publishing a secret in the first place.
Github's own scan only applies when stuff is published to github IIRC, and automatically notifies you and the secret provider - depending upon provider, the secret may be disabled rapidly, or you may be left exposed until you manually fix it. Either way, your 3rd party data is vulnerable for at least some period of time.
The existing exploit tools primarily are intended to run against public github repos to extract other people's secrets, rather than warn you you're about to make a big oops.
Github is the biggest place to put public repos, but not the only one by far. Sshgit could protect you in those situations too.
posted by Absolutely No You-Know-What at 5:08 AM on October 6, 2020 [2 favorites]
Github's own scan only applies when stuff is published to github IIRC, and automatically notifies you and the secret provider - depending upon provider, the secret may be disabled rapidly, or you may be left exposed until you manually fix it. Either way, your 3rd party data is vulnerable for at least some period of time.
The existing exploit tools primarily are intended to run against public github repos to extract other people's secrets, rather than warn you you're about to make a big oops.
Github is the biggest place to put public repos, but not the only one by far. Sshgit could protect you in those situations too.
posted by Absolutely No You-Know-What at 5:08 AM on October 6, 2020 [2 favorites]
The fact that version control tools do this without hesitation is an indictment of the industry and its shameful standards, not a blind spot.
Scanning your code for vulnerabilities really isn't the job of the version control system itself. But if your organization has standards for code-check-ins, then git has a pre-commit hook which will run any arbitrary script, containing your code-standards enforcement code.
Scanning your code for vulnerabilities really isn't the job of the version control system itself. But if your organization has standards for code-check-ins, then git has a pre-commit hook which will run any arbitrary script, containing your code-standards enforcement code.
The pre-commit hook is run first, before you even type in a commit message. It’s used to inspect the snapshot that’s about to be committed, to see if you’ve forgotten something, to make sure tests run, or to examine whatever you need to inspect in the code. Exiting non-zero from this hook aborts the commit, although you can bypass it with git commit --no-verify. You can do things like check for code style (run lint or something equivalent), check for trailing whitespace (the default hook does exactly this), or check for appropriate documentation on new methods.posted by mikelieman at 5:08 AM on October 6, 2020 [4 favorites]
And then there are things like Firebase/Firestore where the API key is apparently not sensitive. That makes sense if you think about the fact it ends up in client-side code, but I did spend a non-trivial amount of time trying to figure out if that was really the case and why the heck everyone was committing them to github.
posted by hoyland at 5:19 AM on October 6, 2020
posted by hoyland at 5:19 AM on October 6, 2020
This. Part of my job is oversight on a consulting Red Team: Every time we run an operation against a company now we almost always find a path (or at least a good part of a path) to a crown jewel in GitHub. And it’s not always a credential (though it can be). It’s finding stuff in GitHub code like:
- Finding out where the one unauthenticated internal mail relay is (without having to scan a large network and set off internal monitoring) and using that to send phishing emails through that bypass all the mail controls and map the faked sender directly to make the mail appear to come from a real user
- Finding the one PowerScript that gets deployed to every workstation that allows you to add administrative users without detection or turn off the endpoint agents without alarm
- Getting the internal domain names, IP addresses, and structure of the internal network (again without having to be on it - incredibly useful when you find an external hole where you can partially access the internal network and need some targets or other insider info)
I mean sometimes you find an entire Nagios backup file with every network device’s monitoring account username/password for a major corporate. I mean it happens. But the amount of just “stuff” on GitHub for a lot of organizations is just scary:
Also last night I was writing a proposal for a company and simply used GitHub to identify a bunch of their internal technologies so I could make the proposal sound like I knew them a little better.
posted by inflatablekiwi at 5:46 AM on October 6, 2020 [10 favorites]
- Finding out where the one unauthenticated internal mail relay is (without having to scan a large network and set off internal monitoring) and using that to send phishing emails through that bypass all the mail controls and map the faked sender directly to make the mail appear to come from a real user
- Finding the one PowerScript that gets deployed to every workstation that allows you to add administrative users without detection or turn off the endpoint agents without alarm
- Getting the internal domain names, IP addresses, and structure of the internal network (again without having to be on it - incredibly useful when you find an external hole where you can partially access the internal network and need some targets or other insider info)
I mean sometimes you find an entire Nagios backup file with every network device’s monitoring account username/password for a major corporate. I mean it happens. But the amount of just “stuff” on GitHub for a lot of organizations is just scary:
Also last night I was writing a proposal for a company and simply used GitHub to identify a bunch of their internal technologies so I could make the proposal sound like I knew them a little better.
posted by inflatablekiwi at 5:46 AM on October 6, 2020 [10 favorites]
who does this? who puts their corporate source code out in public just for the fuck of it? what kind of god damn bozo idiot walks around with no pants on just for jollies and ??? apparently ??? gets paid for it? but it's not just their own pants, it's their CEO's pants, except the ceo doesn't know they aren't wearing any pants???? i do not understand at alllllll
posted by seanmpuckett at 6:24 AM on October 6, 2020 [6 favorites]
posted by seanmpuckett at 6:24 AM on October 6, 2020 [6 favorites]
As a non-programmer, I would like to register my deep disappointment that the "secrets" here are, like, security keys and such, and not confessions of misdeeds, secret love, and longing for the innocence of childhood.
posted by Adridne at 6:29 AM on October 6, 2020 [44 favorites]
posted by Adridne at 6:29 AM on October 6, 2020 [44 favorites]
Anyone old who doesn't slip a hidden CREATED BY WARREN ROBINETT into their project has no soul.
posted by delfin at 6:37 AM on October 6, 2020 [2 favorites]
posted by delfin at 6:37 AM on October 6, 2020 [2 favorites]
I can see that effectively hard-coding passwords in distributable source code is incredibly stupid, but what would be the best practice here?
posted by Joe in Australia at 6:59 AM on October 6, 2020 [2 favorites]
posted by Joe in Australia at 6:59 AM on October 6, 2020 [2 favorites]
I've never understood people who use github public repositories for ANY kind of private code, at all. If it's an open source project, sure, why not. Anything else, why?
This is why I host my own git repos.
posted by signal at 7:00 AM on October 6, 2020 [5 favorites]
This is why I host my own git repos.
posted by signal at 7:00 AM on October 6, 2020 [5 favorites]
Joe in Australia: see step 8 in my ancient guide to making a Twitter bot: "So, how can your script use these secret strings (to authenticate and authorize you) without those supersensitive nuclear launch codes falling into the wrong hands, viz., your GitHub repo? I did it this way...." which provides a way that makes use of .gitignore.
posted by brainwane at 7:02 AM on October 6, 2020 [2 favorites]
posted by brainwane at 7:02 AM on October 6, 2020 [2 favorites]
As a non-programmer, I would like to register my deep disappointment that the "secrets" here are, like, security keys and such, and not confessions of misdeeds, secret love, and longing for the innocence of childhood.
"For security purposes, your new password must contain a mix of..."
posted by mhoye at 7:15 AM on October 6, 2020 [7 favorites]
"For security purposes, your new password must contain a mix of..."
posted by mhoye at 7:15 AM on October 6, 2020 [7 favorites]
> As a non-programmer, I would like to register my deep disappointment that the "secrets" here are, like, security keys and such, and not confessions of misdeeds, secret love, and longing for the innocence of childhood.
A project in 2020 using Prototype and MooTools to build an interactive UI, with Rails backend, tabled layout and tiled GIF for icons, is all of these things.
posted by ardgedee at 7:26 AM on October 6, 2020 [8 favorites]
A project in 2020 using Prototype and MooTools to build an interactive UI, with Rails backend, tabled layout and tiled GIF for icons, is all of these things.
posted by ardgedee at 7:26 AM on October 6, 2020 [8 favorites]
who does this? who puts their corporate source code out in public just for the fuck of it?
My experience its:
a) Generally people who have solved a problem, and have something to contribute, but haven't been given the tools or training to think through where the leakage may occur (again people may remove the obvious stuff like passwords or access keys, but not think about IP addresses, or database schemas in SQL queries etc.)
b) Systems staff who are being asked to release code because the company wants to attract "the best and brightest" talent and wants to showoff what an amazing engineering team they have (typically companies with external technology blogs who love to brand their external Github and show how they are solving the world's most difficult problems in emerging technologies X, Y, and Z so come work for us!!!)
c) Employees who jump between companies frequently and use GitHub as their public resume (and even if they sanitize parts of the code well, you can often compare against their LinkedIn profile and find out which companies the code relates to so removing the company identifiers but keeping all the internal IP addressing data in scripts is ineffective as an example )
d) External vendors using Github - for some reason mobile app developers are the worst. They often drop the entire source for mobile apps into GitHub - often if they are piloting apps we'll find them there. I suspect because a lot of those developers are really using a bunch of contractors for some snippets from all around the world, and having a public repository just works.
And then sometimes its the intern you hired two years ago for a summer, and you gave a *really* hard problem to go solve for two months ("ehhh...go use machine learning or AI or something to mine our logs for fraud or something...come back in three months")......and everyone forgot about the intern and the project...except the intern - who not only solved the problem but wrote up the most well documented code you've ever seen with all sorts of diagrams of how the super secret application and its infrastructure for the company works. And then posted it to GitHub so their lecturers/professors can see the code / so they can get that first job out of school and show the world they interned for XYZ Megacorp. And because they were never a long term employee they didn't go through all the training/weren't monitored closely/never made it to a Threat Intel watch list of employees/didn't even know how use the corporate protected repositories....
posted by inflatablekiwi at 7:30 AM on October 6, 2020 [13 favorites]
My experience its:
a) Generally people who have solved a problem, and have something to contribute, but haven't been given the tools or training to think through where the leakage may occur (again people may remove the obvious stuff like passwords or access keys, but not think about IP addresses, or database schemas in SQL queries etc.)
b) Systems staff who are being asked to release code because the company wants to attract "the best and brightest" talent and wants to showoff what an amazing engineering team they have (typically companies with external technology blogs who love to brand their external Github and show how they are solving the world's most difficult problems in emerging technologies X, Y, and Z so come work for us!!!)
c) Employees who jump between companies frequently and use GitHub as their public resume (and even if they sanitize parts of the code well, you can often compare against their LinkedIn profile and find out which companies the code relates to so removing the company identifiers but keeping all the internal IP addressing data in scripts is ineffective as an example )
d) External vendors using Github - for some reason mobile app developers are the worst. They often drop the entire source for mobile apps into GitHub - often if they are piloting apps we'll find them there. I suspect because a lot of those developers are really using a bunch of contractors for some snippets from all around the world, and having a public repository just works.
And then sometimes its the intern you hired two years ago for a summer, and you gave a *really* hard problem to go solve for two months ("ehhh...go use machine learning or AI or something to mine our logs for fraud or something...come back in three months")......and everyone forgot about the intern and the project...except the intern - who not only solved the problem but wrote up the most well documented code you've ever seen with all sorts of diagrams of how the super secret application and its infrastructure for the company works. And then posted it to GitHub so their lecturers/professors can see the code / so they can get that first job out of school and show the world they interned for XYZ Megacorp. And because they were never a long term employee they didn't go through all the training/weren't monitored closely/never made it to a Threat Intel watch list of employees/didn't even know how use the corporate protected repositories....
posted by inflatablekiwi at 7:30 AM on October 6, 2020 [13 favorites]
As a non-programmer, I would like to register my deep disappointment that the "secrets" here are, like, security keys and such, and not confessions of misdeeds, secret love, and longing for the innocence of childhood.
</nerrrrrrrd>
posted by suetanvil at 7:32 AM on October 6, 2020 [6 favorites]
$ echo "secret_password=\"$(echo "I still yearn for the days of my childhood, when everything was simpler." | shasum | sed -e 's/ .*//')\""
secret_password="e6bf932bc35074c094d20773dd14065f0858436b"
</nerrrrrrrd>
posted by suetanvil at 7:32 AM on October 6, 2020 [6 favorites]
what would be the best practice here?
The simplest (and most common) way to solve this is to have your secrets loaded up on the servers as environment variables, which are then injected into the app upon startup.
posted by sideshow at 7:57 AM on October 6, 2020 [5 favorites]
The simplest (and most common) way to solve this is to have your secrets loaded up on the servers as environment variables, which are then injected into the app upon startup.
posted by sideshow at 7:57 AM on October 6, 2020 [5 favorites]
And this is why I run my own private git server.
posted by grumpybear69 at 8:01 AM on October 6, 2020 [2 favorites]
posted by grumpybear69 at 8:01 AM on October 6, 2020 [2 favorites]
for clarification, not all GitHub repos are public. they can be private.
posted by j_curiouser at 8:17 AM on October 6, 2020 [1 favorite]
posted by j_curiouser at 8:17 AM on October 6, 2020 [1 favorite]
Scanning your code for vulnerabilities really isn't the job of the version control system itself.
Not as yet, but perhaps we should expand our imaginations and make it so. Why isn’t there a flag for git init that automatically configures a bunch of hooks for safety? You wouldn’t ride in a car without a seatbelt, so let’s make a seatbelt for software development.
Also, GitHub now (thank heaven) has unlimited private repos at the free tier, so you should have very little incentive to put yourself at risk of making a very bad uh-oh with your secrets.
posted by Going To Maine at 8:20 AM on October 6, 2020 [2 favorites]
Not as yet, but perhaps we should expand our imaginations and make it so. Why isn’t there a flag for git init that automatically configures a bunch of hooks for safety? You wouldn’t ride in a car without a seatbelt, so let’s make a seatbelt for software development.
Also, GitHub now (thank heaven) has unlimited private repos at the free tier, so you should have very little incentive to put yourself at risk of making a very bad uh-oh with your secrets.
posted by Going To Maine at 8:20 AM on October 6, 2020 [2 favorites]
Sorry not sorry: between this and the River Styx story a couple of posts down, I’ve had “secret secret! I’ve got a secret” stuck in my head whenever I open this thread and now you will too.
(Thank you very much-o)
posted by Huffy Puffy at 8:26 AM on October 6, 2020 [4 favorites]
(Thank you very much-o)
posted by Huffy Puffy at 8:26 AM on October 6, 2020 [4 favorites]
Why isn’t there a flag for git init that automatically configures a bunch of hooks for safety?
Because what is and is not a secret or otherwise sensitive datapoint is not obvious or standardized across applications, and trying to automate that kind of scan will potentially cause more problems than it solves?
posted by grumpybear69 at 8:42 AM on October 6, 2020 [1 favorite]
Because what is and is not a secret or otherwise sensitive datapoint is not obvious or standardized across applications, and trying to automate that kind of scan will potentially cause more problems than it solves?
posted by grumpybear69 at 8:42 AM on October 6, 2020 [1 favorite]
Pretty sure AWS creds look the same almost everywhere, though. The perfect shouldn’t be the enemy of the good.
posted by Going To Maine at 9:02 AM on October 6, 2020
posted by Going To Maine at 9:02 AM on October 6, 2020
I do have git checkin secrets all the time. Using a complex and secure git checkin process multiple programming team (internal or external) check into the right branch. A devops team reviews branches and when they're approved another and merged another git repository managed by a completely different team comes in with the secrets to access a Docker repository and in the end checkin a Docker repository that pushes to the code to production. There's a couple of key things here:
- Let us say a secret is saved in the code. Team Social Media wanted to create a TikTok enhancement and someone put a secret to TikTok's API in there. For the purposes of this lets say git pre-commit hooks and all that's above didn't know to look for TikTok's secret through entropy or any other programmatic means. Our systems will still fail because it would deny any external IP/domain requests further up stream. So not only do we have pre-commit checks we also prevent it at the network level from even accessing the API.
- Let us say TikTok's API is approved now. No team doing code ever has access to the repositories that have the secrets. At this point since we know the API we can now use entropy or regex to scan all commits in history for any mention of TikTok's secret. We an also monitor the authentication handshake between TikTok and our code (there's actually ANOTHER layer here where a team creates another authentication middleware so that our main project's code does not ever directly contact an external API. We only do the handshake through another, internal process).
- Regarding Powershell scripts and network IP addresses. Those are never known to anyone working on the main project, internal or external. If they need a Powershell script, another team gives the requirements to the programming team to make it work. So you might say "run all workstations and update the Wallpaper" which I doubt you'd get past the "run all workstations" step but lets play this as an example. Our developers don't know the internal IP addresses as they're on their own network. They're given fake IP addresses. They can spin up virtual workstations and test the script. They give the script to the devops team who approves it. The DevOps team or a completely different team most likely takes the script and runs it. Ideally whoever runs it and whoever has access to the IP addresses don't know each other. The key being that someone who runs the script who knows the IP addresses can't accidentally commit code because they don't have access to the repository and vice versa.
- Even data for the front-end app is sanitized. It is a project management application and has government secrets, not just github secrets. The developers all have FBI background checks, they still don't know the secret names or the names of people involved. They are given what's known as statistically relevant pieces of text to program off of. What this means is they never know what the code name of the government project is. They're given "xserjeksla sdfd" which is the same amount of characters and spaces, basically filler Lorem Ipsum that matches statistically the relevant text. Simply if I have a "title field" I'm trying to align and I don't know the "title field's" name is NSA Spies On You Today how am I to align it? I'm given a string that's "Plants Gummies Toads and Frogs."
What's the end result? The developers across multiple teams, some internal and some external, don't have to have crazy laptops with soldered USBs. They can be contracted out, albeit to American firms with security clearance. They can have other things on their laptop. No one person knows enough or has enough to commit to production or even commit anything secret in production. It is designed so that that every step is assumed to be compromised and that a human element, or multiple human elements would need to be working in an adversarial manner to break it.
What are the drawbacks? There are some significant ones that most organizations simply don't have the resources to enforce this. You can have a lead programmer also wear a devops hat and also wear a security hat but inevitably they're human and will take shortcuts or mess up. You absolutely needs really good requirements if you're going to give developers "Toads Jump on Gummie Bears" to program off of. This means they're not going to catch problems with poorly written requirements. It also takes agency away from developers who want to be part of making the product. You can't help make the product if you don't know why you're making it. In reality assume they're working on a hanger for a stealth aircraft. They might not know what the stealth aircraft is, or how it works but they have the dimensions and enough to know how to build the hanger.
I have taken these steps in other projects. From the beginning even for small projects I'll store all secrets in another repository. I break all things from local to production into different repositories. My code doesn't "know" how it gets to production. It can build and run locally, that's it. This isn't that hard and frankly enforces good separation of concern and code practices. Before I make anything public I scan the code and kill the ".git" file which should be clean anyway. There's no chances I'm not consciously committing comments accidentally in public.
Really it comes down to anything else in life. How often do people accidentally put their ATM pin number in a tweet? Never.
Obviously my above example is heavily sanitized but it shows you how good, solid practices can be had with just three separate teams. I've been on projects where they don't trust the people and this fundamentally trusts the people but not the process. There's a subtle difference. I don't like a corporate policy checking if I have Notepad++ installed or having my USB sticks soldered. You're asking me to cheat the system. Use the silly monitoring to create a secure product lifecycle that empowers developers.
posted by geoff. at 9:05 AM on October 6, 2020 [6 favorites]
- Let us say a secret is saved in the code. Team Social Media wanted to create a TikTok enhancement and someone put a secret to TikTok's API in there. For the purposes of this lets say git pre-commit hooks and all that's above didn't know to look for TikTok's secret through entropy or any other programmatic means. Our systems will still fail because it would deny any external IP/domain requests further up stream. So not only do we have pre-commit checks we also prevent it at the network level from even accessing the API.
- Let us say TikTok's API is approved now. No team doing code ever has access to the repositories that have the secrets. At this point since we know the API we can now use entropy or regex to scan all commits in history for any mention of TikTok's secret. We an also monitor the authentication handshake between TikTok and our code (there's actually ANOTHER layer here where a team creates another authentication middleware so that our main project's code does not ever directly contact an external API. We only do the handshake through another, internal process).
- Regarding Powershell scripts and network IP addresses. Those are never known to anyone working on the main project, internal or external. If they need a Powershell script, another team gives the requirements to the programming team to make it work. So you might say "run all workstations and update the Wallpaper" which I doubt you'd get past the "run all workstations" step but lets play this as an example. Our developers don't know the internal IP addresses as they're on their own network. They're given fake IP addresses. They can spin up virtual workstations and test the script. They give the script to the devops team who approves it. The DevOps team or a completely different team most likely takes the script and runs it. Ideally whoever runs it and whoever has access to the IP addresses don't know each other. The key being that someone who runs the script who knows the IP addresses can't accidentally commit code because they don't have access to the repository and vice versa.
- Even data for the front-end app is sanitized. It is a project management application and has government secrets, not just github secrets. The developers all have FBI background checks, they still don't know the secret names or the names of people involved. They are given what's known as statistically relevant pieces of text to program off of. What this means is they never know what the code name of the government project is. They're given "xserjeksla sdfd" which is the same amount of characters and spaces, basically filler Lorem Ipsum that matches statistically the relevant text. Simply if I have a "title field" I'm trying to align and I don't know the "title field's" name is NSA Spies On You Today how am I to align it? I'm given a string that's "Plants Gummies Toads and Frogs."
What's the end result? The developers across multiple teams, some internal and some external, don't have to have crazy laptops with soldered USBs. They can be contracted out, albeit to American firms with security clearance. They can have other things on their laptop. No one person knows enough or has enough to commit to production or even commit anything secret in production. It is designed so that that every step is assumed to be compromised and that a human element, or multiple human elements would need to be working in an adversarial manner to break it.
What are the drawbacks? There are some significant ones that most organizations simply don't have the resources to enforce this. You can have a lead programmer also wear a devops hat and also wear a security hat but inevitably they're human and will take shortcuts or mess up. You absolutely needs really good requirements if you're going to give developers "Toads Jump on Gummie Bears" to program off of. This means they're not going to catch problems with poorly written requirements. It also takes agency away from developers who want to be part of making the product. You can't help make the product if you don't know why you're making it. In reality assume they're working on a hanger for a stealth aircraft. They might not know what the stealth aircraft is, or how it works but they have the dimensions and enough to know how to build the hanger.
I have taken these steps in other projects. From the beginning even for small projects I'll store all secrets in another repository. I break all things from local to production into different repositories. My code doesn't "know" how it gets to production. It can build and run locally, that's it. This isn't that hard and frankly enforces good separation of concern and code practices. Before I make anything public I scan the code and kill the ".git" file which should be clean anyway. There's no chances I'm not consciously committing comments accidentally in public.
Really it comes down to anything else in life. How often do people accidentally put their ATM pin number in a tweet? Never.
Obviously my above example is heavily sanitized but it shows you how good, solid practices can be had with just three separate teams. I've been on projects where they don't trust the people and this fundamentally trusts the people but not the process. There's a subtle difference. I don't like a corporate policy checking if I have Notepad++ installed or having my USB sticks soldered. You're asking me to cheat the system. Use the silly monitoring to create a secure product lifecycle that empowers developers.
posted by geoff. at 9:05 AM on October 6, 2020 [6 favorites]
Not as yet, but perhaps we should expand our imaginations and make it so. Why isn’t there a flag for git init that automatically configures a bunch of hooks for safety? You wouldn’t ride in a car without a seatbelt, so let’s make a seatbelt for software development.
Except there's never a reason to not wear a safety belt. There's all kinds of reasons that you don't want VCS to do extra things. Go to Perforce and see how it spins git's features as negatives it solves. "You can delete history, anyone can! Use Perforce!"
posted by geoff. at 9:08 AM on October 6, 2020
Except there's never a reason to not wear a safety belt. There's all kinds of reasons that you don't want VCS to do extra things. Go to Perforce and see how it spins git's features as negatives it solves. "You can delete history, anyone can! Use Perforce!"
posted by geoff. at 9:08 AM on October 6, 2020
who does this? who puts their corporate source code out in public just for the fuck of it? what kind of god damn bozo idiot walks around with no pants on just for jollies and ??? apparently ??? gets paid for it? but it's not just their own pants, it's their CEO's pants, except the ceo doesn't know they aren't wearing any pants???? i do not understand at alllllll
As an IT manager, former sysadmin and former programmer for my sins, if you did a venn diagram of "people who understand security" and "programmers" there would be a distressingly small overlap. It's not any better for most other groups of IT pros to be honest.
Then you add on layers of clueless management, who don't know what they want, but they want it now so they can put a nice new bullet point in the next shiny sales marketing, so corners get masively cut, and 'getting it done' massively outweighs doing it properly, securely, or in any other way than the fastest slapdash gonna-break-later-and-be-a-bastard-to-fix way.
All so other clueless managers will buy it because it sounds great, doesn't it, let's buy it, and only tell our own sysadmin after we've already paid for it and then it's his fault that it doesn't do what we need, or tie into any of our current stuff, and is basically as secure as putting it in the staff fridge with a 'plz don't steal, kthxbye' label. #notbitter
AND programming often seems to be mostly re-using other stuff in ways that was never intended, glueing it haphazardly together and calling it a day, as per the reasons above, e.g. see how 11 lines of code broke the internet.
Tl;DR - xkcd
posted by Absolutely No You-Know-What at 9:09 AM on October 6, 2020 [4 favorites]
As an IT manager, former sysadmin and former programmer for my sins, if you did a venn diagram of "people who understand security" and "programmers" there would be a distressingly small overlap. It's not any better for most other groups of IT pros to be honest.
Then you add on layers of clueless management, who don't know what they want, but they want it now so they can put a nice new bullet point in the next shiny sales marketing, so corners get masively cut, and 'getting it done' massively outweighs doing it properly, securely, or in any other way than the fastest slapdash gonna-break-later-and-be-a-bastard-to-fix way.
All so other clueless managers will buy it because it sounds great, doesn't it, let's buy it, and only tell our own sysadmin after we've already paid for it and then it's his fault that it doesn't do what we need, or tie into any of our current stuff, and is basically as secure as putting it in the staff fridge with a 'plz don't steal, kthxbye' label. #notbitter
AND programming often seems to be mostly re-using other stuff in ways that was never intended, glueing it haphazardly together and calling it a day, as per the reasons above, e.g. see how 11 lines of code broke the internet.
Tl;DR - xkcd
posted by Absolutely No You-Know-What at 9:09 AM on October 6, 2020 [4 favorites]
Or, really, look at shhgit itself. (I shouldn’t miss the content of my own post!)
posted by Going To Maine at 9:26 AM on October 6, 2020
posted by Going To Maine at 9:26 AM on October 6, 2020
Really it comes down to anything else in life. How often do people accidentally put their ATM pin number in a tweet? Never.
I think you would lose this bet.
posted by Going To Maine at 9:28 AM on October 6, 2020
I think you would lose this bet.
posted by Going To Maine at 9:28 AM on October 6, 2020
who does this? who puts their corporate source code out in public just for the fuck of it?
To add an (e) to inflatablekiwi's list: because GPL/LGPL required it.
(Although more often it seems to me open-source compliance is by a "toss an outdated tarfile up on a dark BEWARE OF THE LEOPARD corner of your website and forget about it" and I wonder how many secrets accidentally leak into those?)
posted by We had a deal, Kyle at 10:00 AM on October 6, 2020 [2 favorites]
To add an (e) to inflatablekiwi's list: because GPL/LGPL required it.
(Although more often it seems to me open-source compliance is by a "toss an outdated tarfile up on a dark BEWARE OF THE LEOPARD corner of your website and forget about it" and I wonder how many secrets accidentally leak into those?)
posted by We had a deal, Kyle at 10:00 AM on October 6, 2020 [2 favorites]
> Why isn’t there a flag for git init that automatically configures a bunch of hooks for safety?
posted by ardgedee at 10:21 AM on October 6, 2020 [2 favorites]
$ sed -i "src/secretkeys.txt" .gitignoreposted by ardgedee at 10:21 AM on October 6, 2020 [2 favorites]
To add an (e) to inflatablekiwi's list: because GPL/LGPL required it.
Per chance you don't work for a medical device company who has to do this because their med device software is based on open source projects and they are required to. What's that. Oh no reason for asking. Nothing to see here.
posted by inflatablekiwi at 10:40 AM on October 6, 2020 [2 favorites]
Per chance you don't work for a medical device company who has to do this because their med device software is based on open source projects and they are required to. What's that. Oh no reason for asking. Nothing to see here.
posted by inflatablekiwi at 10:40 AM on October 6, 2020 [2 favorites]
Interesting that sshgit flags shell configuration files e.g. .bashrc, .zshrc, .cshrc.
My .bashrc is in git but I don't think there's anything particularly secret about it.
posted by Klipspringer at 10:51 AM on October 6, 2020
My .bashrc is in git but I don't think there's anything particularly secret about it.
posted by Klipspringer at 10:51 AM on October 6, 2020
I was curious about that too, I keep my shell config files in a git repository just so I'm portable there's nothing in there I could think that would be secret or even useful unless you find my Powerline theme useful.
posted by geoff. at 11:31 AM on October 6, 2020
posted by geoff. at 11:31 AM on October 6, 2020
"for some reason mobile app developers are the worst"
Mobile app development is a cut-throat, low-margin business where you also constantly have to stay on top of the latest thing. Successful firms are only a step away from two students in a garage. Good dev practice is not something you have time for in that world.
posted by i_am_joe's_spleen at 11:56 AM on October 6, 2020 [5 favorites]
Mobile app development is a cut-throat, low-margin business where you also constantly have to stay on top of the latest thing. Successful firms are only a step away from two students in a garage. Good dev practice is not something you have time for in that world.
posted by i_am_joe's_spleen at 11:56 AM on October 6, 2020 [5 favorites]
searches
filename:.bashrc passwordin Github and scans through a few results........
export system_name_redacted_USERNAME="posted by inflatablekiwi at 11:59 AM on October 6, 2020 [1 favorite]name redacted" export system_Name_redacted_PASSWORD="password redacted"
A lot of modern programming techniques are akin to needing to light cigarette and deciding to acquire a hummer to do it, hey it has a lighter! Who cares if it comes with a whole truck you don't need. Of course you'll give the key of the hummer to everybody, you need to unlock the door to access the lighter.
posted by WaterAndPixels at 11:59 AM on October 6, 2020 [10 favorites]
posted by WaterAndPixels at 11:59 AM on October 6, 2020 [10 favorites]
Also searching for metafilter on GitHub is probably a metatalk all of itself. People reporting bugs about the site when viewed with all sorts of weird plugins, people requesting Metafilter be added to bypass projects going around the great Firewall of China (right on!), people doing weird things with netcat against Metafilter and having their commandline history files on Github.
In short, Github is one of the strangest sites I've seen in some time. I have no idea how these people got their netcat requests wedged into their port scanners, or why.
posted by inflatablekiwi at 12:31 PM on October 6, 2020 [6 favorites]
In short, Github is one of the strangest sites I've seen in some time. I have no idea how these people got their netcat requests wedged into their port scanners, or why.
posted by inflatablekiwi at 12:31 PM on October 6, 2020 [6 favorites]
A lot of modern programming techniques are akin to needing to light cigarette and deciding to acquire a hummer to do it, hey it has a lighter! Who cares if it comes with a whole truck you don't need. Of course you'll give the key of the hummer to everybody, you need to unlock the door to access the lighter.
xcd - Containers
posted by Going To Maine at 12:34 PM on October 6, 2020 [7 favorites]
xcd - Containers
posted by Going To Maine at 12:34 PM on October 6, 2020 [7 favorites]
Pretty sure AWS creds look the same almost everywhere, though. The perfect shouldn’t be the enemy of the good.
You are more than welcome to solve this problem for everybody and watch the Brinks trucks full of cash back up to your front door, if it's so easy.
posted by sideshow at 3:07 PM on October 6, 2020
You are more than welcome to solve this problem for everybody and watch the Brinks trucks full of cash back up to your front door, if it's so easy.
posted by sideshow at 3:07 PM on October 6, 2020
I manage dev tools for a large corporation and this is one reason we run our own source control servers. Also as a former developer, it’s absolutely true that most devs know very little about security. The ones who do tend to be the Swiss Army knife rockstars, but that’s a small minority. Turns out that just working with product management to understand business requirement , and then coding those thing to work in a dev environment where scaling and security are not important, are more than a full time job. I’m all for “DevOps Mindset” but you’ve really got to have some people whose only job is thinking security.
But yes, commit hooks that look for secrets and downstream tools that check your work help a lot. There are also tools like Hashicorp Vault (not cheap for the fancy version) that allow you to access credentials as needed for builds and things without needing to know what they are.
posted by freecellwizard at 3:18 PM on October 6, 2020 [3 favorites]
But yes, commit hooks that look for secrets and downstream tools that check your work help a lot. There are also tools like Hashicorp Vault (not cheap for the fancy version) that allow you to access credentials as needed for builds and things without needing to know what they are.
posted by freecellwizard at 3:18 PM on October 6, 2020 [3 favorites]
Pretty sure AWS creds look the same almost everywhere, though. The perfect shouldn’t be the enemy of the good.
You are more than welcome to solve this problem for everybody and watch the Brinks trucks full of cash back up to your front door, if it's so easy.
I mean, they are apparently distinctive enough in folks’ code that something like shhgit works okay and that github feels comfortable scanning for them as well.
posted by Going To Maine at 4:03 PM on October 6, 2020 [2 favorites]
You are more than welcome to solve this problem for everybody and watch the Brinks trucks full of cash back up to your front door, if it's so easy.
I mean, they are apparently distinctive enough in folks’ code that something like shhgit works okay and that github feels comfortable scanning for them as well.
posted by Going To Maine at 4:03 PM on October 6, 2020 [2 favorites]
As a software person, I find this thread pretty embarrassing in terms of how it reflects on our craft, with so many people who would otherwise find appeals to personal responsibility as an organizing principle to be risible suddenly talking up bootstraps. It's a discipline in which you can literally realize vast constructs of the mind with some keystrokes and we're reflexively deeming guardrails for secrets management impossible? It is, for sure, hard to imagine that such a thing might be possible when Worse is Better and Everything is (impoverished) Text, but maybe POSIX compatibility isn't the best we can hope for as a default idiom for computing. I'm sure there'll be plenty of appeals to pragmatism and the indomitability of the established order in response to this, and just as with the luminiferous aether I suspect that the phenomenon's supposed ubiquity has no bearing on its validity.
posted by invitapriore at 8:02 PM on October 6, 2020 [3 favorites]
posted by invitapriore at 8:02 PM on October 6, 2020 [3 favorites]
As a software person, I find this thread pretty embarrassing in terms of how it reflects on our craft, with so many people who would otherwise find appeals to personal responsibility as an organizing principle to be risible suddenly talking up bootstraps. It's a discipline in which you can literally realize vast constructs of the mind with some keystrokes and we're reflexively deeming guardrails for secrets management impossible? .
Didn't this thread start out with tools that help with this? I feel like the arguments are pretty much just about how much it's worth having github try to check by default?
posted by atoxyl at 9:06 PM on October 6, 2020
Didn't this thread start out with tools that help with this? I feel like the arguments are pretty much just about how much it's worth having github try to check by default?
posted by atoxyl at 9:06 PM on October 6, 2020
... it’s absolutely true that most devs know very little about security. The ones who do tend to be the Swiss Army knife rockstars, but that’s a small minority.
It's really automation that's to blame. If you want something to work by its self, there has to be a unencrypted secret somewhere. You can't have a machine come up after a power glitch and require some human to come around and type in some secret phrase to make things work again. That eventually boils down to somewhere on that machine is a file with some secrets that's only protected by the OS and permissions.
In the network infrastructure world you have a second restriction. You can't have a machine rely on full working network services to function, they have to be as stand-alone as you can make them. All circular dependency like.
Development wise I usually followed a patter and had ~/.git_template to match. A directory corpus for ginormous files that had no reason to be in Git, a directory private where secrets went. And then mostly your bog standard *NIX set of directories. The config in etc would reference private which would also have a 'real' config sitting around so that I could use a flag or environment variable to pick the real config.
I'd disapprove of trying to make Git/GitHub try to detect fuckups on General Principals but I'm a grumpy old man who would watch you put your finger in a light socket for the "becha won't do that again" moment.
posted by zengargoyle at 1:28 PM on October 7, 2020 [1 favorite]
It's really automation that's to blame. If you want something to work by its self, there has to be a unencrypted secret somewhere. You can't have a machine come up after a power glitch and require some human to come around and type in some secret phrase to make things work again. That eventually boils down to somewhere on that machine is a file with some secrets that's only protected by the OS and permissions.
In the network infrastructure world you have a second restriction. You can't have a machine rely on full working network services to function, they have to be as stand-alone as you can make them. All circular dependency like.
Development wise I usually followed a patter and had ~/.git_template to match. A directory corpus for ginormous files that had no reason to be in Git, a directory private where secrets went. And then mostly your bog standard *NIX set of directories. The config in etc would reference private which would also have a 'real' config sitting around so that I could use a flag or environment variable to pick the real config.
I'd disapprove of trying to make Git/GitHub try to detect fuckups on General Principals but I'm a grumpy old man who would watch you put your finger in a light socket for the "becha won't do that again" moment.
posted by zengargoyle at 1:28 PM on October 7, 2020 [1 favorite]
I’d disapprove of trying to make Git/GitHub try to detect fuckups on General Principals but
In fairness, git is a free tool from a famously pro-"just-do-one-thing" community of nerds, but Github is something that enterprises both pay money for and that has some serious competitors.
posted by Going To Maine at 2:54 PM on October 7, 2020
In fairness, git is a free tool from a famously pro-"just-do-one-thing" community of nerds, but Github is something that enterprises both pay money for and that has some serious competitors.
posted by Going To Maine at 2:54 PM on October 7, 2020
« Older Penguin of the year 2020: vote early, vote often | Praise Song for the Kitchen Ghosts Newer »
This thread has been archived and is closed to new comments
It was a project for automated payments and had a bunch of keys just lying around.
Worrying!
posted by Just this guy, y'know at 2:34 AM on October 6, 2020 [1 favorite]