Some of the problems that I help security leaders with are the problems of scale and organization. I was watching an episode of the excellent PBS Spacetime on youtube and in a video about the weirdness of the Higgs Boson, there was an even weirder statement that brings everything about scale and organization into focus. Here I’ll paraphrase the video about how it would be weird if there were no layers of complexity between the atoms and a whole whale, as if the atoms that made up the whale could just organize themselves into a fully functional whale. That’s just as weird as expecting hundreds of thousands of people just knowing what to do and self-organizing into a fully functional business.

Most of us got into security from the bottom and worked our ways up - doing security testing or design, working as an individual contributor, and essentially working at the micro-scale in security. Find a vulnerability, fix a vulnerability. As our careers progress, we’re often asked to tackle larger and larger problems, but one obstacle everyone encounters along that path is that our toolboxes tend to break down when problems and tasks get much larger than the micro-scale. Analyzing an application’s architecture for threats is a different skill from analyzing an organization’s SDLC for potential security touch-points. Thankfully, one of the most powerful tools in my operational and strategic toolbox is something that is very familiar to many professionals and developers at the micro-scale of security: The Pattern.

Humans are pattern seeking creatures and we’re lucky that patterns show up so often in the world for us to find. I’m not here to argue that there is or isn’t some intelligent creator behind these patterns in nature, but just that there are ways of organizing building blocks into structures that just seem to work and we humans are pretty good at recognizing them. One pattern that we’ve noticed is a sort of meta pattern called Methodological Reductionism - the theory that if you understand what’s going on with all the component parts of an object, then you can understand the object itself. Sometimes it’s not just objects that are subject to this understanding, but layers and scales that build upon each other as you zoom in or out.

Swimming back to that hypothetical whale, instead of a whole mass of self-directed atoms going out and eating krill, whales do have organizational layers of a sort. If we reduce a whale into its major systems, we can understand how the brain and nervos system directs the muscular system, which is supported by the skeletal system, to put the krill into the digestive system which will break food down into nutrients to be transported by the circulatory system, and so on. Understand the major systems and you might have an idea of how whales do whale things, but what if you want to understand it’s major systems a little better? Break the respiratory system into organs like lungs, windpipes, and blow holes and study the tissues, cells, and proteins until you understand how breathing helps a whale do whale things. That’s reductionism in play - understand the components to better model and understand the whole.

Like whales or particle physics models, companies also have layers. The major systems of a company can be called Business Units or Divisions, they might be divided by industry vertical, geographical region, or functional role. In those large divisions, there are departments, teams, and individual contributors. While I gain an understanding of a security program, I have to understand where it fits in the whole in order to understand how the component parts work together. An organ cut off from the circulatory system cannot operate, a muscle not attached to the skeletal system cannot move it’s limb. A security team without resources cannot function and without policy, cannot leverage authority to affect change. That’s one of the powers of patterns - gap assessment. In a gap assessment, one can find missing inputs, dependencies, internal processes, and outputs for a program and recommend a way of closing the gap.

What if a gap assessment isn’t enough? What if you’re looking to secure one development style over another? Patterns don’t just show up in whales and security programs, but another icon of the fashion industry is the sewing pattern. Sewing patterns are 2-dimensional shapes that are transferred to 2-dimensional fabric that can then be cut and sewn into a 3-dimensional garment. This simplification allows us to create 3d sculpted objects, like a shirt, using 2d drafting tools like pencils, rulers, and paper. Another powerful technique is pattern alteration. If your shirt doesn’t fit because it’s too tight across the back, you can use a pair of scissors to cut the pattern piece apart, spread the pieces out, and fill in the gap with more paper in a way that won’t distort the final 3d shape beyond the intended alteration.

Much like clothing alterations, it’s possible to use standard methods to adjust a training pattern or a secure scanning pattern from a centralized, waterfall approach to a decentralized DevSecOps approach. Carefully separate the policy requirements and objectives from the resources and authority, invest the resources and responsibility into the development teams, provide them the re-crafted policy statements, and fill in the expansion with training, enablement, and cooperation.

Patterns exist everywhere in organizations, if you want to pull the pattern out of a regulatory compliance framework and adapt it for SSDF, find the essential parts and re-arrange them to work where you are. If you want to deploy a decentralized training capability, study the decentralized pipeline roll-out and follow that path. If your vulnerability management process isn’t working as well as your vulnerability discovery process, check the pattern and see what gaps may exist.

When shifting between layers, it’s important to remember another line from that Spacetime video - “Small Generates Large and Large Stabilizes Small.” The whale ingests krill atoms, and as those krill make their way through the systems and get broken down into smaller and smaller parts, they get turned into nutrients and eventually placed into their component proteins and cells. Those atoms, cells, and proteins cluster together into tissues, organs and systems, and make up the whale. In an organization, the higher provides goals, resources, and direction to the lower levels, and the lower levels turn those inputs into filled requirements, work, and value that makes the business work. Diagnosing the lack of an input to a lower level often times means analyzing the operational or strategic priorities and fixing problems at that level - You can’t expect a champions program to thrive without leadership support, and leadership can’t execute security requirements without developers and security professionals on the ground, doing the work.

And you thought patterns were only at the software architecture level. The ability to recognize that patterns exist, what their load-bearing parts are, and reshape them to work for new companies, teams, roles, or domains is vital and will only grow in importance as your responsibilities expand through your career.

There’s always more words to spend on a topic like this one, but I’ve hit my budget for now. Stay secure and never forget the humans.

As parents, getting little boys to pick up their toys is a never ending chore that starts with nagging, escalates to yelling or grounding, and usually ends in defeat as parents have to clean up the mess. Everything changed when we started running our dynamic duo of robotic vacuums every day though. After the first or second run where we demonstrated Steve's insatiable appetite for Lego blocks, toys, and dirty clothes, the boys got the hint. Keeping the floor clean wasn't something they had to do to keep us happy: any mess they left meant that they would lose what they cared about to an uncaring and unyielding automatic overlord.

Steve the Roomba and Bob the Roomba have been taking turns as family member of the week and monthly MVP for a while now. As soon as we started up the automated cleaning cycle, we'd loudly announce "STEVE TIME!" and the boys would frantically clear their floor of all toys, dirty laundry, and trash in the amount of time it'd take us to complete a single round of nagging before. Don't get us wrong, there was also a lot of initial clean-up work to get their room into a state they could keep it clean - mostly tossing old baby toys, installing organizational shelves, and removing the sheer volume of junk that life tends to accumulate. However, once we helped them get to a good place and had some help from Steve the Roomba and Bob the Roomba, they became active players in keeping the stuff they cared about out of the their robotic overlords' way. And their rooms have been cleaner now than they have been in years.

So what does this little parable of robots and refuse have to teach security professionals? It's not our job to be the the bad guys, we can outsource that to automatic, uncaring, faceless processes. Our job is to help engineering, ops, and other shops to keep the stuff they care about squared away. When I design governance suites, I am very deliberate about not having the security team be the enforcement point for software security. Instead, I position the security team as a tutor that helps a student study for an exam, a parent helping their kids get the bedroom to a manageable state, and as a partner who helps the dev team attain their objectives.

When on-boarding applications into a security program, it's best to treat bringing them into the risk management program like any other managerial orientation. Start by communicating the expectations - namely that there are policy and governance standards and requirements that are signed by the CISO, CIO, or CEO. After that, enable them to meet the expectations with tooling, training, and processes they can adopt. Hold them accountable to meeting those standards and help them make corrections when they fall short. At that point, it’s time to bring in automated policy enforcement.

Steve is our home-bound version of the automated policy hammer that keeps everyone putting effort into maintaining good behavior. Much like an automated or policy driven release gate that acts independently of the security team, Steve the Roomba is the Bringer of Consequences. Such gates and automation are allowed to be the "bad guy" when parents and security teams should focus on enabling developers to beat the requirements. This allows security teams to be the trusted partners and advisers who are helping teams get their software out the door.

Just like we can configure our automated vacuum cleaner to skip rooms or avoid certain areas for a day or two, when setting up release gates and enforcement points, it’s important to have pressure release valves such as exceptions, waivers, and bundling of results into tickets. Such policy vehicles allow teams to address technical debt in a sane manner, meet technical and architectural requirements, and deal with the surge of new findings that come with deploying any new tool while not being able to ignore or band-aid away risk.

There's always more words to spend on a topic like this one, but I've hit my budget for now. Stay secure, and never forget the humans.

As an avid user and contributor to a couple of frameworks, I had to learn a thing or two about models. There are many kinds of frameworks, controls, standards, regulations, and yes, models, but I'll stick to my wheelhouse of Maturity Models, Capability Models, and Capability Maturity Models for this post. It's important to understand what each one is, what it can be used for, and how they differ because picking out the right model to guide your security program is a huge first step in bringing formality and structure to a wild west or an ad-hocracy.

No matter what model or framework you choose, there's a couple of calibration steps that you should consider before picking The One. Here are a couple of things that you should judge frameworks on (a sort of framework for frameworks, or models for models):

• Resolution - When looking at who will use a framework, there are so many steps in resolution. Like a 2000x vs a 20x microscope lens, the resolution determines how much detail they're able to measure and provide guidance on. The SLSA framework is a great framework for DevOps tool admins looking to secure a build pipeline, but is too big for a developer to write code against, and is too small for a VP of Application Security to build a security program around. NIST's SSDF and the BSIMM are meant for CISOs and AppSec Directors to be able to survey their kingdoms and some of the neighboring lands, but aren't the best fit for a security architect or pen tester. Pick the right resolution framework for what you need to measure.

• Scope - Resolution impacts scope. A more zoomed in resolution will necessitate a smaller scope. In order to make the framework understandable, manageable, and usable by a single individual or team, framework authors have to limit the amount of content they include in their frameworks. This means there is a trade-off between scope and resolution where the more detail you get, the less ground that detail covers. While broader frameworks provide less detail but cover more ground. Pick a framework that provides the right amount of scope to cover all the things you care about.

• Role or intended Audience - Like scope can impact who will benefit from the results of a framework (Senior Leaders vs Managers vs Practitioners), the framework's target audience or roles can change with what is included in the scope of the framework. There are a pair of NIST frameworks, the CSF and SSDF, that have roughly similar scope areas and resolutions (i.e. they cover the same amount of "stuff" and have the same level of detail), but are meant for different roles and cover different grounds.

This factors may impact your decision when picking a framework and may explain why a DevOps team may adopt SLSA while the product security team or larger development organization may adopt SSDF as their security framework. The differences in scope and resolution mean they're useful for different purposes.

When looking at models and frameworks, the other factor is understanding how they measure what they measure. There are two main approaches and they're not incompatible, in fact they're often combined into one. These three types are capability models, maturity models, and capability maturity models,

A capability model works by measuring "what" or "how much" in terms of providing a map's worth of ground to cover and assigning activities for each little plot that it measures. When using capability models, it's useful for gap assessments because missing activities are missing gaps. Each activity has a minimum bar for maturity, effectiveness, coverage, or depth that has to be crossed before credit can be given, but beyond those bars, no judgments around maturity are provided by the framework. Examples of capability models are BSIMM and SSDF.

The way people "read" capability model results is often "more ground covered = more better" where the goal is to tick all of the boxes and cover all of the ground. I read capability models as a menu of possibilities. If there are gaps, it represents plots of land that could be tilled and farmed for productive use, but not all land needs to be developed. Security-focused capability models present a potential universe of things a reasonable security program could adopt, but don't set a real target for progress.

The counterpart for a capability model is a Maturity Model. These types of models aim to measure "how well" and utilize criteria that measures a capability's or activity's depth, effectiveness, coverage, drawbacks, and/or optimization. A maturity model has the bonus of being able to provide a real report card about how existing capabilities are performing while also providing a target for how to improve. Examples of maturity models are the Process Maturity Framework and Capability Maturity Model (CMM, not CMMI which is a different sort).

A drawback is that how in a capability model, each capability or activity requires a tuned threshold to cross before being mature enough to give credit, a maturity model has to be tuned and calibrated to what it measures. This means that when measuring one thing, each level has to be set to a certain listing of criteria, and measuring another capbility means doubling that work.

To save end-users the heartache of having calibrate maturity levels for every capability, there are combined capability maturity models that measure the maturity of individual capabilities. One example of this is the SLSA framework that assigns maturity levels based on what capabilities are observed. This means that the SLSA framework can be used not only to provide a menu of future expansions, but it can also be used to assign a grade in a report card. Unfortunately because combined capability models are a lot of work to tune and calibrate, they're kind of rare and capability models that are usable as controls listings are far more common.

So now that you know how to pick out a framework that's right for you, go out and measure what you've built already, use it to formulate a target state, and then connect the dots to get from here to there.

There's always more words to spend on a topic like this one, but I've hit my budget for now. Stay secure, and never forget the humans.

Step 0 - Know what you need to do and what you need to do it. There's a reason they promote senior developers to development managers - they know how to do the job they're expecting the folks they manage to do. This means that a manager can help guide more junior folks along the path to success, and get them the resources they need to execute.

Subscribe now

Step 1 - Set Expectations. When setting out a task, it's tempting to just point the managee at the finish line and say "go." It's easy, it's fast, and it doesn't take a lot of thought or time. In fact, many seasoned employees who are manager candidates are at the point where all they need is a direction and a starting pistol shot. However, many more junior employees are not telepathic. When setting expectations:

• Start by explaining what and why of the task. The why is important because it's easy to take for granted, but without understanding why things are done a certain way, that certain way may not be followed.

• Explain requirements around time and milestones to prevent schedule slip.

• Wrap it up by explaining what "good" is. Teachers had scoring rubrics, developers have acceptance criteria.

• Explain the yardstick that you'll be measuring against and let them borrow it from time to time.

Step 2 - Enable your follower. There's entire textbooks and courses about enabling your follower to do the best job that they can, but that's your job as a manager. If you aren't working to set your employees up for success, you are failing. This doesn't mean doing their jobs for them or micromanaging, but it means making sure they have the requisite knowledge, run-books, tooling, and encouragement to get the job done. Here's a wonderful resource about how willingness to accomplish a task can actually decrease with competence and as employees progress and gain more skills, they may need more encouragement to complete new tasks. https://fastercapital.com/topics/the-four-levels-of-follower-maturity.html

As a quick recap:

1. Enthusiastic Beginner - someone who is excited about getting started and learning. They need lots of guidance but are pretty enthusiastic about getting started. Be a coach here, light motivation but lots of training.

2. Disillusioned Learner - After wrapping their arms around what they don’t know, finishing their journey might seem intimidating. They might be reluctant to start or finish tasks that they should know how to do. This is when a cheerleader is needed. Help them master their basic skills and build confidence in their abilities.

3. Capable but Cautious Contributor - Here, someone has pushed past the beginner phase and is skilled enough to accomplish tasks unsupervised, but they don’t. Often times they feel like they need permission to make a risky decision. This level follower needs a Mentor, someone who can walk them through thought processes, decisions, and pump them up to accomplish the task.

4. Self-reliant achiever - This is the highest level follower. Someone that you can point at the finish line and say “go,” and they’ll achieve it. This isn’t the permanent resting place of every employee though, as they move into new areas (like management), they may revert back to earlier stages of learning and confidence and need a change in management style to match.

Step 3 - Measure and Provide Feedback. The worst feedback a manager can recieve is "I had no idea I wasn't doing a good job." It's important to provide constant, honest feedback about how the employee is progressing in the task. Giving quick bits of feedback beyond "this is great" is vital for making sure that requirements are being met and that "done" meets the definition of done. When giving feedback, there's a couple of different approaches and these vary based on the task at hand. There's modeling where you fix a mistake "like this" which may work for simple fixes that are difficult to describe. There's also reminding them of the rubric that they need to meet and expecting them to fix it. Just don't give a single fix at a time, get all the feedback ready and then provide it back in a private, 1 on 1 session.

Step 4 - Profit! Now you know what to do if you're suddenly expected to manage a direct report. I also find that this technique is hugely helpful when training security champions as they looks to security experts as a manager of sorts when learning how to do their champion roles.

There's always more words to spend on a topic like this one, but I've hit my budget for now. Stay secure, and never forget the humans.

So how do you know who to turn to or who to trust as humans lose pace with the sheer volume of output created by generative AI? Even better, as a security professional, how do you maintain trust in the software your company develops when AI helps develop that too? AI and its applications are rapidly transforming how work gets done, how we think about interaction, and how we trust what we read and see in the digital world. Kyle Hill put out a compelling youtube video which goes over how and why generative AI is eroding trust even further on the internet and is definitely worth a watch -

This is a topic I’ve been thinking over for the past few years and here is my 100% human generated, non-AI created To Do list for trusting in the era of AI. As a security professional, we have been used to withholding trust for years and don't spend it easily. Many of those lessons should apply with slight updates:

Trust Humans (a little) - We humans are in this together and are the same batch of schmoes that you've always known. To really establish trust that you are dealing with a human and not an AI, show up and meet the other person. Until you've met someone in person and exchanged social media accounts, you don't really have a good way to trust that they're not being impersonated. Right now, there isn't a really good way to establish trust between real life and the internet, but that's because there hasn't been a pressing, universal need to. Until we figure out a drivers license or ID card-like scheme for the internet, the best guidance we have is to meet people in real life, or rely on people you've met in real life, to establish trust in the human behind the social media account.

Trust Tech (2FA++) - Scripts, bots, and impersonators have been around for a long time. This is kind of a technical version of validating someone's social media accounts by meeting them in person, but instead of requiring an introduction that doesn’t happen inside the internet, these methods will involve using channels that AI doesn’t have easy access to.

Get your Ducks in a Row - Sometimes, we want to trust AI to do something for us. Sometimes we've found an AI that we can trust to be an AI and want it to do something useful for us. One main limitation in AI models is the number of tokens it has to "think" about a given task. If you overwhelm an AI, maybe by giving it too much context, or too long of instructions, it will run out of tokens and the conversation will end or go off the rails. Since most people use text inputs for AI, a token might be a whole word or a part of a word, but essentially if you talk too much to a ChatGPT, it will get overwhelmed and start making mistakes. AI Co-pilot code writing software currently performs best when asked to complete lines, not programs. When instructing AI to help out, don't give it all the ducks to look after, limit your scope to a single discrete duckling sized task.

Count the Fingers (Weirdness Checks) - A common problem that people noticed right away was that when AI created images of hands, it couldn't really get them right. We all learned that if there was a photo of someone with a weird hand with extra fingers, it was AI generated (or an artist's satire of AI). That's an example of a weirdness check that most of us are familiar with by now, but as AI tooling becomes ubiquitous, get used to the weirdness checks and either use them to weed out AI generated content (e.g. by looking for "As an AI model" in reviews to find automatically generated product reviews), or adapt to the weirdness when using AI for productive uses.

Vendor & Open Source Software (OSS) Risk Management - If you're like me, you don't have a pet AI model that you've built from scratch and trained yourself. This means that all your AI needs will be met by "Somebody Else's Software" or hosted on "Somebody Else's Computer." If you're consuming AI, treat them as a vendor and do all the risk management things. If you're building AI, vet the OSS libraries and training data by looking for contributor and repository characteristic risk measures. If you're concerned about your vendors using AI insecurely, count the fingers whenever they're handing stuff over.

Secure the (rest of the) Iceberg - That old security metaphor is still floating around because it's just so dang useful. If you’re new to the game and haven't had the pleasure of encountering the security iceberg, the basic gist of it is that the amount of ice that's above water represents 10% of what you can see, but it was the hidden 90% below the water that sunk the titanic. AI presents new risk, new attack vectors, new attack surfaces, and new threats to establishing trust, but all of the old stuff is there too. If you haven't cleaned house and have the old-school security habits down pat, focusing on AI originated risk may feel like a bit of a misdirection.

AI is coming. It's too alluring to ignore and the promises it whispers are too irresistible. Don't trust all of it, but don't abandon all hope either. Keep your ears to the ground and put a little trust in humans.

There's always more words to spend on a topic like this one, but I've hit my budget for now. Stay secure, and never forget the humans.

So, let’s get started by talking about what I mean when I say “flail.” As I think about it, flail is the human and organizational stress response to failing processes and workflows that are being overwhelmed or damaged by new changes. Flail at the human level looks and feels like an environment everyone is locked into perpetual firefighting behavior. It can lead to stress, burnout, and lost productivity when important items are de-prioritized as overwhelming urgent business-as-usual tasks pile up. At the organizational level, it feels like disorder, inefficiencies, loss of collaboration, and general chaos. In the long run, it leads to talent loss, inability to innovate, and a loss of maturity in core capabilities as turnover bleeds competency from the talent pool.

When solving flail, the first step is realizing that to the people on the ground, flail looks and feels a lot like solving problems. In fact, that’s all employees in a flail infected organization will see - a constant and unending stream of problems that need to be solved in order to complete previously uncomplicated tasks. That’s because the root cause of flail isn’t in their hands. There’s no way for a good employee to “good employee” their way out of organizational flail. When I evaluate clients, I look for a predisposition to flail and the signs I look for are a history of reacting poorly to change, silos, teams that are universally at capacity, and an operating strategy that is slanted towards fire fighting rather than investing in fire-prevention.

So this all sounds like a management problem and not a security one, right? Why am I talking about it here on Secure Humans? Well, that’s because a poorly implemented security program can induce flail when it adds unneeded complexity to previously simple development tasks as security is integrated into the SDLC. When designing plans, I have a couple of preemptive flail prevention solutions that I weave into my plans when I see a client that’s predisposed to, or currently undergoing, flail.

After diagnosing flail or a predisposition to it, my first concern is identifying a palpable, clear problem statement and pairing it with a strategic vision that addresses the problem. Often times, our human instinct is to minimize problems, let them go unsaid, or lead with a solution. However, not everyone has that vital context and when big changes happen, they appear to come from nowhere, or are couched in neutered words that lack any sort of problem worth solving. This means that strategic changes can seem ego driven, or the result of a busy-body looking to make a mark while in office, as opposed to an actual solution to a real problem facing the organization.

The next step involves assessing how middle-management gets their work done. Flail resistant organizations empower their management at every level to identify solvable problems at their level and solve them, and communicate upwards when unsolvable problems happen at their level. How well people managers are able to manage is a huge part in how resilient an organization is against the chaos that comes with strategic shifts.

Since managers are often the referee for work coming into and exiting their shops, they’re also sitting at a key position to minimize flail. When taken to an extreme, strong boundaries to inter-departmental collaboration present as silos, but when completely torn down, those boundaries’ absence lead to an unmanageably large, un-prioritized to-do list for every person in that shop. Instead, I look at workflows that route between shops and departments in treat those managers as stakeholders in the changes that will impact them. They already know how they want work to show up for their shop to work on, and respecting those existing methods is key to minimizing disruption, friction, opportunity cost, and flail that comes with new security asks.

After designing process changes that minimize introduced new chaos, it’s time to acknowledge that stress, friction, and chaos may already be present. This can come in many forms. When planning, I like to build in technical debt friendly risk-exception processes that can keep teams from being overwhelmed by old problems as well as adopting a crawl-walk-run approach to capability maturity. At a management level, delegating work to other teams or just refusing low-priority work can help bleed off some stress for individual teams but isn’t a permanent solution. At the human level, encouraging socialization, hobbies, and other stress relief to keep from being overwhelmed is a huge help that many of us overlook when overworked.

Finally, when building a plan to make strategic changes, try to keep change to a minimum. In any organization, even flailing ones, there’s a lot of processes, workflow, tooling, and shops that work well. The silent 90% tends to get drowned out by the 10% of flailing processes that do need to be addressed. It’s important to understand the totality of all that in order to set keep what’s working intact, and only make the lightest of changes that address the clear problem and meet the all important strategic vision. This also means not applying re-organization after org change after shake-up. It’s often tempting to try to bring out the re-org toolbox and make reactive changes, but that almost always makes things worse while the organization is healing.

When preventing or managing flail, the role of leadership is to set the vision, empower managers to manage their responsibilities, encourage healthy collaboration between business units, departments, and shops, and allow for some stress relief when dealing with the consequences of strategic change. The role of managers is to change the problems that their work-centers, shops, and departments have to solve. When accounting for flail, be deliberate in only mandating changes that solve real problems, and allow problem solvers to solve those problems.

There's always more words to spend on a topic like this one, but I've hit my budget for now. Stay secure, and never forget the humans.

Unfortunately, there isn’t a perfect tool that can be bought and deployed to solve these problems. Getting a handle on API security requires a realization that API-driven cloud solutions are different from other types of software, a product-owner commitment to get a handle on the unseen vulnerabilities built into most naively-developed APIs, and some heavy lifting from development, testing, and security to tackle existing tech debt and quit adding new APIs to that pile.

While the best time to fix issues that appeared in the past is back in the past, all we have is the present. Many security professionals are either firefighters or show up to work every day in front of a whack-a-mole machine because dealing with issues of the past is a full-time job for the present. Instead, the path to a manageable present should be rooted in the future. While we all love to procrastinate, that’s what got us into our current messes, so instead, we have to start with a strategic vision for a safe and manageable future.

Getting out of the purely reactive API security stance is first a matter of putting out the fires before there are sparks. What I refer to as “rogue APIs” happens when developers create API interfaces and endpoints without getting documentation, security, and testing requirements from the security team before deploying their API-driven software. These are the APIs that are most likely to be playing with matches and will pop up during network scans and bug bounty requests in best cases, and incident reports in worst cases.

Dealing with rogue APIs is a two-part exercise - Find and Get Well. The finding part can be difficult as there are often specialized tooling packages that have to be installed, configured, and operated to fingerprint every host in the environment. If the host responds to HTTP or HTTPS calls, it might be an API server. Additional fingerprinting on HTTP hosts is required to verify that it offers an API interface and from there, the hunt for an owner is often performed cooperatively with IT or Ops.

After rogue APIs are found, they have to take their insecure, missing, or incomplete secure development and API documentation practices and “Get Well”. The goal of a get-well plan is to allow API service owners to get in line with the larger API security strategy; this may involve scheduling testing, code reviews, inventory documentation, and the not insignificant effort associated with properly documenting API endpoints. If that wasn’t enough, the devs may also need to work with security to provide test data for those endpoints to facilitate proper penetration testing.

After finding all of the rogue APIs in your environment, it’s time to build a vision for the future where all APIs in this Security Utopia (Greek for “no place” so let’s take that as encouraging) are properly documented, tested, configured with security features and libraries to handle vital functions such as authentication and authorization, and provided with actionable API-specific non-functional requirements. This is essential to getting developers from applying API-naive security practices to API-specific ones to their software.

Those documentation, testing, and security requirements, features, and API-specific NFRs are how security teams make security happen for their API development teams. Where do all those great things come from? Threat Modeling! At least the requirements and suggestions for security features do. By running a threat modeling exercise on a couple of representative APIs in your portfolio, you’ll have a pretty good idea of how to approach API security in general. A threat model works by breaking the design into its component parts, tracing attack paths from threats to assets, and listing security controls that could break those paths. After collecting controls from threat models of a few crown jewels, those can be added to a library of knowledge that is then shared with other teams who can re-use those features and patterns to secure their own APIs.

Unfortunately, I haven’t really found a good tool to solve everything API, so this is a job for Humans. Maybe you’ll need some extra help from humans who have already tackled API security. Bringing in external expertise is going to do way more than trying to crawl wikis or frantically copy notes from the OWASP web page. While that expert is there, they might also have time to grab some coffee for you. It’s a big job so put a fresh pot on.

There's always more words to spend on a topic like this one, but I've hit my budget for now. Stay secure, and never forget the humans.

During our careers, we all start out at the bottom - spending time in an entry level position where we learn how to be an employee, a practitioner, and a team member. Over time, as we accumulate skills, knowledge, and experience we’ll shift into a manager, an enabler, and a mentor who helps others with what we gained during the early years of our careers. How you share your knowledge, insights, and leadership will eventually become your brand. Whether you consciously cultivate your brand or not, you will eventually get one and being proactive is the best way to build a good one.

The moving parts of a personal brand start with your distinct skills and offerings and are what you’re building during your time in those entry level trenches. Are you building a reputation as an expert penetration tester or as someone who can fix problems? Are you a leader or an innovator? What do people think of when your name comes up? That’s the earliest parts of your brand. From there, begin to bring your own personal style into your branding. Do you have a logo, icon, or color scheme that you want to associate with your messaging, or are you looking to cultivate style through your tone and voice? What personal interests or hobbies do you want to fold in to your brand? For example, Joyce Vance’s newsletter is about her chickens and politics. Once you’ve decided how you want to represent yourself, it’s time to start making your mark.

After you’ve built your expertise and specialization, start cultivating your internal brand. This is how you’re thought of inside the business and can be useful for career growth, getting assignments you want to work on, or finding other people who share common interests. The most personal way to grow internal branding is through mentorship and thought leadership where you can guide more junior team members through informal opportunities, or deliver talks through more formal webinars. Often times, presenting can be daunting and one of the best way to start strengthening that skillset is by being a co-presenter or co-author. Other opportunities to build an internal brand can be knowledge sharing through internal training sessions or by participating in committees that provide strategic guidance to the larger business.

Subscribe now

The next step is taking it on the road - External Branding. This is what we come across most often on social media, at conferences, and in the industry at large. A good starting point when planning external branding efforts is figuring out what and how you’re most comfortable sharing your expertise with the larger industry. Not everyone can overcome stage fright and deliver talks, while others really enjoy writing their ideas and sharing longer form texts on blogs or shorter discussions on social media. Newsletter platforms make it easier than ever to reach a personal audience (If you haven’t noticed, I’m partial to Substack). Be professional, insightful, and consistently engaged and after a while, you’ll build a network of peers.

Multiplayer Branding is an option and most people get there through networking in professional groups or communities of interest. These groups will tend to center around individual topics or areas, will cater to professionals in an industry, or do both such as OWASP. Networking has a variety of benefits but can also just be fun if you enjoy talking shop with other people. After a while, you might start getting invitations to work in communities of interest or working groups and can start shaping the larger industry through building and sharing best practices, research, or whitepapers.

Don’t forget to build strong relationships with clients as well. Eventually, after consistently meeting or exceeding expectations, reliably meeting deadlines, and delivering a high quality of work, clients can start to see you as a trusted advisor. If you like the client, that can be a good thing but sometimes may need to be handled with care if not.

After absorbing and contributing to the body of industry knowledge, you may wish to start researching and innovating. Every industry needs people to tread new ground, expand the body of knowledge, and drive the continued evolution - why not you? Start digging into interesting topics that there aren’t good answers for and offer up your own. It could be through informal discussions, empiric experimentation that produces hard data and results, or data driven analysis on existing or new data. Most problems are too big for one person to solve on their own and research often requires collaboration with peers - formally in working groups or informally until the problem is solved.

Finally, continue to grow and adapt. Get feedback from trusted sources. Everyone on the internet has an opinion, but for your own sanity, try to solicit opinions from those you trust and want to learn from. Incorporate that feedback without changing the core of yourself and your brand. Feedback may come in the form of opportunities to explore and grow, or conversations about elements that might not be working. Also, try to keep the core of your brand intact as you move into new areas otherwise you might end up like those who are having to reinvent themselves after investing too much of themselves into bitcoin and other blockchain technologies prior to the crash. If you have a strong enough sense of self, that will keep you flexible without the need for complete re-invention.

Anyway, I hope that this inspired you to start giving back to your peers and the larger industry while getting a little bit of something intangible in return. I wrote this because it’s an article I wish I had read 5-10 years ago as I was growing as a professional. If you’d like to share this with someone you mentor, I’d really appreciate it.

Share Secure Humans

There's always more words to spend on a topic like this one, but I've hit my budget for now. Stay secure, and never forget the humans.

Before I dive in, I'd like to thank you all for sticking around (or at least not unsubscribing) during my absence while I worked to get BSIMM14 out the door this year. I have thoughts about frameworks that I'd like to share here soon, but this isn't that post.

Here at Secure Humans, I've been very mindful not to talk about how exciting AI can be, in favor of focusing on the human element in security. However, AI is coming to help humans be more secure if they're already doing that, or it's coming to help companies be more insecure if they pull secure humans out of the mix. Unsupervised AI use is insecure AI use. I tell people that the next wave of corporate breach headlines, in the vein of unsecured AWS storage buckets, will be from companies using AI-powered tooling and taking the results as-is.

However, AI is coming whether we security professionals want it to or not. The marketing jargon will be too strong, the tools too shiny, and the promise of automated development and security too alluring. The major players have already started unveiling ML models that developers can integrate into their applications, and tool developers have had a full year to start taking advantage.

Subscribe now

AI and ASPM

AI might begin by writing subpar code and making questionable decisions, speeding up processes but not necessarily improving them. This raises the question: are you prepared to swiftly evaluate the effectiveness of these tools? Do you have the necessary visibility and control over your data and tools from a single interface, or does each new tool add complexity to your system? Can you pull all the levers and read the right dials from one place, or does each new tool mean another UI, another data source, and another dashboard to forget about?

That's where ASPM comes into play. Back when humans did all the work, centralizing control often meant using a single interface - Outlook's meeting invite. Now, automation-infused SDLCs mean that such gathering of functionality and metrics into one place needs more automation. You're probably already doing security posture management - gathering an inventory of assets to secure, doing risk analysis, making decisions about security, and reporting on all that - ASPM is all about doing that in a new way.

When I talk about metrics, I often talk about the need for a "Single Source of Truth" (SSoT from here), one spot that all security telemetry is loaded into and then displayed, which lets security teams know what's really happening. ASPM is taking that and turning an SSoT metrics dashboard into an orchestration suite that handles inventory, risk policy, and decision enforcement.

Some companies have attempted to move into the ASPM space by partnering to close gaps in that capability list, but while doing old things is good, this approach misses the spirit of Single Source of Truth 2.0 in favor of simply covering ground. Having the SSoT ethos in place is vital when wrangling AI-provided inputs, outputs, decisions, and policy into the SDLC, as the AI multiplies the amount of data security teams will have to consume and make decisions about.

Instead, if ASPM is on your 2024 security resolutions (do security posture management better), here are some questions you can ask while selecting or building a solution:

1. What do I care about today? Catalog the risks that are scariest and already on your radar. Review the application inventory and see if there are any patterns around the crown jewels. Patterns such as mobile or containerized architecture, cloud hosting, or microservice and API dependencies. Can your ASPM solution tell stories about those trends?

2. What data is feeding my decisions today? List out which tooling, process, and testing inputs are taken into account when allocating security budget and resources. Figure out what tools are providing that data and who is running them. The best ASPM solutions should work with existing tooling while being flexible enough to integrate new sources.

3. What policy is being enforced and when? There are various ways that bad software can be kept out of production. Understand what policy and standards are in place, when they can be applied, and who is approving their enforcement. ASPM will need to factor in cloud, IT, ops, change management, legal, and other sources to really hit that SSoT note.

4. Who needs data to make the right decision at the right time? There are more stakeholders than just developers and security. ASPM solutions will need to provide reports or alerts to other stakeholders who care about the business functions that in-scope software is supporting. Not every stakeholder cares about every metric, and they don't all need to receive it in the same way.

5. What will I care about in the future? The way things are today isn't how they'll be tomorrow. Once the basics are accomplished - find bugs in code, fix bugs in code, repeat until testers and tools are happy - security professionals should start caring about proactive measures that enable developers to develop securely. An ASPM solution shouldn't just care about finding and fixing defects in an endless loop; it should provide metadata that can feed risk decisions. It should look for the use of security features and libraries that enable developers to pull secure-by-design off the shelf. It should reward good actions taken by security champions, developers, and engineers to measure the culture that software is built and operated in.

Once all those questions are asked, an ASPM solution should be sourced that makes sense in the context of the answers.

As AI comes home, make sure that your security posture management is ready because for every problem solved by AI, another will be created. The only way to really be secure in the age of AI is through automation because for every development task that is automated, there needs to be an automated security check to make sure it's not adding risk.

There's always more words to spend on a topic like this one, but I've hit my budget for now. Stay secure, and never forget the humans.

Share Secure Humans

A textbook would say that process friction is a slowing force on a process that requires energy to overcome to maintain momentum. I’ve heard it argued that not all friction is bad. Sometimes, like when a driver applies breaks to reduce speed before a sharp turn, reducing speed before performing dangerous activities such as integrating new libraries, releasing new features, or building for new architectures can prevent incidents and issues. However, most would argue that reducing friction is a nearly universal good.

However, all of these views focus on the “slowing force” part of the definition and forget the human that has to provide energy to keep momentum. Friction pops up in more than just productivity processes. A volunteer lead organization will eventually peter out and disband without a strong leader, and most efforts’ first real existential crisis comes during a leadership change when the original person providing input energy is no longer there to turn the crank. Overcoming development delays due to friction requires developers to put in crunch time hours. Applying governance controls is often done through the power of people with clipboards. If I don’t put energy into typing this newsletter, it doesn’t get sent out.

Friction, then, is a people problem. It’s not so much always caused by people, but it’s primarily there for people to solve. Let’s look at types of friction that may be out there and how we can help apply a little bit of grease to get things moving along.

• Cognitive friction is what most developers have when faced with a security problem for the first time - They’re confused and it takes energy to iron out that confusion. This is often solved by either making the opaque transparent through communication and explanation, or building ready-made solutions that turn complex problems into easy-to-use reusable libraries.

• Collaborative friction is, as the captain from from Cool Hand Luke helpfully pointed out, “a failure to communicate”. When security team members and developers don’t work well or communicate freely, when schedules don’t match up, goals don’t align, or blamestorming prevails, collaborative friction needs soft skills to ease the tensions. Speak the development team’s language, learn their goals and incentives, and open up additional communication channels that integrate with how developers chat.

• Operational friction happens when processes, tooling, or outdated infrastructure is letting people down. I see this all the time with teams forced to use outdated SAST tooling to scan code written in modern languages and framework. Those old tools built for java web apps and c++ monoliths aren’t able to keep up with the built-in security features available in modern frameworks, or react to new threats caused by new architectures. This is where process engineering and tooling budgets are required. Of all the frictions, this is the one that can be solved by throwing money at the problem.

• Regulatory friction comes from outdated, draconian, or difficult regulations or laws that impact how software needs to be built and secured. Some of this friction comes in the form of adapting DevSecOps to work in SOX compliant organizations by redefining “Ops” to have split release/deployment processes and environments to support proper separation of duties. Other pieces of this friction may come from onerous testing requirements PCI DSS’s old requirement to pen test every release. Yet others come from FUD around regulation’s newness like we’re seeing with organizations trying to adapt to the SSDF. This is the type of friction that orgs have the least control over, but flexibility and expectations management go a long way.

• Technology friction can come in the form of outdated tools, tech debt, interoperability and compatibility issues, and good old software bugs. Solving tech friction requires creative solutioning, tech refreshes, and collaboration. Throwing money at tech friction isn’t as helpful as it appears at first blush because on-boarding to new tools may be tougher than adapting or fixing old ones.

• Security Friction. Yes, we have our own friction. Often times this comes in the form of “No, you can’t do that” or “You can’t do it that way” when the easy way is insecure. It might be easier to just echo user input back to the browser via the template engine, but a little bit of friction introduced by using a built-in encoding and validation library has a huge security payoff. However, one early security friction fail came from using the OWASP ESAPI library. Sure, it was better than nothing, but it had to be applied to every single output that would be sent back to the user, and if a single output was left unprotected, the entire application was vulnerable to cross-site scripting.

Hopefully this overview of the types of friction we deal with will help you alleviate unneeded sources of friction and inflict as little as possible when rolling out new processes, tooling, and governance.

Subscribe now

There's always more words to spend on a topic like this one, but I've hit my budget for now. Stay secure, and never forget the humans.

If you’ve spent some time in AppSec, you've undoubtedly come across questions like, “How can I get into Application Security?”, “What should I study if I want to land a job in AppSec?”, and “How can I begin learning about security vulnerabilities?”. We've all tried to navigate through these inquiries, often recommending OWASP, textbooks, or possibly online resources and mailing lists. However, let's add another valuable tool to your armory: Capture the Flag (CTF) Challenges.

A Capture the Flag event is a hacking challenge where cybersecurity professionals put their skills to the test. The end goal is to retrieve a text string that can be submitted into a central CTF dashboard for points. These points contribute to team rankings, turning the whole process into a competitive game. CTF events serve as excellent platforms for experts to hone their skills, share expertise with junior members, allow security teams to assess their skills and shortcomings, and offer an entry point for those interested in cybersecurity. If you are part of a champions program, a red team, or a penetration testing group, scheduling a CTF event as a team-building exercise can be beneficial.

Two weeks ago, our CTF team secured the 16th place among 982 teams. I’m thrilled to share this result. Even though it's not in the top 10, it's a significant improvement from our 26th place in 2022 and 85th in 2021. To halve our ranking, we had to more than double our score.

This advancement is a result of a decision we made just over a year ago to establish a dedicated team and develop a CTF capability. In this period, we’ve expanded our CTF participation from five members to over sixty and have broadened our areas of expertise to include containers, AI/ML, and the cloud. Building a CTF team is like any capability, but instead of being able to mitigate vulnerabilities in open source or enforce governance, the outcome is that people who might not specialize in security have a chance to become expert security practitioners.

The initial step was assigning a leader. The common saying "When everyone is responsible, nobody is responsible" holds true here. To avoid a lack of leadership, we appointed Aris, who already had a passion for CTF, learning, and competition. Although he had been leading an off-the-books CTF initiative, we recognized the need for a more structured approach to achieve substantial results.

We managed to secure Hack The Box CTF licenses as an initial measure. But if you’ve run an appsec program before, you know that buying a tool is radically different than using the tool. To build momentum, Aris began actively using those HTB licenses. He allocated licenses for specific learning tasks to consultants and set up a weekly learning session where those HTB challenges would be solved by a group.

A leader without a team isn’t a leader at all. To begin building the team, Aris identified experts in various categories from our existing consulting pool to assist with relevant challenges folks who dealt with AI/ML professionally were tapped to provide guidance on AI/ML challenges. Our webapp pen testers were already primed to deal with the traditional webapp CTFs. We even discovered a digital forensics enthusiast among us, although they deny their expertise if asked outright.

The knowledge we've accumulated has done more than just boost our ranking. It has served as a crucible, revealing our weaknesses and illuminating our deficiencies. Last year, I discovered a need to transition from Perl to Python for scripting and to familiarize myself with Ghidra and Volatility for forensics. The forward progress I made this year is that I only need to teach myself Ghidra. CTF participation is an effective way to keep technical skills up to date as professional responsibilities become less technical.

To sum up, the journey of building a CTF team has been illuminating and challenging for not only the participants but our entire organization. It's a perfect representation of the saying 'iron sharpens iron.' By competing against the best in the field, we've honed our skills and forged a team ready to confront the diverse challenges of cybersecurity.

Therefore, if you're seeking to break into AppSec or aiming to upskill, don't underestimate the benefits of CTF events. They offer not just a competitive and entertaining learning environment, but also a pragmatic way to understand and mitigate cyber threats. Whether you're learning solo or as part of a team, I'd encourage you to participate in a CTF event. Trust me, it's an investment of time and effort that yields substantial returns.

There's always more words to spend on a topic like this one, but I've hit my budget for now. Stay secure, and never forget the humans.

P.S. Apologies for the absence! Life has taken an exciting turn recently. The kids have been home from school for the summer and my office doesn’t have doors. We’ve also had some family events that have disrupted non-essential activities. But now that things seem to be winding down, I plan to publish these posts with greater frequency going forward.

"So, so what? / I'm still a rock star / I got my rock moves / And I don't need you" ~Pink, So What (2008)

Last week, the community that develops and maintains the open-source chess engine Stockfish witnessed an age-old dance, not between pawn and rook, but between security researchers and developers. This time-honored ritual is more formally known as the Security Defect Report.

The vulnerability discovered in Stockfish is somewhat of a classic, so much so that even the original OWASP Top 10 list included a buffer overflow on it. The issue was flagged by user ZealanL, who found that if a player inputs a position list (essentially a text-based representation of an ongoing chess game) with too many moves, the excess moves begin to overwrite data outside of their designated area, affecting other parts of the Stockfish machine state. The best-case scenario is a crash, but the worst-case scenario is being tricked into injecting code that is disguised as a chess game.

Surely, the developers saw this report, thanked ZealanL for their findings, and immediately began securing this input, right? Well, not exactly. What ensued was a back-and-forth debate, a dance all too familiar to seasoned security researchers and developers. The Captain from Cool Hand Luke might have diagnosed this as a failure to communicate, and many echoed Pink's defiant lyrics, asking, "So what?" The developers didn't need ZealanL's input and eventually, due to the escalating tension, the conversation was closed.

The "So What" does matter. The initial response was that Stockfish was free to crash on any illegal move list, such as a move-list that was impossible without breaking the rules of chess—like having too many queens on the board at once. It became clear that the developers respected the rules of chess more than they cared about the rules of software security. They felt that if a user was audacious enough to violate the rules of chess, they could deal with crashing their own instance and any subsequent consequences of such an action. Security experts countered that this was a Buffer Overflow, a major concern, but their warning fell on the developers' disinterested ears.

This cultural disconnect happens daily in development shops and security labs worldwide. Everyone has their "So Whats" that drive their daily priorities and actions. The "So What" dance begins when a security professional advises, "When you do this, don't do that," and the developer responds, "Why shouldn't I do that?" What follows is a back-and-forth where each side speaks according to their own set of priorities, yet never truly understanding the other's perspective.

One common strategy security researchers use is creating a proof-of-concept to demonstrate the severity of a vulnerability. Developers then expect such a proof with every vulnerability reported. This dance occurs where trust is lacking and vision is not shared. If security only presents restrictions, developers will leverage every ticketing system, workflow, policy clause, and chain of command to keep them at arm's length, effectively creating a silo.

This silo enables them to do what they're incentivized to do—create functionality and push new features—while minimizing obstacles. It’s a self-defense mechanism.

In future posts, I'll discuss breaking down silos, but the "quick" fix involves having leadership redefine developers' priorities to include security, or the security team fostering interpersonal relationships with the development team—not only showing up when there are problems. Outreach and threat intel briefings are beneficial, as are security expos and talks from external speakers. Security champions are another excellent method for embedding security professionals into pre-existing silos to help dismantle them.

In the end, the thread was closed by user Vondele without addressing or resolving the issue despite the dedicated efforts of ZealanL and other security researchers to demonstrate the problem, communicate the issue, and synthesize a fix. The rationale was that crashing on bad input was acceptable, and a classic OWASP Top 10 alumni vulnerability like the buffer overflow didn't pose a threat to their game of chess.

Security doesn't always win; sometimes it gets outplayed. However, that doesn't mean security has to lose in your organization. Never stop communicating, never stop advocating for prioritizing security, and never stop fighting for the fact that for your company to do what it does well, it must do it securely.

There's always more words to spend on a topic like this one, but I've hit my budget for now. Stay secure, and never forget the humans.

NIST’s Secure Software Development Framework (SSDF) has an entire section devoted to root cause analysis, vulnerability management, and hopping off the find and fix treadmill. According to NIST, the process begins before defects are discovered and instead relies on understanding what vulnerabilities could exist in the software. From there, an analysis and testing process ensures that software is free from those identified defects. The SSDF concludes by closing the loop and making changes to how software is developed in the SDLC to prevent those issues from popping up in the future.

That’s how you manage vulnerabilities. Sure, there’s all sorts of additional work involved with risk ranking, tracking, mitigating, workflows, POA&Ming, metrics, training, and enough tracking systems to make a software sales guy’s quarter, but I only have enough words in my weekly budget to talk about the work. Let’s break the work down into its component steps so that you go beyond fixing the bugs you find and instead prevent the bugs that could be.

Vulnerability management starts before vulnerabilities are identified. Threat intelligence is vital because you can’t find something you’re not looking for. Automated security scanning tools and pen tests solve this problem by outsourcing that knowledge load to the tool developers and pen testing agency, but they’re not experts in your application. All they can look for are vulnerabilities that could apply to anybody else. Instead, get smart on threats that are specific to your industry, the libraries your app is consuming, and the architecture that you’re building for.

After understanding the threats, begin doing something with that information. If there’s an existing off-the-shelf rule that checks for it in your app, great! If not, write requirements and acceptance criteria that ensure it’s not written in to the code and that it’s tested out by release. After you’ve done this once, write it down because there’s going to be a lot of this. For every plausible threat, counter it with a detection, prevention, or mitigation technique that works across the entire portfolio.

Once the detection techniques are hammered out, every time a security defect is confirmed, treat it as an escape event. If a screening technique found an issue, perform root cause analysis to understand how that issue was introduced in the first place. Match the solution to the problem. If there was a knowledge gap, fill it with training. If there was a vendor or third-party issue, make it an issue with the understanding that future defects of the same type will not be tolerated. If there is a governance issue, update some documents. If there is a tooling issue, fix the tools or fill the functionality gap.

Close the loop by going back to the threat intelligence step a couple paragraphs back and treating each root cause analysis as a source of more intelligence. Integrate it into the internal training and metrics reporting so everyone can learn from it. If it was released to production, it’s vulnerability disclosure time.

Anyway, hopping off the find and fix treadmill isn’t easy, but it’s necessary. In the coming years, it’ll be mandatory for suppliers of software to the US Government and eventually it’ll become industry best practices. Due diligence is about to get a lot more due.

There's always more words to spend on a topic like this one, but I've hit my budget for now. Stay secure, and never forget the humans.

P.S. I’m sorry this issue was delayed, but I have an excuse! Here, have pictures of the crayfish boil I put on yesterday instead of writing this.

Not everyone eats the mudbugs, so I had some gulf shrimp ready too.

I had live crayfish over-nighted in from the bayou.

Smile!

I picked up a taste for crawdads when I was in tech school on the gulf. Ever since then I’ve put on an annual boil for friends wherever I am. It’s the closest thing I have to a tradition that isn’t already on the calendar.

It’s springtime here in the south, and that means it’s bug season. Every season is bug season; it’s just a matter of which ones are in season. This year, like every year, I started by going to the outdoor section to stock up on ant granules, outdoor traps, door and window perimeter treatments, and other ways to repel insects in my yard. After that, I wandered over to the indoor section to grab the borax and sugar ant traps, spot sprays, and cleaning supplies to deal with any bugs that worked their way indoors. I was dealing with the same problem of bugs but in wildly different ways with different products and in different locations.

AppSec vs SSCRM have the same shared outcome, but different methods, locations, and tools. Just like a borax and sugar trap would catch ants out in my yard so well that it would be quickly overwhelmed or depleted, running automated security test tooling on all the code that’s out there in the open source community would quickly saturate the tooling and resources available to one company. Instead, areas of supply chain risk have different tooling and processes out there to deal with different areas and kinds of risk.

And there are different areas play in SSCRM. Each type of software and its source have unique risks, visibility problems, and control mechanisms that SSCRM teams have to account for. Let’s walk through Open Source Software, Bespoke Software, COTS software, and Partner Software to see what levers firms can pull.

Open source management has been the key focus area for many firms over the past five years. We’ve all heard the ancient rites of protection for this area: review and approve incoming open source software, store in a safe repository for internal use, scan with software composition analysis, and build a software bill of materials. That’s a good baseline, but do you want to make it better? Start vetting the open source projects. When reviewing the software, do more than look at existing vulnerabilities posted to the National Vulnerability Database (NVD) and check the license risk.

• Examine the pattern of commits, is it one overworked individual or a team of regulars?

• Are all the project sites and documentation up to date or do they redirect to parking sites?

• Are existing CVEs closed out regularly, and do they follow a repeating pattern?

• Could this package be subject to typosquatting, hostile takeover, or other library-driven malware attacks?

• Was the last update made sometime this decade?

Bespoke or third-party software is like software your firm builds, but built by someone else. There is generally a contract bidding period and request for proposals, a negotiations period, and then after work milestones are hit, acceptance, and if you’re lucky, a support period. Beyond those steps, it may not feel like there is much a firm can do to enforce security on a peer. Just like you can’t bust your way into your neighbor’s kitchen and lay down sugar and borax traps for them, you can’t show up to a contract developer’s site and start running SAST tooling. Instead, you have to communicate, observe, and test what you can.

• During the RFP phase, include vendor vetting questions: do they have a secure development lifecycle in place? What testing methodologies and secure coding standards do they adhere to? What testing artifacts do they offer up? Can they spell SBOM? What are their developers trained in?

• During the contracting phase, communicate expectations about compliance with industry best practices, or even NIST’s Secure Software Development Framework. Don’t just front load expectations onto their build and dev process, set out expectations for what acceptable looks like to security and how the vendor will respond to bugs that are discovered after delivery.

• Before accepting the code, test what you can. Run a binary scan or dynamic tests to ensure the executable deliverable runs as expected. If source code is provided, scan it.

COTS software is the toughest nut to crack. Nobody is going to get Adobe, Microsoft, or other behemoths to change the way they develop their flagship products for a single contract. Instead, dig into threat intelligence feeds that mention the products and find other firms that have already analyzed and secured installations. This is where having an active threat intel capability that participates in industry events is vital. Follow best practices, consume and emit intel, and patch whenever is most inconvenient for the bean counters in accounting so they know you’re doing something.

The newest category in this discipline is the platform partner. Cloud partners, PaaS, SaaS, Tacos-as-a-Service vendors, cloud-hosted CI/CD toolchain providers, cloud-based scanners, AI-driven tooling, are all software packages that run on Somebody Else’s Computer. The process here combines a little bit of all three of the previous areas. It takes the research and vetting required for selecting open-source packages and libraries and shifts it to a cloud install. It pulls the expectations management and SLA considerations from bespoke and third-party development and places those responsibilities on cloud or development teams to understand precisely what responsibilities are shared with “us” and shared with “them”. Finally, it takes the threat intel and industry participation requirements from COTS vendors and brings them to the cloud team because nobody is going to get AWS to radically change things for a single contract. Don’t forget the privacy concerns - any data you are providing to a partner is leaving your perimeter and you need the contract and vendor assurances it will be safe.

The lines between Application Security and SSCRM are blurring. There are new stakeholders who are responsible for reducing software risk across the whole portfolio, and it’s time to start collaborating. Any gaps in ownership or responsibility for security or risk can give attackers a foothold or allow risk to blossom and thrive.

There's always more words to spend on a topic like this one, but I've hit my budget for now. Stay secure, and never forget the humans.

... And if your boss is asking for vulnerability counts, their boss doesn't actually care about how many problems you're finding. Crafting metrics is the fine art of getting information that decision-makers need so they can make a decision. If your metrics spark more questions than insights, you're probably not reporting metrics: You're reporting data, like the raw count of problems your tools are churning out.

Good metrics make the ineffable understandable. The weakness of cybersecurity is that it’s just so darn abstract. You can’t walk up to a Mean-Time-To-Remediate and poke at it with a finger. It’s difficult to communicate abstract concepts like residual risk carried across the application portfolio, and decision-makers struggle to make the right call in the face of unknown unknowns like that. Good metrics should aim to take the abstract and ethereal goodness, badness, progress, and problems and communicate them in a way that even the most technophobic could grasp. Armed with these newfound insights, decision-makers get to make decisions.

What types of decisions should you be feeding with your metrics? Short-term risk management decisions and longer-term strategic decisions. Good metrics also have a couple of additional roles like providing a summary of how things are going or progressing and raising awareness or getting buy-in to help solve problems. When compiling data and building reports, it helps to be deliberate in asking “Why?” of every data point that’s included.

Want to start reporting better metrics? Here are the steps:

1. Decide if what’s measured actually matters to people. The most common failure mode of metrics is taking the canned reports from a tool and reporting those numbers and graphics up as if they mean something. Nine times out of ten, default metrics dashboards are built to provide information to tool operators. If someone’s not operating that tool, they don’t care what metrics get reported out. Shift to measuring and reporting information that decision-makers need to make informed decisions.

2. Build a narrative that feeds a decision. Humans are storytellers; we communicate via narratives and consequential threads. It's folly to assume that when someone is promoted to a certain level that they get read into the secret society of risk analyzers and are given the special ability to consume raw data and output human-understandable risk information. As the ground level technical expert, management is looking to you to tell them if things are bad or good, worsening or improving. In my experience, the best way to build metrics is the same way you get a toddler to go to bed. Begin with a story. "Once upon a time, there was a bar chart, and when that bar chart went up..." What decision inputs or context does this metric provide? Was it good? Was it bad? Is the bar chart going up because you're deploying tooling and getting more visibility into risk? That could be a good thing. What happens when the bar chart reaches a certain threshold? Does someone need to step in and change how things are operating? Does there need to be a freeze or a halt?

3. Collect data from the appropriate data sources. When measuring how a cross-country road trip is going, people don’t report on how the tires are wearing. While it is tangentially related data because the further a car has driven, the more wear the tires have, it’s not actually helpful when phoning ahead and letting people know if you’ll be late for dinner or not. Just because a metric exists and is ready to report doesn’t mean it’s worth reporting. Remember how the default tool metrics might only be useful to tool operators? On the flip side, the data that does matter may not be immediately available. Some metrics might require drafting new scripts that periodically poll an API and stash the data away somewhere so trend-line and time-based metrics can be reported up.

4. Math it out. How is all the data going to be combined and transformed from raw numbers to knowledge and decision inputs? Build baselines, thresholds, and targets to provide context about what’s normal and what needs intervention. When the numbers cross a threshold or stray too far from the baseline, do something about it. When targets are achieved, pop some bubbly and have a celebration. Have a guide that explains all the data sources a metric relies on, what calculations are made, what important thresholds shouldn’t be crossed, and what each metric means in case everyone forgets why a certain graph is being sent up.

5. 5. Combine the metrics into the whole picture. While simple metrics can be represented as a single number, more complex metrics may need the context built into the graph. This context can take the form of past measurements to show trend lines, thresholds, and baselines to provide a quick check on if things are going well or not, and related metrics that might have a causal relationship. Is there security training that educates developers on how to avoid SQL injection? Put that training consumption metric next to SQL Injection trends in teams that have completed the training vs. those that haven't. Different audiences need different views, and engineering might need one dashboard, developers another, and leadership a third. Take their needs and decisions into account when providing information to them.

6. Iterate. Metrics should be part of a conversation between the sender and the receiver. Over time, continually improve the metrics that are reported to better reflect the real world.

From here, whenever rolling out something new, treat it like a science experiment. Build a hypothesis, collect data, run trials, and see if the effort bears fruit. Or compile metrics that can tell a scary story to get buy-in for fixing how bad things might be.

There's always more words to spend on a topic like this one, but I've hit my budget for now. Stay secure, and never forget the humans.

The first contact developers have with application security is usually a web app or API penetration test, or at least it was when AppSec was getting started a decade or two ago. Things may have changed, but the pen test is still a rite of passage for many dev teams who wish to expose their applications to the wider world. For such an important milestone that has been achieved for years, many teams still read the results wrong.

It’s okay to use pen testing as a first round of defect discovery, but it tends to lose its usefulness after repeated uses. The find-and-fix loop created by retesting via pen tests will eventually shrink to nothing over time as the feature set and code base stabilizes and security defects are identified and eliminated. At this point, the cost per identified defect may be in the thousands of dollars when only one or two defects are reported per test.

There are cheaper ways to find vulnerabilities than pen testing. Design reviews and threat modeling can identify security problems way before there is even a single line of code to compile and run a dynamic scan against. Security policy can drive requirements for teams to pro-actively select libraries, design patterns, and security features that stop findings from manifesting in applications. When committing code to the repository, it can be checked for goodness by tools and checklists on the way in, and the whole code base can be scanned by a SAST tool when it’s pulled from the repository at build time. Automated abuse cases can exercise common attack patterns that a pen tester would exercise as part of their checklist and catch the risk before it moved to production.

Subscribe now

For a security aware dev team that is doing that and more, pen testing should be the last line of defense, not the first and only source of defects. Yet this is often not how pen testing is sold and delivered. A third-party vendor with training, tools, and experience can test an application with very little demand on the dev team. It’s far easier to run a pen test than it is to integrate and tune a SAST tool into the pipeline, or decompose the application’s planned design into its component parts and think like a hacker. This low barrier to entry often leaves pen testing as the first, last, and only line of defense in some organizations.

How do you read a pen test “the correct way” then?

It starts by treating every defect as the tip of a whole iceberg. In my experience, security defects and vulnerabilities travel in packs. Where you find a SQL injection caused by a naive assumption that special characters can’t cause the SQL query to do untoward things to the data stored in the database, you’ll find cross-site scripting caused by that same naive assumption that special characters can’t cause a victim’s browser to do untoward things to their session cookies. When you find a vulnerability hidden behind an admin login panel, you’ll often find another vulnerability exposed to internal employees based on the same naive trust that “only trusted users have access”.

Treat each finding as a bigger problem by following the iceberg below the waterline. There’s a reason the iceberg shows up in so many security services and tooling sales slides, there’s not many other good visual metaphors for “Where you see one, there’s multitudes that are hiding” like the old Titanic sinker. When you begin moving from a single reported defect to realizing that there may be entire related families of vulnerabilities that live in your code, you’re reading the pen testing results right.

How do you act on this? Root cause analysis. Identify the gaps in your tooling, training, experience, visibility, and processes that let each pen test finding live in your code as it traversed the whole software development life cycle. A defect as critical as SQLi or XSS can be stopped in many places such as security requirements and acceptance criteria that mandates input validation, output encoding, and secure libraries. They can be found by just about every decent security code scanner out there, and most dynamic scanners will look for these issues as well. SQLi and XSS cheat sheets exist for use by QA testers who can look for these issues manually, and those same cheat sheets can be turned into automated security test cases that should flunk any update that comes bundled with XSS.

Other issues may need specific prevention such as server hardening guidelines to prevent misconfiguration issues and coding guidelines that mandate proper logging in case of a forensic investigation after a security incident. Design reviews and threat modeling may be needed to highlight business logic flaws that could lead to applications being vehicles for fraud by allowing nonsensical transactions, or they may identify missing or improperly applied controls like encryption that can prevent data loss in case of hardware theft.

All told, the right way to read a pen test report after you have a few security capabilities under your belt is to evaluate how effective those security capabilities. The first question when evaluating a pen test result shouldn’t be “How do we fix this?” but instead it should be “How do we catch these earlier?”

The answer to that question may highlight a gap in the tools that you think are scanning your code or doing unit testing for you, or they may highlight a gap in your governance that needs to be updated, socialized, and trained on.

You should still fix the instance reported in the report, too. Though if that’s where you stop, you’re reading your pen test results wrong.

Subscribe now

There's always more to say on a topic like this one, but I've hit my budget for now. Stay secure and never forget the humans.

Share Secure Humans

The ingredients for a successful application security program come from many sources; the expertise and experience of the AppSec team, the cooperation of the developers, the security tooling from engineering, and the lanes of authority that come from governance.

However, the ingredient I’ve seen missing in broken AppSec program can come from a frank discussion between the CISO and the AppSec director: a shared understanding of goals. When I was a communications officer, my boss used to joke that “communicators make the worst communicators” and that’s a trend that I’ve seen extend into this side of my career.

What should be communicated between the Chief Information Security Officer and the Application Security Program Owner? Security, mostly. The CISO’s job is to deal with Information Security, as its Chief Officer. They accomplish their job of securing information by delegating bits and pieces down to Identity and Access Management teams, Audit & Compliance, Ops, Risk Management, AppSec, and others.

The CISO, in turn, gets their role delegated to them by the CEO, COO, or CIO to ensure that the CISO enables the larger business to do what it does, but securely. This is why it’s important for AppSec to be aligned with the business as a whole because good AppSec lets the business do what it does, but securely.

Alignment starts by asking the question of “Why Security?” On the surface, this exercise seems silly because we all know that security is awesome, but there are people out there who don’t think that about security. That’s kind of a good thing because we wouldn’t have jobs in security if everyone was on board with it. Understanding the reason it’s important to secure operations, assets, data, and software will give purpose and shape the larger AppSec program goals. If a bank was hacked, it wouldn’t be able to do bank things like handle money or service customers. If a health-care provider were hacked, they couldn’t provide patient care. If a power company was hacked, they couldn’t keep the lights on.

The goal of an AppSec program shouldn’t be a self-licking ice-cream cone. It’s mission statement shouldn’t be “to grow security maturity” or “to reduce vulnerabilities”, it should be “To reduce software risk that could prevent us from keeping the lights on for customers” or “To enable providers to securely treat patients”. Having a frank conversation between the CISO and the AppSec program owner should establish that as the banner goal and everything in the program builds towards that.

A clear, business-relevant goal is invaluable for an AppSec program. It sets a clear vision that can be the first step in breaking silos. A well-defined goal will inform what’s measured and reported for good and meaningful metrics. It also makes prioritizing what happens where much easier. At a person-to-person level, it changes the security conversation from “You should fix this problem because the tool said its bad” to “We should fix this problem because it’s a threat to keeping the lights on.”

Once the goal and vision are established, defining roles and responsibilities are next. The AppSec program owner's role is to plan, implement, and execute, while the CISO's role is to remove obstacles, provide resources, and secure buy-in from their peers. When I see processes that are broken because development teams or engineering teams can argue their way out, it’s not necessarily a sign of a broken process. There isn’t a process so perfect that some overworked manager can’t push back. Instead, it’s an authority issue.

The CISO needs to not only delegate authority to the AppSec program owner, but they need to ensure that their peers - the CIO, COO, CFO, and others - understand what the AppSec Program owner’s authority is. This allows the other C-Suite officers, SVPs, and other leaders to communicate that message down their lines of control. Top down communication is vital to moving security beyond friendly teams and allowing program owners to keep security tasks prioritized in the backlog.

There's always more to say on a topic like this one, but I've hit my budget for now. Stay secure and never forget the humans.

In my career, one near-universal constant is that those who have a reputation for getting things done get overwhelmed by people who want them to get things done for them. Developers are no exception, and something they've put in place between the askers and the doers are the backlog and work tracking systems like Rally and Jira. If a task isn't in a ticket, it's a favor. When the tickets pile up, favors give way to tickets.

Often, security professionals will have friendly contacts who will perform favors like setting up a tool, running a scan, or fixing an issue. Arrangements like this can work for years in small, informal shops, so many security teams never realize that other ways of getting tasks handled exist or are necessary.

Once upon a time, I was working to build an application security practice in a company that was transforming to SAFe Agile. Over the course of a couple of quarters, I noticed a pattern. If I asked for tool installations or prioritizing a bunch of security bugfixes during certain months of the year, I had a much shorter delay before the start of work. This was because my requests were being entered into the team's backlog prior to the quarterly PI Planning meetings.

For those of you not properly initiated into SAFe, the PI planning meeting is a big meeting where dev teams, infrastructure, and other support teams come together with backlogs of tasks that have dependencies on other teams. At this meeting, teams negotiate features and stories to ensure that blockers are resolved and everything is in place for successful launches.

As a security team, working in this structure was hugely helpful to us because we were able to negotiate our enabling security stories into their planned work efforts.

It's important to remember that we had strong CISO support in this organization. Without top-down support for properly and realistically prioritizing security asks, our asks would most likely have been washed out as low-priority tasks. Instead, we were able to plan our support and workload on a quarter-by-quarter schedule, and teams got used to working with us instead of for or against us.

Not every organization has a PI planning ritual, but most developers are used to pulling tasks out of the backlog. To ensure that security tasks are pulled out as often as needed, it takes a group effort between the AppSec Team, security-minded developers, and leadership to ensure that the product owner and dev-team lead know what to include and when. I have another bunch of words planned about how to engage senior leadership to get them on board with security, but that's for the future.

There are many other ways that developers and security teams can support each other and help build more secure software together, but it's vital to respect the workflow to get work done.

There's always more to say on a topic like this one, but I've hit my budget for now. Stay secure and never forget the humans.

I deal with many challenges in my day-to-day as an application security expert. Many are technical, the result of technologies not growing and evolving at the same rate until like differing metals with different rates of thermal expansion, something breaks with a pop and something that used to be whole is now in pieces. But many more are human.

Humans are, despite what you may think or feel about the whole lot, essential to this interconnected world of modern marvels that are all just a few thumb taps away on your smart phone. Understanding how they work together to accomplish the impossible is far more important than understanding how that impossible was accomplished. The means to understanding that is culture.

Culture here isn’t talking about the fuzzy bits that form in old yogurt or keeping one’s pinky up while sipping tea from bone china, but instead it is how companies get stuff done. The way I think of culture is stolen from Lance Hayden, PhD (author of People Centric Security) and boils culture down to a shared set of beliefs, incentivized behaviors, and values within a community. When a security culture aligns with what’s expected things go well, but when a culture clashes with expectations then frustration and failure abound.

There are many reasons an appsec program can fail: lack of senior leadership support, lack of funding, a tool-based silver bullet thought process, or an inability to keep up with changes in software development styles. However, the sneakiest and most frustrating failures lie in cultural disconnects.

There’s 2 main axes on the culture spectrum in this simplified model. The vertical axis reflects centralized control vs decentralized control. At the top of the axis lie organizations where there is a strong central governing authority. Often there is a single appsec shop that writes the rules and a single policy that is enforced everywhere. Very little escapes the Big Brother who lives perched atop the vertical axis in our model. At the bottom lies decentralization or loose control. These have power delegated to the teams who execute according to pressures and drivers they face. Most security professionals view the bottom half of the axis as a no-man’s land of chaos and lawlessness, but most developers thrive in an area where they are free to pursue their requirements however best they see fit. Some of the most interesting security developments come from teams with the freedom to solve the security problem as best they can, as long as they are motivated to solve security problems.

The horizontal axis reflects focus, internal vs external focus. An internally focused organization is one that that writes its own requirements and business drivers, and works to achieve them. They have a product that they trust is the best in the market and strive to achieve that vision. The externally focused teams are primarily concerned with responding to market or regulator demands. A decentralized externally-focused team is most likely the small product engineering team that make their bonuses by making the customer happy, while a centralized externally-focused team keeps everyone happy by passing external audits.

Classically, I’ve considered the military to be the archetype of the centrally-controlled internally-focused quadrant and they respect processes. If you give an aircraft maintainer a checklist that’s signed by the SecAF, they’ll execute that checklist. Their decentralized internally-focused counterpart is the hospital where highly competent doctors and nurses are trusted to provide the best care they can to patients. The centrally-controlled externally-focused quadrant is dominated by banks who have to pass audits of all kinds. The decentralized externally focused quadrant in the bottom right tend to be engineering firms with distinct product lines or value propositions that they sell based on contracts with large buyers who have exacting standards.

Understanding what culture your business has is the first step in working with it, not against it. When getting started in security, the centrally controlled organizations tend to be more compliant and security professionals have an easier time there. All someone has to do in a bank and say “It’s for the audit” and teams will jump on board. In a process culture like the military, it only takes a good process or two and they’re working on security.

When dealing with decentralized trust (internally-focused) and autonomy (externally-focused) cultures, the trick is building something really good with early adopters, and then getting buy-in on an awesome process or tool. Once stakeholders understand how this new security widget or dance helps them deliver better patient care, or better products to buyers, they’ll be all-in. The downside is that it’s a constant battle to build faith in application security.

As always, metrics are the answer to messaging, but that’s another day. Build good metrics that tell good stories, and you’ll get to do good work.

There’s always more words to spend on a topic like this one, but I’ve hit my budget for now. Stay secure and never forget the humans.

You are an application security practitioner, decision maker, or leader faced with the challenge of getting developers to build better software. There’s a lot of news happening but you’re having a hard time figuring out what news and intel is actionable and what’s fluff. Things like ransomware aren’t what keep you up at night, but the fact that Log4J introduced a whole new class of JNDI injection and nobody seemed to care does. 

I am a mid-level appsec professional who gets to spend a lot of time thinking about systemic problems that lead to software vulnerabilities. My time is split between making sense of news stories so we can learn lessons from those who didn’t learn from past mistakes, helping appsec managers build programs that let smart people do smart things, and thinking about the questions that I’m going to be asked about in a year so we have answers much sooner.

If you’re still here and reading this, then you might be interested in following along with my rants and rambles. I’m not going to commit to something weekly, but I’ll probably drop a couple hundred words whenever something particularly interesting comes along. 

Right now, if you’re interested in how AI and ML will be a blessing and a curse to software security, federal cybersecurity regulations like the SSDF and EO, and how all this affects people, then you might want to stick around and for some of my thoughts and musings.

Subscribe now