DJ Schleen

Hey everyone! It’s almost time for a #devsecops takeover on the TechWell Hub!

I just dropped the 2020 DevSecOps Community Survey yesterday - check it out, let me know what you think.

https://bit.ly/DJ2020Survey

Tom Stiehm

Thanks @DJ Schleen, not to jump the gun but I have question. What do you see as the biggest challenges to adopting security practices into DevOps?

DJ Schleen

Great question! There are a ton of challenges to gain successful adoption. It depends on which team is trying to implement the practice. Is it the development team or the security team - and that differs between companies large and small. The biggest challenge is to ensure everyone is comfortable with change and can communicate with candor. Once the culture is there (or growing), the challenge is to find the best place in an automated pipeline to put security controls without sacrificing the speed of delivery and deployment.

Technique is a big challenge as well. Has value stream mapping been done? Can you measure flow? How can you make security invisible?

Tom Stiehm

Follow on, most automation is geared toward gathering data that a person has to analyze at some point. How do you allow to analyze of findings without impacting speed of delivery and deployment? What is the best techniques you have found for doing that?

DJ Schleen

@Tom Stiehm Oh man. That sounds like somebody is going to hate their job. :slightly_smiling_face: This is where I get into my essentialist mode…

ForbesForbes

The Art Of Essentialism

When I was at SXSW I learned about a discipline from author Greg McKeown called Essentialism. Essentialism is the art of discerning between external noise and internal voice. It’s not a task and time management tactical list. It’s more than that.(4 kB)

https://thumbor.forbes.com/thumbor/fit-in/1200x0/filters%3Aformat%28jpg%29/https%3A%2F%2Fi.forbesimg.com%2Fmedia%2Famp%2Fimages%2Fforbes-logo-dark.png

A few thoughts on analysis - first begin by looking at the data collected and identify if it is truly valuable, consumable, and actionable. Repeat that often .Look for ways to consolidate noise. Ask what really needs to be accumulated and stored. Security tools are very good at finding imaginary issues (false positives), and creating a work item for EVERY issue they find. Deduplication is a must.

For data related to a detected vulnerability - consolidation of the issue with pointers to the places where the issue was identified is invaluable.

You could give that one issue to a developer to address in the entire code base, add in a technical security resource if necessary, remediate the issue, and it’s gone.

After it’s been resolved, add a policy that fails a build if it is ever introduced again and shift way left.

Tom Stiehm

@DJ Schleen Great insight, one of the issues I have found with moving to a DevSecOps mindset is the existing dev to Sec relationship is often adversarial, what good techniques have you found to move that toward a cooperative relationship? I have been on both sides of the table of a security review so I understand the perspectives involved and have had some success in bridging the gap so I would love to find out what other people have done that has been successful in order to expand my DevSecOps transformation toolkit.

DJ Schleen

Security orgs need to check their ego at the door and realize what they know and don’t know. Ask most folks in your security department to show a developer exactly why injection issues are a problem. More often than not they can’t. But….

That being said, a security org that’s been around for years is most likely just getting up to speed on application security… it’s only become an concern in the past few years. Theres more to these groups than appsec - there’s threat hunters, ethical hackers, third party risk, GRC, identity and access management, security operations… and more. Saying that “security should code” is like telling an electrician that they need to be a plumber.

I’m getting to your question…

Developers are surprisingly more interested in security than the security org thinks as well. Security needs to understand that as well.

So…

Create a security champion program. Where one or more developers from each feature team, pod, etc that are interested in security become a part of. Give them training on relevant secure coding practices for the platforms they develop on. Schedule a regular “Attack and Educate Brown Bag” where they get together with the appsec team and discuss new security issues in the world, and a demonstration on how to exploit something. Show how to get a shell on a server and everyone gets interested quickly.

Learn more than you think your developers know…

I spent most of 2017 with my security team learning everything about AWS, Kubernetes, Terraform, etc. This was so we could have conversations with our developers and architects and they realized we knew what they were talking about and could add value to the discussion - not just from a security perspective, but from an implementation perspective. Jaws drop and respect forms.

Whew… i could talk on that forever.

Tom Stiehm

You have until 1 PM and I like to hear about your experiences and opinions.

David

@DJ Schleen what do you think are some of the top security concerns for DevOps/DevSecOps during the COVID-19 crisis?

Tom Stiehm

Follow on, most automation is geared toward gathering data that a person has to analyze at some point. How do you allow to analyze of findings without impacting speed of delivery and deployment? What is the best techniques you have found for doing that?

DJ Schleen

@David My top concern would be remote work and ensuring the communication channels between home and office are secure. That’s not just for DevOps/DevSecOps but for any practice.

David

@DJ Schleen One problem I have had in terms of implementing security into DevOps is getting management to buy in. What are some ways that you were able to get managers/leaders on board with making security a bigger priority when it comes to DevOps?

DJ Schleen

That always seems like the hardest thing to do right? I’ve been told in the past that “Security wasn’t a priority” from my VP…

Great leaders usually have the following traits - they have their direct report’s best interests in mind, give time and productive feedback, establish goals for their direct reports and let them run until tackled. Oh… and delivering the goals of the business.

A few things then…

Reasonable leaders will listen and consider the ideas and concerns of their team.

They take those ideas and seek alignment with business goals.

They assess risk (timeline, security, budgetary, brand reputation)

Then come to a conclusion. At the end of it all, some businesses accept the risk of a lack of security, some are mandated to integrate it with regulatory compliance, and some do it for competitive advantage (I curated some interesting stats on that in the Community Survey)…

The selling points are: There is no impact to delivery and deployment times*. Our customers get safer software, sooner. Security is a non-functional requirement and is essentially a QA activity (but not really). Cost to maintain the security systems has been considered. Training plans have been identified. A software bill of materials can be created and used to identify all the third party components in an application (important when 97% of web applications aren’t developed by the internal teams - it’s open source code), how do we find what application is using what code in the event of a breach. Then of course if that doesn’t work just say Equifax.

David

Awesome thank you very much

owen

@DJ Schleen a non-technical question for you. I know you coach youth hockey. How’d you get into that? Who’s your all-time favorite hockey player?:ice_hockey_stick_and_puck:

DJ Schleen

How did you find that out? Dang you’ve seen my Twitter feed eh?

Well the “eh” probably gave it away - I’m a Canadian (actually Americanuck - dual citizen)

My favorite hockey team is the one and only Ottawa Senators. Note the engravings on the top of the original Stanley Cup (which is from Ottawa). This is the 1893 cup. We won it 11 times. Original dynasty. One of the original 5 NHL clubs. (the original 6 are the teams after the World Wars)

Fun fact - The Sens won the cup one year and drop kicked it into the Rideau Canal - it was pulled out by a lady who planted flowers in it. Then returned to the hockey nation.

Favorite hockey player of all time - Dominik Hasek (The Dominator). My son plays goalie in U10 (I coach) and his name is Dax. He is known on his team as “The Daxinator” #39

owen

That’s an awesome nickname! I’m not a proper hockey fan, but I used to go to Washington Capitals games and cheer for “Olie the Goalie.”

Did you play growing up?

DJ Schleen

It’s somewhat of a rite of passage in Canada lol. I sure did. We had a rink right down the street at a school (every school has an outdoor rink in Canada it seems) which I went to with my friends to play. There was an ice storm once that left an inch or more of ice on some packed snow in front of the house I was renting and my room mate and I strapped the skates on and played hockey on that too.

Played roller hockey when I moved to Colorado and when my breezers and pants ripped I went back to ice hockey. Got my coaching certifications under my belt - one more and then I pretty much can coach any level of hockey.

David Croughwell

@DJ Schleen As someone relatively new starting out in the security field and being the individual tasked with managing vulnerability and patching exceptions for both OSS and Containers where would you focus your attention? Both on the management side but also personally so that I can improve my capabilities so that when speaking to a developer I can understand their perspective in regards to the entire DevOps/DevSecOps pipeline.

DJ Schleen

With OSS (Open Source Software) you want to do three things:

#1 Proxy incoming components so you know what is coming in the door from any third party repository (Nuget, Github, Maven, etc)

#2 Integrate OSS tooling into the pipeline on check-in of code. Have it scan for vulnerable component versions and if it finds one with a binary compatible non-vulnerable version then automatically create a pull request for the developer to merge. Don’t fail, don’t send it back, don’t create an external work item. Deal with it where developers have the least amount of friction.

#3 Create policies for restrictive licenses. I prefer to only use MIT licenses. Using the wrong one can create unnecessary trademark risk, or instantly require you to open source your proprietary software

For containers - definitely do some research:

https://docker-curriculum.com/

https://docker-curriculum.com/images/logo-small.png

https://dzone.com/articles/docker-layers-explained

Then utilize scanners (they are fast just like OSS scanners) like Anchore, Clair, etc to identify vulnerabilities. You can look at Trivy as well https://github.com/aquasecurity/trivy

aquasecurity/trivy | Apr 10th, 2019 | Added by GitHub

I really dig Anchore for containers, Sysdig Falco for container hosts, and Trivy is my favorite

To answer the second part of your question, you should look at what technologies your developers are using and do some hands on tinkering to learn the concepts. Definitely learn a programming language (flow control is the same across them). I like Go - it’s terse, fast, and cross compiles to any platform.

Resources:

https://continuousdelivery.com/

David Croughwell

Thank you!

DJ Schleen

https://tour.golang.org/welcome/1

^^ quick and dirty overview and you code in browser to learn

Kelly M

Hi @DJ Schleen, just joining the convo and I am curious to know how you got into DevSecOps?

DJ Schleen

Hey Kelly! It was really serendipitous. Elevator story: I taught myself to code at 10 so I could write video games, got fascinated with how bits and bytes could fly over a telephone line so I ripped my computer and modem apart, started breaking into computer systems - and then websites, started the first web design company in Eastern Canada in the early 90's and ran the modem pool and server infrastructure at an ISP, moved to the US, learned ITIL and ALM and applied it at BMW, and eventually realized when I was developing software about 10 years ago that there was an actual name for the practice (Dev-Sec-Ops)

It was DevOps then, but with a various colored hacker hat in the past I began adding security controls in.

I also hate to do manual things so automated everything with scripts.

It swang into full gear at Aetna/CVS where I was a security architect implementing large scale security programs.

Whew… so much for quick elevator story.

Kelly M

haha! If you didn't have to type it, it wouldn't seem as long!

Gene Gotimer

Hi @DJ Schleen. Is there one particular, must-have security tool that you should have in your pipeline? Or maybe, if you have a DevOps pipeline, what is the one tool you should add to start making it a DevSecOps pipeline?

DJ Schleen

Open Source Software scanning and Container Security Analysis tools are your best bang controls to inject into an automated pipeline. They execute quickly and are very frictionless additions. OSS accounts for 85%-90% of code, and a mis-configured Static Analysis tool could scan each one of these third party code bases as well as your own code.

Once you have these two in, look at SAST (which seems backwards for a lot of folks, but then you are catching the vulnerabilities introduced by your developers.

As for DAST (Dynamic Analysis) - I have a love/hate relationship. You need it for compliance reasons, but it is a dog. Ensure you are optimizing your configuration as you could end up with a scan running for a week. Don’t integrate it into your pipeline EVER. Run it parallel.

The love part of DAST is for mobile applications.

it works well there and the footprint is small enough usually to come back quickly.

My rule of thumb is if any security tool executes longer than 2 minutes in a pipeline, it’s either the wrong tool, mis-configured, or you need to look at the code base to see if you are scanning unnecessary elements best covered by something else.

Gene Gotimer

I like that guide of 2 minutes. As a developer, I hate waiting for feedback from a slow pipeline.

DJ Schleen

Hey everyone! I’m wrapping up my takeover! Thank you so much for the questions this morning. I truly enjoyed this morning and hope to drop in again. If anyone wants to reach out to me with any additional or followup questions feel free to send me a message on Twitter (https://twitter.com/djschleen) - it’s the best way to get a hold of me. I’m out…

owen

Thanks for swinging by DJ! Really appreciate all the insight and random hockey facts. Excited to hear your keynote at #agiledevopsvirtual in June!