DevOps at Sonatype

Lessons Learned from Using Repository Managers and Software Supply Chain Tools

Manfred Moser - @simpligility

  • Community Advocate, Author & Trainer

Brian Fox - @brian_fox

  • VP of Product Management

Sonatype - @sonatype - www.sonatype.com

What Are You Learning Today?

What worked? What caused problems?

→ Apply our lessons to your situation

What does Sonatype actually do?

Manages and develops:

  • Central Repository (aka Maven Central)
  • OSSRH and related Forges
  • Nexus Repository Manager
  • Nexus IQ Server
  • Nexus IQ Data Services
  • Related documentation, websites, blogs, videos,…

Software Component Warehouse

Nexus Repository Manager

  • 70% market share
  • > 61K active server installations
  • Open Source: Maven2, npm, RubyGems, NuGet, Docker, …
  • Commercial: Component Information, Staging, Smart Proxy, User Token, …
  • Nexus 3 - distinct new codebase

Software Supply Chain Solutions

Solutions:

  • Nexus Firewall
  • Nexus Auditor
  • Nexus Lifecycle

Include

  • Nexus IQ Server
  • Integrations with Jenkins, Hudson, Bamboo
  • Eclipse IDE/M2e Plugin
  • SonarQube integration
  • Command-line tool and REST API

All of this is backed by…

Nexus IQ Data Services

Data and services for:

  • Age
  • Popularity
  • Security vulnerabilities
  • License information
  • Multiple component formats

→ Constantly updated, curated and accurate data.

Central Repository

aka. Maven Central
  • Largest Maven2 format repository
  • High performance, global CDN
  • Default in Apache Maven and others

And the components come from… OSSRH and Forges

  • OSSRH - large deployment of Nexus Repository Manager
  • Apache, JBoss, java.net … - secondary Nexus instances
  • Community support - on-boarding and documentation

Central and OSSRH Numbers

  • > 17 billion download in 2014
  • > 1 million GAV coordinates
  • Currently about 100k projects total
  • Approx. 3000 new projects each month (GA)
  • 10 - 30 project verified and onbarded per day
  • Approx. 30.000 new releases each month (GAV)

Who Helps at Sonatype

  • Internationally distributed
  • Multiple-time zones
  • Remote work the rule, not the exception
  • Roughly 100 people
Tip Western North America to Eastern Europe
images/nexus-team-timezones.png

Teams

  • Numerous smaller teams
  • Different focus of teams
  • Cross-team members
  • Dynamic grouping around efforts - task force

Process

In a nutshell - nothing special, no surprises.

images/usual-process.png

Process

  • Scrum framework
  • Kanban inspired
  • Backlog refinement
  • Regular meetings

→ Differs per team!

Everyone has their own process. You need to figure out what works for you!

Product Owner Team

Multi-disciplinary team:

  • Security
  • Development
  • Architecture
  • User experience
  • Documentation

Communication

  • Good old phone and VOIP
  • Atlassian HipChat
  • Google Hangouts
  • join.me
  • PagerDuty
Tip Using video more has helped avoid misunderstandings.

Track and Plan

  • Atlassian JIRA
  • Trello
  • Basecamp
  • Aha.io
  • Salesforce

Tool Lessons:

  • Different people use different tools
  • Overlap is inevitable
  • Be prepared to implement integrations
  • Tools come and go - be agile

(Maven) Project Complexity

Find balance for

  • Number vs size of projects
  • Multi-module vs multiple projects
  • Consider release cycle
  • Branching, Git and CI integration
  • IDE functionality
  • Build time
Tip Example Nexus OSS and Nexus Pro

Develop

  • Feature branches
    • short lived
    • sometimes shared between
    • automatic Bamboo feature branch build creation
    • feature flags for longer lived efforts
  • IDE
    • Eclipse IDE
    • IntelliJ IDEA
  • Lots of OSX, some Windows & Linux

Test

Unit, functional and manual

  • Junit
  • Geb
  • Spock
  • Pax Exam
  • Selenide
Tip No tests, no merge!

Document

Multiple output formats from:

  • Atlassian Confluence
  • Google Docs
  • Asciidoc
  • Pelican

Instituting development workflows including

  • Git-based versioning
  • and branching,
  • pull requests and reviews
  • and CI builds

is very useful!

Continuously Build

  • Atlassian Bamboo with Elastic Bamboo
  • > 100 build plans
  • Feature branch builds increases number
  • Automated test, release and deployment
  • Base plan build with shared artifact
  • All plans - similar setup
  • Share outputs as artifacts
Tip Consistency helps users and administrators.

Validate

  • SonarQube - integrated in Bamboo and GitHub
  • License check with Maven plugin
  • Pull requests and code reviews
    • No merges without build passing and code review
  • Component policy with Nexus Lifecycle

Release

  • Workflow and notification with Nexus staging
  • Including validation with Nexus Lifecycle
    • Security checks
    • License checks
    • Architecture checks (e.g. component age)
  • Usage of release build number - 2.11.4-01
  • Same release stuff on OSSRH
Tip No matter what you do .. there is always a chance something goes wrong.

Software Supply Chain Management

We are dogfooding our own tools
  • Nexus Repository Manager
  • Nexus Lifecycle

including Bamboo integration and IDE integration.

Nexus Repository Manager

  • Component source for consumers
  • Component target for producers
images/producers-consumers.png

Colocate For Performance

Continuous integration is consumer and producer.

Best practice:

  • Get it close together
  • And sync to another repository if needed.
images/nexus-bamboo-rso.png

Nexus Repository Manager Tips

Here are a few things that work for us

  • Versioning and component deployment
    • Only SNAPSHOT versions of master are deployed
    • Feature branch versions are not deployed
  • Multiple server installations
    • In different networks
    • Smart proxy between them
  • Release with Staging
    • Dogfooding ourselves
    • Thousands of users and projects on OSSRH

Nexus Lifecycle

  • Define risks we care about
  • Open source contributions change our policy
  • Understand our process and tooling
  • Limit overhead in our build automation

We gain

  • Visualized risk through rule-based automation
  • Streamlined component selection based on real time data

Nexus IQ Server Deployment

images/nexus-iq-server-integration.png

Policy Configuration

Simplified version:

images/sonatype-policy.png

Resulting Report

Overview section in notification:

images/nexus-clm-report.png

Black Listing and White Listing

Define

  • Which components are okay to be used?
  • Which components are not okay to be used?

Problem

  • Too many criteria
  • Complex and labor intensive to figure out criteria and values
  • Usage influences criteria
  • Different usage for different projects
Important It just doesn’t work! Too slow. Not scalable.

Golden Repository

Only the good components can be in the repository.

Problems:

  • Components age like milk, not wine!
  • A golden repository per project?
  • Does not scale
Important On the surface it looks easy. It’s not!

Perimeter Protection

Nexus Firewall

  • Requires up to date and accurate information
    • As provided by Nexus IQ Data Services
  • Tremendous help to reduce influx
  • But does not control usage
Tip Helps, but is not the full solution. Just like a network firewall. Its not enough.

Nexus Lifecycle Lessons

Once we had Nexus Lifecycle and started using it…

  • Surprised how many components are used
  • Blocking a release for policy violations
    • is a big stick
    • but it works
  • Shared ownership helps - socialize the resolution/enforcement process
  • Initial introduction forced some cleanup of old issues
  • Ongoing low noise and fast results increases usage, adoption

→ Without the automation this would be not achievable!

Operations - Service Management

Nexus as component warehouse with Ansible

images/service-management.png

Community

  • Actively work with vendors
  • Including open source projects
  • Help upstream to help yourself
    • Report issues
    • Release testing
    • Contributions
  • Avoid forking third party libraries
    • But do it cleanly when necessary
    • And send back upstream

What’s Next?

Tip Come to our booth for demos, T-shirts and more.

The End

Want to help us → we are hiring!

Questions, remarks & discussion

Slides

Resources