Pinterest engineering blog

  • Back to all posts
  • Nov 4, 2016
  • Share

Open-sourcing Rocksplicator, a real-time RocksDB data replicator

Bo Liu

Bo Liu is an engineering lead on the Serving System team, which is part of the Infrastructure team

Pinterest’s stateful online systems process tens of petabytes of data every day. As we build products and scale billions of Pins to 150 million people, we need new applications that work in a way where computation co-locates with data. That’s why we adopted RocksDB. It’s adaptable, supports basic and advanced database operations with high performance and meets the majority of requirements for building large-scale, production-strength distributed stateful services.

Pinterest engineering blog

  • Back to all posts
  • Mar 4, 2016
  • Share

Open-sourcing KingPin, building blocks for scaling Pinterest

Shu Zhang

Shu Zhang is a software engineer on the Infrastructure team

When we first started building Pinterest, we used Python as our development language, which helped us build quickly and reliably. Over the years we built many  tools around Python, including Pinball, MySQL_utils and pymemcache, as well as a set of libraries  used  daily for service communication and configuration management.

Pinterest engineering blog

  • Back to all posts
  • Feb 12, 2016
  • Share

Open-sourcing Teletraan, a code deployment system

Baogang Song

Baogang Song is an engineering lead on the Teletraan project

Teletraan is a code deployment system we built to deploy hundreds of our internal services. It supports tens of thousands hosts and has been running in production for over one year. Teletraan empowers engineers to deliver their code to Pinners quickly, safely and reliably, at scale.

Pinterest engineering blog

  • Back to all posts
  • Oct 28, 2015
  • Share

Open-sourcing Pinterest MySQL management tools

Rob Wultsch

Rob Wultsch is a database engineer on the Cloud Engineering team, and the primary author of the MySQL management tools

In the past, we’ve shared why you should love MySQL and how it helped Pinterest scale via sharding. At Oracle Open World today, we announced that we’re open-sourcing the vast majority of our automation that maintains our MySQL infrastructure.

Pinterest engineering blog

  • Back to all posts
  • Apr 10, 2015
  • Share

Learn to stop using shiny new things and love MySQL

A good portion of the startups I meet and advise want to use the newest, hottest technology to build something that’s cool, but not technologically groundbreaking. I have yet to meet a startup building a time machine, teleporter or quantum social network that would actually require some amazing new tech. They have awesome new ideas with down-to-earth technical requirements, so I kept wondering why they choose this shiny (and risky) new stuff when all they need is a good ol’ trustworthy database.

Pinterest engineering blog

  • Back to all posts
  • Mar 11, 2015
  • Share

Open-sourcing Pinball

Pawel Garbacki, Mao Ye, Changshu Liu and Jooseong Kim

Pawel is a software engineer on the Monetization team. Mao, Changshu and Jooseong are software engineers on the Data team.

As we continue to build in a fast and dynamic environment, we need a workflow manager that’s flexible and can keep up with our data processing needs. After trying a few options, we decided to build one in-house. Today we’re open-sourcing Pinball, which is designed to accommodate the needs of a wide range of data processing pipelines composed of jobs ranging from simple shell scripts to elaborate Hadoop workloads.

Pinterest engineering blog

  • Back to all posts
  • Feb 25, 2015
  • Share

Open-sourcing PINCache

Garrett Moon

Garrett is an iOS engineer on the Mobile team

Because the Pinterest iOS app downloads and processes an enormous amount of data, we use a caching system to cache models and images to avoid eating into our Pinners’ (users’) data plans. For quite some time we used TMCache to persist GIFs, JPEGs and models to memory and disk, but after using it in production, Pinners were reporting the app was hanging.

Subscribe to RSS - opensource