Creating a PyPi pull-through cache that is ready for Kubernetes

Packages are fun, let’s cache some!
  • If the main service goes down, you still have access to packages that are in the cache as backup
  • Remember left-pad?
  • It could be within your own network, in other words, an “offline cache.”
  • Can reside close to your CI servers lowering install times and internet traffic
  • Lower costs in cloud environment due to no internet traffic needed

What is a pull-through cache?

Choosing the caching software

  • Settings them up requires good knowledge of how to configure them, and there is a lot of moving parts.
  • Getting them running on Kubernetes, especially Nexus, is not always easy due to the lack of cloud-specific documentation.
  • Keeping the application container stateless can be tricky. Storing data in object storage can cost you extra due to licensing, or it can be challenging to do. (This, of course, makes the assumption that you want to keep things stateless as one usually does in Kubernetes)
  • PyPIServer: Version 1.4.2, released on 2020–10–10
  • PyPICloud: 1.2.4, release date 2021–06–10
  • DevPi (devpi-server): 6.0.1, release date 2021–06–23

Setting up PyPICloud

Creating a flexible configuration file

Creating a Docker setup

Deploying the application to Kubernetes

  • K8S_SECRET_DB_URL: For settings the database URL
  • K8S_SECRET_STORAGE_BUCKET: For pointing to your GCS bucket
  • K8S_SECRET_GOOGLE_APPLICATION_CREDENTIALS: For authenticating with Google Cloud

Using the cache

PIP

[global]
timeout = 5
index-url = https://pypi.<YOUR_DOMAIN>.com/simple

Poetry

[[tool.poetry.source]]
name = “my-pypi-cache”
url = https://pypi.<YOUR_DOMAIN>.com/simple
default = true

Pipenv

Conda

--

--

--

Anders is a Finnish IT company, whose mission is sustainable software development with the greatest colleagues of all time.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

5 Things You Should Know to Break Into Developer Field

Course Review: Numpy Linear Algebra

Digital Wallet(DID solution)

Python: List Comprehension tricks

Calculating and Visualizing the Magnetic Field Due to a Long Straight Wire With Python

How can one generate revenue with an elearning app like Udemy?

Using GraphiQL without installing GraphiQL

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Anders Innovations

Anders Innovations

Anders is a Finnish IT company, whose mission is sustainable software development with the greatest colleagues of all time.

More from Medium

Setup PyFlink Development Environment

Elasticsearch Installation and Configuration

Using Materialize and Redpanda to Analyze Raspberry Pi Temperature Data

An Open Source High-Performance Aws Kubernetes Cluster