Pulp 2.1.1 Released

The Pulp team is proud to announce the availability of its 2.1.1 release. This is a minor release containing the following:

• Another round of performance improvements to the yum repository sync process. Our testing saw the following changes:
  • Sync time reduced from ~65 mins to ~35 mins. In Pulp 2.1.0 there is a significant delay as the yum metadata is transferred from Grinder to Pulp. To the user this appeared as if the process was frozen. The metadata retrieval is now done in Pulp itself, greatly reducing this initial metadata processing step.
  • Resyncing an existing repository reduced from ~25-30 mins to ~6-10 mins.
  • Memory footprint reduced by 30%.
• Enhancements to our applicability API to support the next iteration of Katello.
• Additional bug fixes and enhancements.

The release can be found in our stable repositories (pulp-v2-stable in the .repo file if you’ve downloaded it). Installation and upgrade instructions can be found in the installation section of the user guide. All of our documentation can be accessed from our documentation summary page.

Pulp 2.2.0

Our May sprint is the our last feature and development sprint for the 2.2.0 release. Starting the first week in June, beta builds for this release will be available. I’ll provide more of a preview on what to expect when I have a bit more time.

Pulp 2.1.0 Released

The Pulp team is proud to announce the availability of its 2.1.0 release. Highlights from this release include…

• Pulp Nodes functionality, allowing one Pulp server to manage and push content to other Pulp servers (previously known as a CDS in Pulp v1).
• Support for Puppet Master consumers to have modules remotely installed and upgraded through the Pulp admin client.
• Significant speed and memory consumption improvements when synchronizing yum repositories.
• Support for upgrading from Pulp 1.1 installations.
• A number of bug fixes.

The full release notes can be found in our user guide.

The release can be found in our stable repositories (pulp-v2-stable in the .repo file if you’ve downloaded it). Installation and upgrade instructions can be found in the installation section of the user guide. All of our documentation can be accessed from this list.

Pulp 2.1.1

Work is nearing completion for our 2.1.1 release. This release is primarily to support an upcoming Katello release. Of particular interest will be the inclusion of another round of performance improvements when synchronizing yum repositories. Beta builds for this release are scheduled to begin next week with the release coming in early May.

Pulp Presentation on YouTube

On March 14, I presented Pulp to the Triangle Linux User Group in Raleigh, NC. In addition to an overview of what Pulp does and how its pieces fit together, I talked about some of the upcoming 2.1 features such as Pulp Nodes. I also included a demo of the new Puppet consumer features.

You can watch a video of the presentation here. Skip ahead to 26 minutes for my portion of the meeting.

TriLUG is a very active user group with a wide variety of impressively-talented people and consistently well-attended meetings. The TriLUG email list has been a tremendous resource for me over the years, and it was a pleasure to be this month’s presenter. If you would like to watch future meetings online, look for the “Triangle Linux User Group” on Google+ and join the live Hangout.

Developer Guide Released

Today we are happy to present the official and mostly-new Pulp Developer Guide, available from our documentation page. The most notable addition is a detailed guide on how to add support for new content types, including examples. The Developer Guide also includes sections on Pulp’s architecture, how to integrate with Pulp, as well as the policies and conventions used by the Pulp Team.

This release is a continuation of the Pulp Team’s commitment to providing documentation that is comprehensive, accurate, and above-all: useful. Please have a look, and as always, your feedback is valued and appreciated.

Pulp 2.0.7 Released

The Pulp team would like to announce the availability of its 2.0.7 release. This is a bug fix release for the initial 2.0 release earlier this year.

The release can be found in our stable repositories (pulp-v2-stable in the .repo file if you’ve downloaded it).

The full list of bugs included in this release can be found here.

Roadmap

Roadmap

With Pulp 2.0 out, it’s time to start looking ahead to where we go from here. The release at the start of the year made for a clean break to align future releases against calendar quarters. The current state of the roadmap depicts the releases and rough feature alignment for the rest of this year.

And I do mean rough, especially once you get past the 2.2 release in July. We have our ideas on what we want to accomplish, but with our close ties to Red Hat projects, we have a number of outside influences that have the potential to keep most planning from being written in stone. That also means that things may be shuffled around due to community request. What I’m getting at is the expected disclaimer that everything on there is subject to change. As a particular release draws closer, the aligned features will stabilize, so this will very much be a living document.

The roadmap can be found here.

Release 2.0.7

While I’m talking releases, there will be a critical bug fix release to the 2.0 release in the next week or so. The list of bugs aligned to it can be found here.

Release 2.1.0

In addition to the features mentioned in the roadmap, the list of bugs aligned to the 2.1.0 release in early April can be found here.

Pulp v2.0 Released

I am extremely excited to announce the immediate availability of Pulp 2.0. This version of Pulp represents an entirely new approach, evolving Pulp into an extendable framework for content management. No longer limited to RPM-related content, users have the ability to customize their Pulp installation to handle their specific content management needs. This release sees the first usage of this ability, providing support for both RPM-related content (RPMs, errata, etc.) as well as handling Puppet modules, including synchronizing all or a subset of modules hosted at Puppet Forge.

Quick Links

• Documentation Home
• Mailing List
• Chat: #pulp on Freenode

Existing Users

Pulp v2 beta users can upgrade to the release version. The .repo files have been updated with the v2 stable repository as the default enabled repository. Alternatively, simply enable the [pulp-v2-stable] repository definition in your existing copy.

Unfortunately, this release does not support upgrading from a Pulp v1 installation. Work is being done to support this in the next two months, but it didn’t make it into this release.

What is Pulp?

(I shamelessly copied this directly from www.pulpproject.org, so don’t be surprised if this looks familiar)

Pulp is a platform for managing repositories of content, such as software packages, and pushing that content out to large numbers of consumers. If you want to locally mirror all or part of a repository, host your own content in a new repository, manage content from multiple sources in one place, and push content you choose out to large numbers of clients in one simple operation, Pulp is for you!

Pulp has a well-documented REST API and command line interface for management.

Pulp is completely free and open-source, and we invite you to join us on GitHub!

With Pulp Server, you can…

• Pull in content from existing repositories to the Pulp server. Do it manually one-time-only or on a recurring schedule.
• Upload new content to the Pulp server.
• Mix and match uploaded and imported content to create new repositories, then publish and host them with Pulp.
• Publish your content as a web-based repository, to a series of ISOs, or in any other way that meets your needs.

Pulp Consumers can….

• Register with Pulp server, bind to a repository, then have their installed content managed from the server.
• Pulp server centrally manages and reports on what content is on each consumer.

Pulp in 2013

With both 2012 winding down and Pulp 2.0 approaching release, I wanted to lay out some information on where Pulp will be going in the new year.

Pulp 2.0 Release

For the past few weeks, we have been focusing on getting the next version of Pulp in a releasable state. The release is nearly complete and the final beta build should be available before the end of this week.

This is a good time to mention another house keeping item. The entire team will be on vacation starting on December 24th and extending until January 2nd, so don’t be surprised to find the chat room quieter than usual. That said, the community itself is rather helpful, so any questions will likely still be answered.

Following that, the release should be during the first week of January. However, anyone looking to get an early start over the holiday should feel free to use the release candidate build which will be located in our beta repository. This build will very likely become the final release when we return in January.

Pulp 2.0.x Release

When the 2.0 release becomes available, it will unfortunately not support upgrading from a 1.1 Pulp installation. That functionality is largely finished but still needs some polish and testing. I’m looking to release a minor update in mid-February containing this functionality.

Alongside this release will be the Pulp Developer Guide. That guide will contain:

• Documentation on our REST APIs
• Documentation for integrating with Pulp’s event notification system over HTTP or AMQP.
• Guides for using the plugin infrastructure, including information on writing server-side plugins, client-side extensions to the client, and agent handlers to field requests for new content types.

Pulp 2.x Releases

Going forward, I’m looking to have a much more regular release cycle. The plan is to have a new point release every three months. Each release will feature a handful of marquee features, minor tweaks and improvements, and, of course, bug fixes. The 2.1 release is planned for early April and will feature the first pass at the v2 implementation of CDS functionality.

I’ve even gone so far as to start putting together a roadmap to facilitate this release cycle. I have every intention of making the roadmap visible, I’m just still trying to figure out the best way to do that. I’ll post more about it when I come to a decision.

Beyond

Once 2.0 is released, one of the main focuses will be adding support for new content types. As a team, we have plans for what we want to support. In addition to that, we want to encourage contributions from the community. Interest has already been expressed by a number of people for writing support for Debian packages. The Developer Guide is the first major step in this direction. Future work will also focus on providing libraries to facilitate the more mechanical parts of plugins, such as components to support more robust downloading of files (throttling, proxy support, duplicate file detection) from an external source.

Alongside the point releases, I’m looking into setting up a demo of sorts to talk about the new features and give the community a chance to provide feedback directly with the team. I’m still figuring out the technology (conference call, google hangout, etc.) and I’ll have more information in the future.

Wrapping Up

This is probably a good time to both thank and congratulate the team on an impressive year. The work on 2.0 has set us up to greatly increase the capabilities of Pulp well beyond simply supporting RPMs and errata. I’m excited to see where we go from here.

AMQP Notifications

Introduction

Last week I talked about Pulp’s ability to send notifications via email upon occurrence of specified events. This week I am going to show you how Pulp can send those same notifications to an AMQP message broker.

How Is This Valuable?

AMQP is an industry standard for integrating separate systems, applications, or even components within an application through asynchronous messages. If you want to take programmatic action based on an event that happened inside Pulp, this feature is for you.

How It Works

1. When an event fires inside Pulp, the notification system applies whatever handlers have been setup for it. One of those handlers is for AMQP. The body of the event becomes the body of an AMQP message, and the type of the event becomes the “subject” (that’s AMQP lingo) of that message. The message is then sent over the network to a message broker.
2. The message broker delivers the message to a topic exchange (later I will show you how to specify a named exchange to use).
3. The message is then sent to any clients that are subscribe to this exchange through queues.

Setup

$ pulp-admin event listener amqp create --help
Command: create
Description: create a listener
 
Available Arguments:
 
  --event-type - (required) one of "repo.sync.start", "repo.sync.finish",
                 "repo.publish.start", "repo.publish.finish". May be specified
                 multiple times. To match all types, use value "*"
  --exchange   - optional name of an exchange that overrides the setting
                 from server.conf

We have already talked about how to specify an event type, but here you can also specify an exchange name. If you don’t specify one, it will default to the value pulled from /etc/pulp/server.conf in the “[messaging]” section. If you don’t set one there either, Pulp will default to “amq.topic”, which is an exchange guaranteed to be available on any broker. Regardless of what name you choose (we suggest “pulp” as a reasonable choice), you do not need to create the exchange or take any action on the AMQP broker. Pulp will automatically create the exchange if it does not yet exist.

As for selecting event types, if you are unsure, I suggest going with “*” to select all of them. The client can choose which types of messages they want to subscribe to based on hierarchically matching against the event type (called a “subject” in AMQP). It is cheap and fast to send a message to a broker, so I like to fire and forget. Let the clients decide which subjects they care about. More on subject matching here.

Pulp already uses Qpid as an AMQP message broker for communication with consumers, so your Pulp installation probably has a broker setup and ready to go.

Listening In

In the playpen of the Pulp repository, I created a small utility that lets you connect to a topic exchange and print messages to stdout.

$ ./listen.py -h
usage: listen.py [-h] [-e EXCHANGE] [-s SUBJECT] [-a ADDRESS] [-p PORT] [-q]
 
print messages sent to a qpid exchange
 
optional arguments:
  -h, --help            show this help message and exit
  -e EXCHANGE, --exchange EXCHANGE
                        name of a qpid exchange (default: amq.topic)
  -s SUBJECT, --subject SUBJECT
                        message subject to bind to
  -a ADDRESS, --address ADDRESS
                        hostname to connect to (default:localhost)
  -p PORT, --port PORT  port to connect to (default: 5672)
  -q, --quiet           show message subject only

Let’s fire up the listener and see some messages come across. For our purposes, I am going to use quiet mode which will only show the subjects. Event bodies can get very detailed, and they haven’t changed since you saw them in my last post. After starting listen.py, I will start a repo sync, and then we will see events come across the topic exchange.

$ ./listen.py -e pulp -q
pulp.server.repo.sync.start
pulp.server.repo.sync.finish
pulp.server.repo.publish.start
pulp.server.repo.publish.finish

All four events fired. What if we only care about publish events? Let’s specify a subject with a wildcard.

$ ./listen.py -e pulp -q -s pulp.server.repo.publish.*
pulp.server.repo.publish.start
pulp.server.repo.publish.finish

All four events still hit the “pulp” exchange, but the client’s subscription only matched these two. Now let’s get fancy and only match “finish” events, whether for sync or publish.

$ ./listen.py -e pulp -q -s pulp.server.repo.*.finish
pulp.server.repo.sync.finish
pulp.server.repo.publish.finish

As you can see, wildcard matching within a topic exchange could be very valuable.

Summary

This feature allows the community to integrate their own systems and applications with Pulp in ways we might not even imagine using an open industry-standard protocol. As with last time, I invite you to please share with us if you have a use case for AMQP messaging with Pulp. Leave a comment here, post on our email list, or catch us on IRC (#pulp on freenode).

Email Notifications

Introduction

Did you know that Pulp has an event system? You can subscribe to specific events and be notified in several ways. This has been a popular feature because it allows other projects to integrate with Pulp, and in cases like email notification, it allows humans to monitor Pulp’s server activity. Today we are going to look at receiving notifications by email.

Diving In

Let’s take a look at the event listener system in the command line interface.

$ pulp-admin event listener --help
Usage: pulp-admin listener [SUB_SECTION, ..] COMMAND
Description: manage server-side event listeners
 
Available Sections:
  amqp    - manage amqp listeners
  email   - manage email listeners
  restapi - manage rest-api listeners
 
Available Commands:
  delete - delete an event listener
  list   - list all of the event listeners in the system

Next week I will write about the AMQP notification type, but today we will focus on email. From this section, you can list all event listeners and delete specific listeners. To create and update, we drill down by type.

$ pulp-admin event listener email create --help
Command: create
Description: create a listener
 
Available Arguments:
 
  --event-type - (required) one of "repo.sync.start", "repo.sync.finish",
                 "repo.publish.start", "repo.publish.finish". May be specified
                 multiple times. To match all types, use value "*"
  --subject    - (required) text of the email's subject
  --addresses  - (required) this is a comma separated list of email addresses
                 that should receive these notifications. Do not include spaces.

To add an email notifier, you must specify what types of events to listen to, what the email subject should be, and who should receive the emails. Let’s create an email listener now.

$ pulp-admin event listener email create --event-type="repo.sync.start" --subject="pulp notification" --addresses=someone@redhat.com
Event listener successfully created
 
$ pulp-admin event listener list
Event Types:       repo.sync.start
Id:                5081a42ce19a00ea4300000e
Notifier Config:   
  Addresses: someone@redhat.com
  Subject:   pulp notification
Notifier Type Id:  email

Settings

Take a look in /etc/pulp/server.conf at the “email” section, and read the comments. For testing, I use port 1025 on localhost and run the python command as suggested to run a dummy MTA. (see below for example)

In production, it is strongly recommended that Pulp submit email to an MTA on localhost, or at least a highly available MTA on the local network. Like many applications, Pulp makes no effort to retry if the MTA is unavailable or rejects a message.

Email Content

The emails themselves are bare-bones and not designed for luxury. In fact, the body of the email is simply a pretty-printed JSON representation of the actual event body. The messages have all of the relevant information that is available, but don’t expect a friendly greeting. That said, we want to improve this in the future, and we would love to hear about your use case. Let us know what format would be valuable to you.

In another window I ran:

pulp-admin rpm repo sync run --repo-id=pulp2

When the sync started, an email was sent whose output looked like this:

$ python -m smtpd -n -c DebuggingServer localhost:1025
---------- MESSAGE FOLLOWS ----------
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Subject: pulp notification
From: no-repy@your.domain
To: someone@redhat.com
X-Peer: 127.0.0.1
 
{
  "call_report": {
    "task_group_id": "aaa8f2ec-964c-4d62-bba9-3191aad9c3ea", 
    "exception": null, 
    "task_id": "79be77a8-1a20-11e2-aeb9-1803731e94c4", 
    "tags": [
      "pulp:repository:pulp2", 
      "pulp:action:sync"
    ], 
    "reasons": [], 
    "start_time": "2012-10-19T19:09:16Z", 
    "traceback": null, 
    "schedule_id": null, 
    "finish_time": null, 
    "state": "running", 
    "result": null, 
    "progress": {}, 
    "principal_login": "admin", 
    "response": "accepted"
  }, 
  "event_type": "repo.sync.start", 
  "payload": {
    "repo_id": "pulp2"
  }
}
------------ END MESSAGE ------------

Summary

Notification by email is a useful way to integrate Pulp with other systems, and it is the method of choice for notifying humans. For more powerful integration options, be sure to read my next post about our AMQP notifier.

Are notification features valuable to you? Tell us about your use case in a comment here, on our email list, or on IRC (#pulp on freenode).