Pulp 2.1.1 Released

The Pulp team is proud to announce the availability of its 2.1.1 release. This is a minor release containing the following:

• Another round of performance improvements to the yum repository sync process. Our testing saw the following changes:
  • Sync time reduced from ~65 mins to ~35 mins. In Pulp 2.1.0 there is a significant delay as the yum metadata is transferred from Grinder to Pulp. To the user this appeared as if the process was frozen. The metadata retrieval is now done in Pulp itself, greatly reducing this initial metadata processing step.
  • Resyncing an existing repository reduced from ~25-30 mins to ~6-10 mins.
  • Memory footprint reduced by 30%.
• Enhancements to our applicability API to support the next iteration of Katello.
• Additional bug fixes and enhancements.

The release can be found in our stable repositories (pulp-v2-stable in the .repo file if you’ve downloaded it). Installation and upgrade instructions can be found in the installation section of the user guide. All of our documentation can be accessed from our documentation summary page.

Pulp 2.2.0

Our May sprint is the our last feature and development sprint for the 2.2.0 release. Starting the first week in June, beta builds for this release will be available. I’ll provide more of a preview on what to expect when I have a bit more time.

Pulp 2.1.0 Released

The Pulp team is proud to announce the availability of its 2.1.0 release. Highlights from this release include…

• Pulp Nodes functionality, allowing one Pulp server to manage and push content to other Pulp servers (previously known as a CDS in Pulp v1).
• Support for Puppet Master consumers to have modules remotely installed and upgraded through the Pulp admin client.
• Significant speed and memory consumption improvements when synchronizing yum repositories.
• Support for upgrading from Pulp 1.1 installations.
• A number of bug fixes.

The full release notes can be found in our user guide.

The release can be found in our stable repositories (pulp-v2-stable in the .repo file if you’ve downloaded it). Installation and upgrade instructions can be found in the installation section of the user guide. All of our documentation can be accessed from this list.

Pulp 2.1.1

Work is nearing completion for our 2.1.1 release. This release is primarily to support an upcoming Katello release. Of particular interest will be the inclusion of another round of performance improvements when synchronizing yum repositories. Beta builds for this release are scheduled to begin next week with the release coming in early May.

Pulp 2.0.7 Released

The Pulp team would like to announce the availability of its 2.0.7 release. This is a bug fix release for the initial 2.0 release earlier this year.

The release can be found in our stable repositories (pulp-v2-stable in the .repo file if you’ve downloaded it).

The full list of bugs included in this release can be found here.

Roadmap

Roadmap

With Pulp 2.0 out, it’s time to start looking ahead to where we go from here. The release at the start of the year made for a clean break to align future releases against calendar quarters. The current state of the roadmap depicts the releases and rough feature alignment for the rest of this year.

And I do mean rough, especially once you get past the 2.2 release in July. We have our ideas on what we want to accomplish, but with our close ties to Red Hat projects, we have a number of outside influences that have the potential to keep most planning from being written in stone. That also means that things may be shuffled around due to community request. What I’m getting at is the expected disclaimer that everything on there is subject to change. As a particular release draws closer, the aligned features will stabilize, so this will very much be a living document.

The roadmap can be found here.

Release 2.0.7

While I’m talking releases, there will be a critical bug fix release to the 2.0 release in the next week or so. The list of bugs aligned to it can be found here.

Release 2.1.0

In addition to the features mentioned in the roadmap, the list of bugs aligned to the 2.1.0 release in early April can be found here.

Pulp v2.0 Released

I am extremely excited to announce the immediate availability of Pulp 2.0. This version of Pulp represents an entirely new approach, evolving Pulp into an extendable framework for content management. No longer limited to RPM-related content, users have the ability to customize their Pulp installation to handle their specific content management needs. This release sees the first usage of this ability, providing support for both RPM-related content (RPMs, errata, etc.) as well as handling Puppet modules, including synchronizing all or a subset of modules hosted at Puppet Forge.

Quick Links

• Documentation Home
• Mailing List
• Chat: #pulp on Freenode

Existing Users

Pulp v2 beta users can upgrade to the release version. The .repo files have been updated with the v2 stable repository as the default enabled repository. Alternatively, simply enable the [pulp-v2-stable] repository definition in your existing copy.

Unfortunately, this release does not support upgrading from a Pulp v1 installation. Work is being done to support this in the next two months, but it didn’t make it into this release.

What is Pulp?

(I shamelessly copied this directly from www.pulpproject.org, so don’t be surprised if this looks familiar)

Pulp is a platform for managing repositories of content, such as software packages, and pushing that content out to large numbers of consumers. If you want to locally mirror all or part of a repository, host your own content in a new repository, manage content from multiple sources in one place, and push content you choose out to large numbers of clients in one simple operation, Pulp is for you!

Pulp has a well-documented REST API and command line interface for management.

Pulp is completely free and open-source, and we invite you to join us on GitHub!

With Pulp Server, you can…

• Pull in content from existing repositories to the Pulp server. Do it manually one-time-only or on a recurring schedule.
• Upload new content to the Pulp server.
• Mix and match uploaded and imported content to create new repositories, then publish and host them with Pulp.
• Publish your content as a web-based repository, to a series of ISOs, or in any other way that meets your needs.

Pulp Consumers can….

• Register with Pulp server, bind to a repository, then have their installed content managed from the server.
• Pulp server centrally manages and reports on what content is on each consumer.

Pulp in 2013

With both 2012 winding down and Pulp 2.0 approaching release, I wanted to lay out some information on where Pulp will be going in the new year.

Pulp 2.0 Release

For the past few weeks, we have been focusing on getting the next version of Pulp in a releasable state. The release is nearly complete and the final beta build should be available before the end of this week.

This is a good time to mention another house keeping item. The entire team will be on vacation starting on December 24th and extending until January 2nd, so don’t be surprised to find the chat room quieter than usual. That said, the community itself is rather helpful, so any questions will likely still be answered.

Following that, the release should be during the first week of January. However, anyone looking to get an early start over the holiday should feel free to use the release candidate build which will be located in our beta repository. This build will very likely become the final release when we return in January.

Pulp 2.0.x Release

When the 2.0 release becomes available, it will unfortunately not support upgrading from a 1.1 Pulp installation. That functionality is largely finished but still needs some polish and testing. I’m looking to release a minor update in mid-February containing this functionality.

Alongside this release will be the Pulp Developer Guide. That guide will contain:

• Documentation on our REST APIs
• Documentation for integrating with Pulp’s event notification system over HTTP or AMQP.
• Guides for using the plugin infrastructure, including information on writing server-side plugins, client-side extensions to the client, and agent handlers to field requests for new content types.

Pulp 2.x Releases

Going forward, I’m looking to have a much more regular release cycle. The plan is to have a new point release every three months. Each release will feature a handful of marquee features, minor tweaks and improvements, and, of course, bug fixes. The 2.1 release is planned for early April and will feature the first pass at the v2 implementation of CDS functionality.

I’ve even gone so far as to start putting together a roadmap to facilitate this release cycle. I have every intention of making the roadmap visible, I’m just still trying to figure out the best way to do that. I’ll post more about it when I come to a decision.

Beyond

Once 2.0 is released, one of the main focuses will be adding support for new content types. As a team, we have plans for what we want to support. In addition to that, we want to encourage contributions from the community. Interest has already been expressed by a number of people for writing support for Debian packages. The Developer Guide is the first major step in this direction. Future work will also focus on providing libraries to facilitate the more mechanical parts of plugins, such as components to support more robust downloading of files (throttling, proxy support, duplicate file detection) from an external source.

Alongside the point releases, I’m looking into setting up a demo of sorts to talk about the new features and give the community a chance to provide feedback directly with the team. I’m still figuring out the technology (conference call, google hangout, etc.) and I’ll have more information in the future.

Wrapping Up

This is probably a good time to both thank and congratulate the team on an impressive year. The work on 2.0 has set us up to greatly increase the capabilities of Pulp well beyond simply supporting RPMs and errata. I’m excited to see where we go from here.

Pulp v2 Community Release 2

The second community release of the v2 codebase is now available. It can be accessed by enabling the pulp-v2-stable repository in the *-pulp.repo files (due to the versioning scheme, all other repositories should be disabled to ensure the v2 builds are installed).

Overview

By now I’ve talked pretty extensively about how the Pulp’s v2 codebase is a radical shift from our v1 offering. This release is the second milestone in our reworking of Pulp from its humble beginnings as an RPM repository manager into a much more powerful and extensible platform (while still maintaining its RPM support).

There are a number of new features and bug fixes, but as with the last community release, this is intended more as a preview release rather than production level responsibilities.

Installation & Upgrade

Installation

Installation instructions can be found in our Pulp User Guide.

The 2.0 and 1.x releases cannot be installed at the same time.

In this release, the SELinux policy RPM is not installed automatically with the server. SELinux support can be added once the server is installed by installing the pulp-selinux RPM.

Upgrades

Upgrades from 1.x are not currently supported. This will be added in a future release, most likely Community Release 4.

Features

I’ll be completely honest, we cut this branch a few weeks ago. QE took some time to test and there was a period of about a week and a half where it sat ready to be released but was held up for dumb reasons. As such, I’m a bit fuzzy on what made it into CR-2 v. what will be in CR-3. There’s a ton of work that’s been done, all of it awesome, it’s just a matter of this release coming in the middle of a change to how we want to handle releases which made these release notes tricky to write. We’re cleaning up our release notes tracking process and this will get better, I promise. – jdob

• Event Notification System – The Pulp server can be configured to send events based on activities within the system. This release includes the basis for that framework as well as the ability to configure events to be sent to a REST API. The next release will include notifications sent by e-mail or posted to an AMQP message bus.
• Profiler Plugin Framework – A profiler is a server-side plugin that provides type-specific support for operations related to consumers. This release includes the definition, framework, and hooks into the appropriate places in the Pulp server workflows to enhance the capabilities of Pulp’s remote installation functionality. Also included is the first usage of these plugins in the form of server-side translation of errata installation requests into the appropriate RPM upgrades based on the consumer’s latest reported package profile. The profiler also includes the ability to calculate applicable errata on a given set of consumers, however this ability is only accessible through the REST APIs and has yet to be exposed in the CLI.
• Increased Installation Options – In addition to the previously supported installation of RPMs, the client now supports installation of errata and package groups on a consumer.
• Repository Group Support – Repository groups have been added back to Pulp. In this iteration, Pulp’s criteria system is supported to greatly enhance the interface for manipulating group membership. Work continues in this area to flush out the operational features around groups.
• Standardized Criteria UI – In v2, Pulp exposes a strong query syntax for all types of resources in what is referred to as a Pulp criteria. Criteria can include anything from a direct field matching to regular expressions. This release includes the beginning of standardizing the client’s format for defining a criteria when searching in Pulp.
• Client Support for Users – Pulp’s user and roles system has been reworked. The v2 incarnation is included in this release.
• Performance and Stability Enhancements – I realize this sounds like a generic catch-all, but given the newness of the code and QE’s ability to settle in and hammer on CR-1, there really was an impressive amount of bug fixes. A number of problem areas, for example the performance of retrieving a large number of repositories, were identified and addressed.

Future Work

Work has already begun on Community Release 3. That release will include the Puppet support written about in the past few articles. Another notable change is the removal of createrepo from the metadata generation process, greatly increasing the performance of publishing yum repositories. This release should be available within the next two months to get these two major features, along with many others, in the community’s hands.

Support

Any and all comments, suggestions, and bug reports are appreciated. In the coming months we’ll be working to polish the new codebase and every bit helps. As usual we can be reached on our mailing list and in #pulp on Freenode.

Customizing Puppet Module Repositories in Pulp

In my last post, I covered mirroring Puppet Forge modules in Pulp. Pulp offers much more than the simple mirroring of content. This article covers creating custom repositories from both existing and newly uploaded content.

Uploading Puppet Modules

The ability to synchronize modules from an external source such as Puppet Forge is extremely useful in a number of use cases. However, there are also situations where a module does not exist in a source containing the proper structure to be imported by Pulp. In those cases, Pulp offers the ability to upload modules into a repository.

All upload-related commands are found under the pulp-admin puppet repo uploads section of the admin client. In addition to the upload command itself, that section contains commands to pause an upload and resume it at a later time or see a list of in progress and paused uploads.

Uploading a module is done through the pulp-admin puppet repo uploads command. The ID of the repository into which the modules will be uploaded is required. Modules to upload are indicated either by individual file (--file) or by directory (--dir). Either argument may be specified more than once and may be used in conjunction with each other. In the case of uploading a directory, only files ending in .tar.gz will be uploaded, so it is safe to have non-module content in the directory as well.

For the example below, a new repository is created to house the uploaded modules. Note that it does not have a feed; this repository cannot be synchronized against an external source, however it may still have modules uploaded or copied into it.

$ pulp-admin puppet repo create --repo-id upload-repo
Successfully created repository [upload-repo]

A directory of modules will be uploaded as an example. The output updates in regular intervals as the upload proceeds. At any time, the upload may be paused by pressing ctrl+c. The queued files that have not completed uploading will be remembered and can be resumed at a later point. Files that have been partially uploaded will retain the offset and begin uploading where they left off.

$ ll /home/jdob/vault/code/data/puppet-modules
total 12
-rw-rw-r--. 1 500 500 1895 Aug 30 14:21 adob-good-2.0.0.tar.gz
-rw-rw-r--. 1 500 500 1949 Aug 30 14:22 jdob-valid-1.0.0.tar.gz
-rw-rw-r--. 1 500 500 1952 Aug 30 14:22 jdob-valid-1.1.0.tar.gz
 
$ pulp-admin puppet repo uploads upload --repo-id upload-repo --dir /home/jdob/vault/code/data/puppet-modules
+----------------------------------------------------------------------+
                          Puppet Module Upload
+----------------------------------------------------------------------+
 
Extracting necessary metdata for each file...
[==================================================] 100%
Analyzing: jdob-valid-1.1.0.tar.gz
... completed
 
Creating upload requests on the server...
[==================================================] 100%
Initializing: jdob-valid-1.1.0.tar.gz
... completed
 
Starting upload of selected packages. If this process is stopped through ctrl+c,
the uploads will be paused and may be resumed later using the resume command or
cancelled entirely using the cancel command.
 
Uploading: adob-good-2.0.0.tar.gz
[==================================================] 100%
1895/1895 bytes
... completed
 
Importing into the repository...
... completed
 
Deleting the upload request...
... completed
 
Uploading: jdob-valid-1.0.0.tar.gz
[==================================================] 100%
1949/1949 bytes
... completed
 
Importing into the repository...
... completed
 
Deleting the upload request...
... completed
 
Uploading: jdob-valid-1.1.0.tar.gz
[==================================================] 100%
1952/1952 bytes
... completed
 
Importing into the repository...
... completed
 
Deleting the upload request...
... completed

Displaying the list of repositories reflects the three modules that have been uploaded:

pulp-admin puppet repo list
+----------------------------------------------------------------------+
                              Repositories
+----------------------------------------------------------------------+
 
Id:                 upload-repo
Display Name:       upload-repo
Description:        None
Content Unit Count: 3

Searching for Puppet Modules in a Repository

Before covering copying modules between repositories, it makes sense to explain Pulp’s query syntax.

Pulp isn’t merely meant as a pass through for content as it comes into and leaves the system. Pulp provides an extremely powerful and flexible query syntax for analyzing content in the system.

One usage of this query capability is seen when searching for modules in a repository. The simplest example is to list all modules in the repository (excuse the lame data, these are some test modules I threw together for unit tests):

$ pulp-admin puppet repo modules --repo-id upload-repo
Name:         good
Version:      2.0.0
Author:       adob
Dependencies: 
Description:  Good Module Description
License:      Apache License, Version 2.0
License :     None
Project Page: http://example.org/adob-good
Source:       http://example.org/adob-good/source
Summary:      Good Module Summary
Tag List:     None
Types:        
 
Name:         valid
Version:      1.0.0
Author:       jdob
Dependencies: 
  Name:                jdob/dep-alpha
  Version Requirement: >= 1.0.0
  Name:                ldob/dep-beta
  Version Requirement: >= 2.0.0
Description:  Valid Module Description
License:      Apache License, Version 2.0
License :     None
Project Page: http://example.org/jdob-valid
Source:       http://example.org/jdob-valid/source
Summary:      Valid Module Summary
Tag List:     None
Types:        
 
Name:         valid
Version:      1.1.0
Author:       jdob
Dependencies: 
  Name:                jdob/dep-alpha
  Version Requirement: >= 1.1.0
  Name:                ldob/dep-beta
  Version Requirement: >= 2.1.0
Description:  Valid Module Description
License:      Apache License, Version 2.0
License :     None
Project Page: http://example.org/jdob-valid
Source:       http://example.org/jdob-valid/source
Summary:      Valid Module Summary
Tag List:     None
Types:

All of the fields in the module’s Moduleinfo file are inventoried in Pulp (checksums are there as well, but hidden to make the client more readable). The tag list is not found in the Moduleinfo; my plan is to eventually support specifying these at upload time. The tag list is, however, captured when synchronizing modules from Puppet Forge and is a candidate for querying against with Pulp.

Before going into the query abilities, the examples will read better if the output data is limited. The --fields option can be used to indicate which fields should be displayed:

$ pulp-admin puppet repo modules --repo-id upload-repo --fields name,version
Name:    good
Version: 2.0.0
 
Name:    valid
Version: 1.0.0
 
Name:    valid
Version: 1.1.0

Any field in the module’s metadata is available for searching. Values to match can be expressed in a number of ways:

• Literals: “name is httpd”
• Regular Expressions: “name matches mysql.*”
• Enumerated Values (including negation): “author is one of ‘jdob’ or ‘adob’”
• Comparisons: “version is greater than 1.0.0″
• Association Information: “modules added to the repository since August 1, 2012″

Any of the above can be combined to form more complex queries. A few examples are as follows:

$ pulp-admin puppet repo modules --repo-id upload-repo --fields name,version --str-eq "name=valid"
Name:    valid
Version: 1.0.0
 
Name:    valid
Version: 1.1.0
 
$ pulp-admin puppet repo modules --repo-id upload-repo --fields name,version --gt "version=1.0.0"
Name:    good
Version: 2.0.0
 
Name:    valid
Version: 1.1.0
 
$ pulp-admin puppet repo modules --repo-id upload-repo --fields name,version,author --match "author=.*dob$"
Name:    good
Version: 2.0.0
Author:  adob
 
Name:    valid
Version: 1.0.0
Author:  jdob
 
Name:    valid
Version: 1.1.0
Author:  jdob

Copying Modules Between Repositories

Puppet Modules in Pulp are meant to serve as building blocks for repositories. Once a module is in Pulp, be it via a sync from an external feed or uploaded, it can be copied or moved between repositories freely. The term “copying” is slightly misleading; to save disk space, the module is only ever stored once on the filesystem, it is simply associated with more than one repository.

A common usage of this ability is the promotion of content through a live -> unstable -> stable workflow (or dev -> qa -> prod depending on your terminology). Since each repository is served at its own URL, modules can be kept separate from production environments until they are tested and copied to the production repositories in Pulp.

The syntax for specifying which modules to copy is identical to the search syntax above, providing both the ability to easily copy an entire repository as well as specifically choosing a subset of modules for inclusion. Copied modules are not available at the hosted repository URL until the repository is explicitly published, allowing users to make a number of changes and validate the new contents before the modules are ever made available to the repository’s consumers.

For the example below, a new repository (again, feedless) is created as the destination for the copied modules. This new repository can be thought of as a higher repository in the workflow chains described above. Only a subset of modules — those written by jdob — will be promoted to the newly created repository. The final command in the example is used to verify only the desired modules have been copied.

$ pulp-admin puppet repo create --repo-id dest-repo   
Successfully created repository [dest-repo]
 
$ pulp-admin puppet repo list                      
+----------------------------------------------------------------------+
                              Repositories
+----------------------------------------------------------------------+
 
Id:                 upload-repo
Display Name:       upload-repo
Description:        None
Content Unit Count: 3
 
Id:                 dest-repo
Display Name:       dest-repo
Description:        None
Content Unit Count: 0
 
$ pulp-admin puppet repo copy --from-repo-id upload-repo --to-repo-id dest-repo --str-eq "author=jdob"
Progress on this task can be viewed using the commands under "repo tasks".
 
$ pulp-admin puppet repo modules --repo-id dest-repo --fields name,version,author
Name:    valid
Version: 1.0.0
Author:  jdob
 
Name:    valid
Version: 1.1.0
Author:  jdob

I won’t go into too much detail on the “repo tasks” message in the example above. The short version is that the Pulp server has a number of checks to ensure the integrity of a repository is maintained. In this case, the copy will execute asynchronously as soon as the repository is available. This is to prevent, for instance, conflicts that could arise if the repository was currently in the process of synchronizing with the external source. A link to an article with more information on this functionality, called the Pulp Coordinator, can be found in the References section below.

References

None of the above functionality is specific to Puppet. Pulp (the platform) is a framework that supports all of these features. Puppet is merely the latest type of content the team is flushing out support for. I’ve written about a number of these features in more detail with regard to their usage in the RPM support in Pulp. These articles, despite their RPM focus, provide more detail on the capabilities of Pulp.

• Unit Copy
• Coordinator
• Writing Custom Client Extensions

Mirroring Puppet Forge with Pulp

Introduction

One of the biggest changes in Pulp v2 is the ability to manage non-RPM content types. The first use case we had in mind for this is to inventory Puppet modules and serve them from the Pulp server. This includes both uploading modules into the Pulp server as well as selectively downloading and keeping up to date modules served at Puppet Forge.

Keep in mind that this is an early preview of code I’m still working on and is subject to changes in the coming weeks.

Repository Creation & Configuration

Pulp organizes content into repositories. The granularity of a repository is up to the user. A typical usage is to stage content from a live stream (such as Puppet Forge), to a testing environment, ultimately migrating the tested modules into a production usage. This paradigm can be modeled using separate Pulp repositories and copying the modules between them as appropriate.

Pulp will use the modules.json file located at Puppet Forge to determine which modules should be downloaded. Technically speaking, there is nothing that limits Pulp to using Puppet Forge. Any host can be used so long as the modules.json format is supported. Additionally, modules.json offers a limited query syntax for scoping which modules are included in its metadata. Pulp uses this query syntax to limit a repository to contain only certain modules.

For existing Pulp users, the pulp-admin client is undergoing some changes to support both RPM and Puppet content (and any further additions we support in the future). All of the Puppet repository related commands are located under the puppet section in the root of the client.

$ pulp-admin puppet repo
Usage: pulp-admin repo [SUB_SECTION, ..] COMMAND
Description: repository lifecycle commands
 
Available Sections:
  copy    - copy modules from one repository into another
  group   - repository group lifecycle commands
  publish - run, schedule, or view the status of publish tasks
  remove  - remove copied or uploaded modules from a repository
  sync    - run, schedule, or view the status of sync tasks
  uploads - upload modules into a repository
 
Available Commands:
  create - creates a new repository
  delete - deletes a repository
  list   - lists repositories on the Pulp server
  search - searches for Puppet repositories on the server
  update - changes metadata on an existing repository

Creating a repository is done, not surprisingly, through the create command. This command requires a ID to be specified to uniquely identify the repository in Pulp. While all other configuration is optional, when mirroring Puppet Forge there are two options in particular of interest:

The feed option is used to indicate the URL of the host from which to download modules. This must refer to the location in which the modules.json file resides. For Puppet Forge, this is simply http://forge.puppetlabs.com.

The query argument is used to limit which modules are downloaded into the repository. It may be specified multiple times if a single query is not enough to fully describe all of the desired modules.

For example, the following command will create a repository that will download Apache and MySQL related modules:

$ pulp-admin puppet repo create --repo-id blog-repo --feed http://forge.puppetlabs.com/ --query httpd --query mysql
Successfully created repository [blog-repo]

There are two options related to the serving of Puppet modules from the Pulp server itself. The serve-http and serve-https options are used to indicate which of these protocols the Pulp server will make the repository available over. These may be enabled individually or both may be active at once (technically, none can be active in the event a repository should contain modules for use in copying to other repositories but never itself be exposed). By default, Puppet repositories will be served over HTTP but not HTTPS.

The list of repositories is retrieved using the list command. A more advanced repository search is supported but I won’t cover it here.

$ pulp-admin puppet repo list
+----------------------------------------------------------------------+
                              Repositories
+----------------------------------------------------------------------+
 
Id:                 blog-repo
Display Name:       blog-repo
Description:        None
Content Unit Count: 0

The repository reflects that there are zero content units (where “unit” is the generic Pulp term for a piece of content it manages) because the modules have not been downloaded yet. This is done through the Pulp “sync” process.

Sync and Publish

A repository sync operation is the process through which the external feed for a repository is contacted to check for changes to its content. On the initial sync, all modules (matching any queries if specified) will be downloaded to the Pulp server and inventoried into its database. On subsequent sync operations, only new modules and new versions of existing modules will be downloaded. Additionally, any modules that were once present in the feed but have been removed will be removed from the Pulp repository as well.

The publish process is the opposite. When Pulp publishes a Puppet repository, it takes the current modules in the repository and serves them over HTTP and/or HTTPS from the Pulp server. This includes modules added to a repository from a sync operation as well as those uploaded by a user or copied from another Pulp repository.

By default, this publish operation happens automatically on the tail end of a successful sync operation. Users may trigger a publish explicitly in cases where only local changes to a repository have been made (upload, copy) or if there is no external feed configured for the repository.

A sync can be run immediately using the sync run command. By default, the command will continue to poll the Pulp server and display the progress as it synchronizes and publishes the repository. This output can be skipped using the --bg command or simply by pressing ctrl+c; the sync will continue on the Pulp server.

Below is a sample output from the sync command. Remember that the repository was configured with two queries (httpd and mysql) and is set to only publish the modules over HTTP.

$ pulp-admin puppet repo sync run --repo-id blog-repo
+----------------------------------------------------------------------+
                  Synchronizing Repository [blog-repo]
+----------------------------------------------------------------------+
 
This command may be exited by pressing ctrl+c without affecting the actual
operation on the server.
 
Downloading metadata...
[==================================================] 100%
Metadata Query: 2/2 items
... completed
 
Downloading new modules...
[==================================================] 100%
Module: 20/20 items
... completed
 
Publishing modules...
[==================================================] 100%
Module: 20/20 items
... completed
 
Generating repository metadata...
[-]
... completed
 
Publishing repository over HTTP...
... completed
 
Publishing repository over HTTPS...
... skipped

Each repository has a separate location under /pulp/puppet. Within that directory, the published repository will use the same format as Puppet Forge. Modules are located under a directory with the first letter of the author’s name and then futher under the author’s name. The module name will be–.tar.gz. Additionally, Pulp will generate a modules.json file in the root of the repository, allowing systems that use this for parsing to use a Pulp server instead of Puppet Forge. The modules.json file will be customized to include only modules in the repository when it was published.

Below are some screenshots of navigating the published repository. Firefox hides the “http” portion, but the port 80 line shows the serve-http flag was honored.

Digging into the system and releases directories shows the breakdown by the first character of each author name:

Finally, digging all the way down the hierarchy reveals the modules themselves:

Conclusion

This article only covered the basic concepts for mirroring Puppet Forge: repository creation, configuration, and synchronization/publishing. Pulp offers a number of other features in the areas of searching for modules, uploading user-defined modules, and copying modules between repositories. Future work will involve registering a puppet master with the Pulp server and using Pulp to control the deployment of modules to the puppet master.

This functionality is not yet available in a released version of Pulp (to be honest, I just finished writing the bulk of it in the past few days). However, there are options for playing with this functionality before it makes it into a formal Community Release. See the repository definition files in our repositories for information on how to enable the weekly testing builds or Community Release candidate builds. This early in the feature development, all input is welcome and will likely be incorporated.

For more information, see the Pulp mailing lists or ping me in our chat room (jdob in #pulp on Freenode).

Community Contribution: pulp-console

A member of the community (stpierre in #pulp) has written an interactive shell on top of the v1 client-side APIs. The script initializes a connection to the Pulp server and makes accessible our client-side stub objects for running commands on the server. The source and README can be found on his github project:

https://github.com/stpierre/pulp-console

Warm fuzzy feelings to stpierre for writing this. Unless, of course, it breaks your Pulp server, in which case: “Bad stpierre”.