Extracting dependencies from python sdist archives

I recently blogged about Python packaging with py2pack . Meanwhile I created a new project called metaextract (available on github and pypi) based on the experience I made while improving py2pack.

metaextract does only one thing which is extracting the metadata from a python archive. Here’s an example – the output is JSON:

$ metaextract oslo.log-3.16.0.tar.gz 
{
 "data": {
 "data_files": null, 
 "entry_points": {
 "oslo.config.opts": [
 "oslo.log = oslo_log._options:list_opts"
 ]
 }, 
 "extras_require": {
 "fixtures": [
 "fixtures>=3.0.0 # Apache-2.0/BSD"
 ]
 }, 
 "has_ext_modules": null, 
 "install_requires": [
 "debtcollector>=1.2.0", 
 "oslo.config>=3.14.0", 
 "oslo.context>=2.6.0", 
 "oslo.i18n>=2.1.0", 
 "oslo.serialization>=1.10.0", 
 "oslo.utils>=3.16.0", 
 "pbr>=1.6", 
 "pyinotify>=0.9.6", 
 "python-dateutil>=2.4.2", 
 "six>=1.9.0"
 ], 
 "scripts": null, 
 "setup_requires": [
 "pbr>=1.8"
 ], 
 "tests_require": [
 "bandit>=1.1.0", 
 "coverage>=3.6", 
 "hacking<0.11,>=0.10.0", 
 "mock>=2.0", 
 "oslosphinx!=3.4.0,>=2.5.0", 
 "oslotest>=1.10.0", 
 "python-subunit>=0.0.18", 
 "reno>=1.8.0", 
 "sphinx!=1.3b1,<1.3,>=1.2.1", 
 "testrepository>=0.0.18", 
 "testscenarios>=0.4", 
 "testtools>=1.4.0"
 ]
 }, 
 "version": 1
}

The data is directly extracted from setuptools (or distutils). You can also run metaextract directly for a setup.py file:

$ python setup.py --command-packages=metaextract metaextract

or use it from your code:

$ python
>>> import pprint
>>> from metaextract import utils as meta_utils
>>> pprint.pprint(meta_utils.from_archive("oslo.log-3.16.0.tar.gz"))
{u'data': {u'data_files': None,
 u'entry_points': {u'oslo.config.opts': [u'oslo.log = oslo_log._options:list_opts']},
 u'extras_require': {u'fixtures': [u'fixtures>=3.0.0 # Apache-2.0/BSD']},
 u'has_ext_modules': None,
 u'install_requires': [u'debtcollector>=1.2.0',
 u'oslo.config>=3.14.0',
 u'oslo.context>=2.6.0',
 u'oslo.i18n>=2.1.0',
 u'oslo.serialization>=1.10.0',
 u'oslo.utils>=3.16.0',
 u'pbr>=1.6',
 u'pyinotify>=0.9.6',
 u'python-dateutil>=2.4.2',
 u'six>=1.9.0'],
 u'scripts': None,
 u'setup_requires': [u'pbr>=1.8'],
 u'tests_require': [u'bandit>=1.1.0',
 u'coverage>=3.6',
 u'hacking<0.11,>=0.10.0',
 u'mock>=2.0',
 u'oslosphinx!=3.4.0,>=2.5.0',
 u'oslotest>=1.10.0',
 u'python-subunit>=0.0.18',
 u'reno>=1.8.0',
 u'sphinx!=1.3b1,<1.3,>=1.2.1',
 u'testrepository>=0.0.18',
 u'testscenarios>=0.4',
 u'testtools>=1.4.0']},
 u'version': 1}
Advertisements

Comparing rpm package versions with python

I was searching a bit to find a solution to compare versions for RPM packages in Python. So this is a reminder how to do it with the Python bindings for RPM:

$ python
>>> import rpm
>>> v1 = rpm.hdr()
>>> v2 = rpm.hdr()
>>> v1[rpm.RPMTAG_EPOCH] = 0
>>> v2[rpm.RPMTAG_EPOCH] = 0
>>> v1[rpm.RPMTAG_RELEASE] = "0"
>>> v2[rpm.RPMTAG_RELEASE] = "0"
>>> v1[rpm.RPMTAG_VERSION] = "1.2.3"
>>> v2[rpm.RPMTAG_VERSION] = "1.2.4"
>>> rpm.versionCompare(v1, v2)
-1

The return values are documented in the git repository .

  • 1 means v1 is higher than v2
  • 0 means v1 and v2 are qual
  • -1 means v2 is higher than v1

Looks like this is not very well documented so hopefully this helps others.

Improved python packaging for openSUSE

The standard tool to create new Python packages for openSUSE is py2pack . The tool had some open issues so I decided to spend some time during SUSE’s Hack week to improve the tool.

The main problem with py2pack was, that the metadata detection (to get Requires and BuildRequires for RPM .spec files) was error prone because it parsed the setup.py from a sdist tarball to get the needed metadata. This was failing when

  • variables are used for i.e. install_requires or extras_require
  • No install_requires are specified (i.e. because pbr together with a requirements.txt file is used)
  • the used regular expressions are not matching for various reasons

To get rid of theses problems, another way for getting the metadata was needed. And the new way is a custom distutils command . This command runs the setup.py to receive the metadata. The command can also be used standalone when py2pack is installed:

$ py2pack fetch oslo.log
$ tar xfz oslo.log-3.11.0.tar.gz 
$ cd oslo.log-3.11.0/
$ python setup.py --command-packages=py2pack get_metadata
running get_metadata
{
 "install_requires": [
 "pbr>=1.6", 
 "six>=1.9.0", 
 "oslo.config>=3.10.0", 
 "oslo.context>=2.4.0", 
 "oslo.i18n>=2.1.0", 
 "oslo.utils>=3.11.0", 
 "oslo.serialization>=1.10.0", 
 "debtcollector>=1.2.0", 
 "pyinotify>=0.9.6", 
 "python-dateutil>=2.4.2"
 ], 
 "entry_points": {
 "oslo.config.opts": [
 "oslo.log = oslo_log._options:list_opts"
 ]
 }, 
 "extras_require": {
 "fixtures": [
 "fixtures>=3.0.0 # Apache-2.0/BSD"
 ]
 }, 
 "tests_require": [
 "hacking<0.11,>=0.10.0", 
 "discover", 
 "python-subunit>=0.0.18", 
 "testrepository>=0.0.18", 
 "testscenarios>=0.4", 
 "testtools>=1.4.0", 
 "mock>=2.0", 
 "oslotest>=1.10.0", 
 "coverage>=3.6", 
 "sphinx!=1.2.0,!=1.3b1,<1.3,>=1.1.2", 
 "oslosphinx!=3.4.0,>=2.5.0", 
 "reno>=1.6.2", 
 "bandit>=1.0.1"
 ]
}

And of course the command is integrated into py2pack so running:

$ py2pack generate oslo.log -f python-oslo.log.spec

generates a working .spec file. I did a new release on pypi so this work is integrated in version 0.6.3.

Happy packaging!