Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Script to deploy linux wheel to S3 on release. #1745

Merged
merged 1 commit into from
Jan 9, 2015
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 9 additions & 6 deletions ci/travis/after_success-release-linux.sh
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ echo
echo "Running after_success-release.sh..."
echo

echo "Installing boto..."
pip install boto --user || exit
echo "Installing wheel..."
pip install wheel --user || exit
echo "Installing twine..."
Expand All @@ -32,11 +34,12 @@ sudo pip install twine || exit
echo "Creating distribution files..."
# This release build creates the source distribution. All other release builds
# should not.
python setup.py sdist bdist || exit
python setup.py sdist bdist bdist_wheel || exit
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why use twine below in lieu of python setup.py sdist bdist bdist_wheel upload ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following the recommendation of the official Python Packaging Guide.

(Recommended): Use twine

twine upload dist/*

The biggest reason to use twine is that python setup.py upload (option # 2 below) uploads files over plaintext. This means anytime you use it you expose your username and password to a MITM attack. Twine uses only verified TLS to upload to PyPI protecting your credentials from theft.


echo "Created the following distribution files:"
ls -l dist
# These should get created on linux:
# nupic-0.0.33-cp27-none-linux-x86_64.whl
# nupic-0.0.33.linux-x86_64.tar.gz
# nupic-0.0.33-py2.7-linux-x86_64.egg
# nupic-0.0.33.tar.gz
Expand All @@ -47,10 +50,10 @@ twine upload dist/nupic-${NUPIC_VERSION}*.egg -u "${PYPI_USERNAME}" -p "${PYPI_P
echo "Uploading source package to PyPi..."
twine upload dist/nupic-${NUPIC_VERSION}.tar.gz -u "${PYPI_USERNAME}" -p "${PYPI_PASSWD}"

# This doesn't work because PyPi rejects linux platform wheel files.
# We can't upload the wheel to PyPi because PyPi rejects linux platform wheel
# files. So we'll push it up into S3.
# See: https://bitbucket.org/pypa/pypi-metadata-formats/issue/15/enhance-the-platform-tag-definition-for

# python setup.py bdist_wheel || exit
# echo "Created the following linux platform wheel:"
# ls dist/*.whl
# echo "Not doing anything with this wheel at this time."
wheel_file=`ls dist/*.whl`
echo "Deploying ${wheel_file} to S3..."
python deploy-wheel-to-s3.py "${wheel_file}"
56 changes: 56 additions & 0 deletions ci/travis/deploy-wheel-to-s3.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
#!/usr/bin/env python
# ----------------------------------------------------------------------
# Numenta Platform for Intelligent Computing (NuPIC)
# Copyright (C) 2013, Numenta, Inc. Unless you have an agreement
# with Numenta, Inc., for a separate license for this software code, the
# following terms and conditions apply:
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License version 3 as
# published by the Free Software Foundation.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
# See the GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see http://www.gnu.org/licenses.
#
# http://numenta.org/licenses/
# ----------------------------------------------------------------------

import os
import sys
import boto
from boto.s3.key import Key

# This script assumes the following environment variables are set for boto:
# - AWS_ACCESS_KEY_ID
# - AWS_SECRET_ACCESS_KEY

REGION = "us-west-2"
BUCKET = "artifacts.numenta.org"
RELEASE_FOLDER = "numenta/nupic/releases"



def upload(artifactsBucket, wheelFileName, wheelPath):
key = Key(artifactsBucket)
key.key = "%s/%s" % (RELEASE_FOLDER, wheelFileName)
print "Uploading %s to %s/%s..." % (wheelFileName, BUCKET, RELEASE_FOLDER)
key.set_contents_from_filename(wheelPath)



def run(wheelPath):
wheelFileName = os.path.basename(wheelPath)
conn = boto.connect_s3()
artifactsBucket = conn.get_bucket(BUCKET)
upload(artifactsBucket, wheelFileName, wheelPath)



if __name__ == "__main__":
wheelPath = sys.argv[1]
run(wheelPath)