Initial implementation of mtransfer tool (#754)

README.rst

@@ -6,7 +6,8 @@ mtools
 
 **mtools** is a collection of helper scripts to parse, filter, and visualize
 MongoDB log files (``mongod``, ``mongos``). mtools also includes ``mlaunch``, a
-utility to quickly set up complex MongoDB test environments on a local machine.
+utility to quickly set up complex MongoDB test environments on a local machine,
+and ``mtransfer``, a tool for transferring databases between MongoDB instances.
 
 .. figure:: https://raw.githubusercontent.com/rueckstiess/mtools/develop/mtools.png
    :alt: mtools box
@@ -26,15 +27,19 @@ The following tools are in the mtools collection:
    special sections like restarts, connections, distinct view
 
 `mplotqueries <http://rueckstiess.github.io/mtools/mplotqueries.html>`__
-   visualize log files with different types of plots (requires matplotlib)
+   visualize log files with different types of plots (requires ``matplotlib``)
 
 `mlogvis <http://rueckstiess.github.io/mtools/mlogvis.html>`__
    creates a self-contained HTML file that shows an interactive visualization
    in a web browser (as an alternative to mplotqueries)
 
 `mlaunch <http://rueckstiess.github.io/mtools/mlaunch.html>`__
    a script to quickly spin up local test environments, including replica sets
-   and sharded systems (requires pymongo)
+   and sharded systems (requires ``pymongo``)
+
+`mtransfer <http://rueckstiess.github.io/mtools/mtransfer.html>`__
+   an experimental script to transfer databases between MongoDB instances by
+   copying WiredTiger data files (requires ``pymongo`` and ``wiredtiger``)
 
 For more information, see the `mtools documentation
 <http://rueckstiess.github.io/mtools>`__.

doc/index.rst

@@ -17,6 +17,7 @@ utility to quickly set up complex MongoDB test environments on a local machine.
    mloginfo.rst
    mlogvis.rst
    mplotqueries.rst
+   mtransfer.rst
    contributing.rst
    changelog.rst
 
@@ -36,15 +37,19 @@ The following tools are in the mtools collection:
    special sections like restarts, connections, distinct view
 
 :ref:`mplotqueries`
-   visualize log files with different types of plots (requires matplotlib)
+   visualize log files with different types of plots (requires ``matplotlib``)
 
 :ref:`mlogvis`
    creates a self-contained HTML file that shows an interactive visualization
-   in a web browser (as an alternative to mplotqueries)
+   in a web browser (as an alternative to ``mplotqueries``)
 
 :ref:`mlaunch`
    a script to spin up local test environments quickly, including replica sets
-   and sharded systems (requires pymongo)
+   and sharded systems (requires ``pymongo``)
+
+:ref:`mtransfer`
+   an experimental script to transfer WiredTiger databases between MongoDB
+   instances by copying data files (requires ``pymongo`` and ``wiredtiger``)
 
 The `mtools source code </~https://github.com/rueckstiess/mtools>`__ is available
 on GitHub under an `Apache 2.0 license

doc/install.rst

@@ -109,7 +109,7 @@ processes.
 pymongo
 -------
 
-*required for mlaunch*
+*required for mlaunch and mtransfer*
 
 `pymongo <https://api.mongodb.com/python/current/>`__ is MongoDB's official
 Python driver. ``mlaunch`` uses this to configure and query local MongoDB
@@ -131,3 +131,11 @@ numpy
 
 `numpy <https://numpy.scipy.org/>`__ is a Python module for scientific
 computing and numerical calculations.
+
+wiredtiger
+----------
+
+*required for mtransfer*
+
+`WiredTiger </~https://github.com/wiredtiger/wiredtiger/>`__ is the default
+storage engine for MongoDB.

doc/mlaunch.rst

@@ -693,3 +693,12 @@ Optional Parameters
 
    This command displays a list of all nodes, their status and port number, and
    in addition, their startup commands.
+
+Disclaimer
+~~~~~~~~~~
+
+This software is not supported by `MongoDB, Inc. <https://www.mongodb.com>`__
+under any of their commercial support subscriptions or otherwise. Any usage of
+mtools is at your own risk. Bug reports, feature requests and questions can be
+posted in the `Issues
+</~https://github.com/rueckstiess/mtools/issues?state=open>`__ section on GitHub.

doc/mlogfilter.rst

@@ -403,3 +403,12 @@ log file:
 .. code-block:: bash
 
    mlogfilter mongod.log --from "end -2h" --to +1h
+
+Disclaimer
+~~~~~~~~~~
+
+This software is not supported by `MongoDB, Inc. <https://www.mongodb.com>`__
+under any of their commercial support subscriptions or otherwise. Any usage of
+mtools is at your own risk. Bug reports, feature requests and questions can be
+posted in the `Issues
+</~https://github.com/rueckstiess/mtools/issues?state=open>`__ section on GitHub.

doc/mloginfo.rst

@@ -392,3 +392,12 @@ For example:
    local1.myCollection       insert       None         None            None                 None
    invoice-prod.invoices     insert       12768411     22233323        86313                12344
    invoice-prod.invoices     insert       12868411     22233323        86313                12344
+
+Disclaimer
+~~~~~~~~~~
+
+This software is not supported by `MongoDB, Inc. <https://www.mongodb.com>`__
+under any of their commercial support subscriptions or otherwise. Any usage of
+mtools is at your own risk. Bug reports, feature requests and questions can be
+posted in the `Issues
+</~https://github.com/rueckstiess/mtools/issues?state=open>`__ section on GitHub.

doc/mlogvis.rst

@@ -42,3 +42,12 @@ Version
 -------
 ``--version``
    shows the version number and exits.
+
+Disclaimer
+~~~~~~~~~~
+
+This software is not supported by `MongoDB, Inc. <https://www.mongodb.com>`__
+under any of their commercial support subscriptions or otherwise. Any usage of
+mtools is at your own risk. Bug reports, feature requests and questions can be
+posted in the `Issues
+</~https://github.com/rueckstiess/mtools/issues?state=open>`__ section on GitHub.

doc/mplotqueries.rst

@@ -571,3 +571,12 @@ Remove Overlays
    To remove all overlays, you can use this command. It will delete all
    existing overlays, and the next (or current, if a log file is specified as
    well) call to **mplotqueries** will not show additional overlays.
+
+Disclaimer
+~~~~~~~~~~
+
+This software is not supported by `MongoDB, Inc. <https://www.mongodb.com>`__
+under any of their commercial support subscriptions or otherwise. Any usage of
+mtools is at your own risk. Bug reports, feature requests and questions can be
+posted in the `Issues
+</~https://github.com/rueckstiess/mtools/issues?state=open>`__ section on GitHub.

doc/mtransfer.rst

@@ -0,0 +1,161 @@
+.. _mtransfer:
+
+=========
+mtransfer
+=========
+
+``mtransfer`` allows WiredTiger databases to be exported from one MongoDB
+instance and imported into another.
+
+
+Caveats
+~~~~~~~
+
+The ``mtransfer`` script is EXPERIMENTAL and has a number of important usage caveats:
+
+- MongoDB must be started with the ``--directoryperdb`` flag.
+- ``mtransfer`` does not work with sharding, the encrypted storage engine, or
+  MMAPv1 data files.
+- To export or import a database, MongoDB must not be running using either the
+  source or destination database paths.
+- A database must be imported to all nodes in a replica set or query results
+  will be inconsistent.
+- A database cannot be imported to any node in the replica set it was exported
+  from. Collections have unique identifiers, and this would violate that uniqueness.
+- ``mtransfer`` currently only supports database files compressed with the
+  default `snappy` library.
+
+While there are some sanity checks built into the script, manipulating MongoDB
+files directly is inherently dangerous. Take care to test and backup your data.
+
+Installation
+~~~~~~~~~~~~
+
+The ``mtransfer`` script requires the
+`wiredtiger Python library <https://pypi.org/project/wiredtiger/>`__
+which is currently not installed by default with ``mtools``.
+
+The ``wiredtiger`` library can be installed via ``pip`` or built from source,
+but in either case requires:
+
+- A C compiler.
+- The ``snappy`` and ``zlib`` development packages installed.
+
+To install via ``pip`` use either of:
+
+.. code-block:: bash
+
+   pip install mtools[mtransfer]
+
+.. code-block:: bash
+
+   pip install wiredtiger
+
+If you are encountering errors using or installing the ``wiredtiger`` module
+via `pip`, you may need to `Build and install WiredTiger from source
+<http://source.wiredtiger.com/develop/build-posix.html>`__.
+
+Usage
+~~~~~
+
+.. code-block:: bash
+
+  mtransfer [-h] [--version] [--verbose] [--dbpath DBPATH]
+                    {export,import} database
+
+General Parameters
+~~~~~~~~~~~~~~~~~~
+
+Help
+----
+``-h, --help``
+   shows the help text and exits.
+
+Version
+-------
+``--version``
+   shows the version number and exits.
+
+Verbosity
+---------
+``--verbose``
+   shows extra information
+
+Commands
+~~~~~~~~
+
+``mtransfer`` reads or writes the data files in a MongoDB instance
+using the WiredTiger storage engine.
+
+Database Path
+-------------
+``--dbpath path`` the path to the MongoDB database files (defaults to
+the current working directory where the script is run).
+
+Command
+-------
+``export``
+creates a file ``mtransfer.bson`` in the specified database directory.
+This must be copied along with the other files in the directory to the
+destination server(s).
+
+``import`` reads the ``mtransfer.bson`` file from the specified database
+directory and updates MongoDB's metadata to include the database.
+
+Database
+---------
+The name of the database to export / import.  The MongoDB database name
+must match the directory name on disk for the export, and the MongoDB
+database name will be set to the directory name for the import.
+
+Example Database Transfer and Rename
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+On the origin server
+--------------------
+
+Before starting, ensure MongoDB is not running.
+
+1. Change into the origin directory and create 'olddb/mtransfer.bson'
+   with exported metadata:
+
+.. code-block:: bash
+
+  cd /from/dbpath
+  mtransfer export olddb
+
+2. Copy the database files to the destination
+
+.. code-block:: bash
+
+  rsync -av olddb destination:/to/dbpath
+
+On the destination server
+-------------------------
+
+Before starting, ensure MongoDB is not running.
+
+3. Rename the database directory (optional)
+
+.. code-block:: bash
+  
+  cd /to/dbpath
+  mv olddb newdb
+
+4. Import the database (with the new name, if renamed)
+
+.. code-block:: bash
+
+  mtransfer import newdb
+
+5. Start ``mongod`` and confirm the transferred database
+   is now available.
+
+Disclaimer
+~~~~~~~~~~
+
+This software is not supported by `MongoDB, Inc. <https://www.mongodb.com>`__
+under any of their commercial support subscriptions or otherwise. Any usage of
+mtools is at your own risk. Bug reports, feature requests and questions can be
+posted in the `Issues
+</~https://github.com/rueckstiess/mtools/issues?state=open>`__ section on GitHub.