[AIRFLOW-3811][3/3] Add automatic generation of API Reference

[mik-laj]

Make sure you have checked all steps below.

Jira

https://issues.apache.org/jira/browse/AIRFLOW-3811?filter=-1

Description

This is one of the PR series aimed to add automatically API Reference.

• [AIRFLOW-XXX][1/3] Syntax docs improvements - [AIRFLOW-XXX][1/3] Syntax docs improvements #4789
• [AIRFLOW-3968][2/3] Refactor base GCP hook - [AIRFLOW-3968][2/3] Refactor base GCP hook #4790
• [AIRFLOW-3811][3/3] Add automatic generation of API Reference - [AIRFLOW-3811][3/3] Add automatic generation of API Reference #4788

I would ask you to accept changes in order.

Documentation preview with all changes is available:
http://neighborly-sun.surge.sh/autoapi/index.html

The library of autoapi was used to build it.

Tests

No applicable

Commits

• My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.
  • All the public functions and the classes in the PR contain docstrings that explain what it does

Code Quality

• Passes flake8

[potiuk]

This will be super helpful to get the API reference fully automatically generated

[Fokko]

Fully agree with @potiuk, this was overdue for a long time!

Awesome work @mik-laj

[Fokko]

The CI tells us Currently, 12 warnings found. Would it also be possible to get these in the stdout?

[mik-laj]

@Fokko This is fixed, but in another PR from series. I have divided all the changes into a few PRs to make easiter to review of changes and to make history easier to read.

[mik-laj]

I prepared a build with all changes: https://travis-ci.org/PolideaInternal/airflow/jobs/499332426

[potiuk]

With my naming changes it should be fine

[mik-laj]

The required changes appeared on the master. Documentations is built correctly.

Latest preview: http://puny-spark.surge.sh/_autoapi/index.html

PTAL @ashb @Fokko @XD-DENG @feng-tao @cixuuz

[potiuk]

It would be great to get it merged :). I think it's quite a good one to decrease friction of using Airflow by first time developers. I remember how I missed it the first time I started to use the project. Currently I am pretty OK in looking at the code, but having reference fully automatically genereated from Docstrings - especially that Airflow is so well documented - is a big +

[ashb]

Nice overall - and a much better approach!

Couple of comments/questions

[Fokko]

Looking good @mik-laj

Can you fix the merge conflict?

[mik-laj]

I added this fragment in this PR.

[mik-laj]

@Fokko i updated a PR.

[Fokko]

Thanks @mik-laj

Lets wait for the CI

[mik-laj]

@Fokko Travis have unreleased errors on one job. Can you run it again?

----------------------------------------------------------------------
2) ERROR: test_integration_run_dag_with_scheduler_failure (tests.contrib.minikube.test_kubernetes_executor.KubernetesExecutorTest)
======================================================================
   --------------------- >> end captured logging << ---------------------
   urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): 10.20.5.63:30809
   airflow.logging_config: DEBUG: Unable to load custom logging, using default config instead
   airflow.utils.log.logging_mixin.LoggingMixin: DEBUG: Cannot import  due to  doesn't look like a module path
   root: INFO: Generating grammar tables from /usr/lib/python3.5/lib2to3/PatternGrammar.txt
   root: INFO: Generating grammar tables from /usr/lib/python3.5/lib2to3/Grammar.txt
   airflow.settings: INFO: Configured default timezone <Timezone [UTC]>
   airflow.utils.log.logging_mixin.LoggingMixin: INFO: Creating new FAB webserver config file in: /home/airflow/webserver_config.py
   airflow.utils.log.logging_mixin.LoggingMixin: INFO: Reading the config from /home/airflow/airflow.cfg
   airflow.utils.log.logging_mixin.LoggingMixin: INFO: Creating new Airflow config file in: /home/airflow/airflow.cfg
   -------------------- >> begin captured logging << --------------------
   --------------------- >> end captured stdout << ----------------------

   [2019-03-26 13:38:20,174] {connectionpool.py:205} DEBUG - Starting new HTTP connection (1): 10.20.5.63:30809
   -------------------- >> begin captured stdout << ---------------------

   ConnectionError: HTTPConnectionPool(host='10.20.5.63', port=30809): Max retries exceeded with url: /api/experimental/dags/example_kubernetes_executor_config/paused/false
 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3ba8bc61d0>: Failed to establish a new connection: [Errno 111] Connection refused',))
      raise ConnectionError(e, request=request)
    .tox/py35-backend_postgres-env_kubernetes/lib/python3.5/site-packages/requests/adapters.py line 516 in send
      r = adapter.send(request, **kwargs)
    .tox/py35-backend_postgres-env_kubernetes/lib/python3.5/site-packages/requests/sessions.py line 646 in send
      resp = self.send(prep, **send_kwargs)
    .tox/py35-backend_postgres-env_kubernetes/lib/python3.5/site-packages/requests/sessions.py line 533 in request
      return session.request(method=method, url=url, **kwargs)
    .tox/py35-backend_postgres-env_kubernetes/lib/python3.5/site-packages/requests/api.py line 60 in request
      return request('get', url, params=params, **kwargs)
    .tox/py35-backend_postgres-env_kubernetes/lib/python3.5/site-packages/requests/api.py line 75 in get
      'dags/{dag_id}/paused/false'.format(host=host, dag_id=dag_id)
    tests/contrib/minikube/test_kubernetes_executor.py line 134 in start_dag
      result_json = self.start_dag(dag_id=dag_id, host=host)
    tests/contrib/minikube/test_kubernetes_executor.py line 173 in test_integration_run_dag
   Traceback (most recent call last):
----------------------------------------------------------------------
1) ERROR: test_integration_run_dag (tests.contrib.minikube.test_kubernetes_executor.KubernetesExecutorTest)
======================================================================

[codecov-io]

Codecov Report

Merging #4788 into master will increase coverage by 0.07%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #4788      +/-   ##
==========================================
+ Coverage   75.67%   75.75%   +0.07%     
==========================================
  Files         458      458              
  Lines       29856    31072    +1216     
==========================================
+ Hits        22594    23539     +945     
- Misses       7262     7533     +271

Impacted Files	Coverage Δ
airflow/operators/bash_operator.py	91.22% <ø> (ø)	⬆️
airflow/contrib/hooks/gcp_transfer_hook.py	95.55% <ø> (ø)	⬆️
airflow/operators/python_operator.py	95.8% <ø> (ø)	⬆️
...rflow/contrib/operators/kubernetes_pod_operator.py	98.55% <ø> (-0.07%)	⬇️
airflow/hooks/hdfs_hook.py	92.5% <100%> (ø)	⬆️
airflow/models/__init__.py	89.95% <0%> (-3%)	⬇️
airflow/contrib/operators/aws_athena_operator.py	67.69% <0%> (-2.77%)	⬇️
airflow/utils/db.py	88.28% <0%> (-2.02%)	⬇️
airflow/logging_config.py	96.55% <0%> (-1.01%)	⬇️
airflow/configuration.py	94.07% <0%> (-0.33%)	⬇️
... and 16 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5c49ae3...eaecc3b. Read the comment docs.

[ashb]

@mik-laj Trying to pull this in to 1.10.3 and I'm getting some odd warnings (which we treat as errors):

/Users/ash/code/python/incubator-airflow/docs/_api/airflow/www_rbac/utils/index.rst:86: WARNING: Unexpected indentation.
/Users/ash/code/python/incubator-airflow/docs/_api/airflow/www_rbac/utils/index.rst:87: WARNING: Block quote ends without a blank line; unexpected unindent.

I turned on keep files, and bigquery_hook/index.rst:276 maps to this line in my source

airflow/airflow/www/utils.py

Lines 119 to 120 in da224b9

	client and server. It also allows to make a bookmark on a specific paging state.
	:param current_page:

Comparing to master:

/~https://github.com/apache/airflow/blob/master/airflow/www/utils.py#L91-L92

The issue here is a the lack of a new line - I'm just not sure how this didn't fail CI as a result?

Edit: Ah, becaue _api/airflow/www is ignored right now. I'll just need to ignore www_rbac too.

[zhongjiajie]

Awesome, Thanks @mik-laj .

[fifar]

This PR will try to import snakebite here which is not compatible with python3. I encountered this error

    from airflow.operators.sensors import ExternalTaskSensor
  File "~/env/lib/python3.6/site-packages/airflow/operators/sensors.py", line 35, in <module>
    from airflow.sensors.hdfs_sensor import HdfsSensor as HdfsSensorImp
  File "~/env/lib/python3.6/site-packages/airflow/sensors/hdfs_sensor.py", line 25, in <module>
    from airflow.hooks.hdfs_hook import HDFSHook
  File "~/env/lib/python3.6/site-packages/airflow/hooks/hdfs_hook.py", line 24, in <module>
    from snakebite.client import Client, HAClient, Namenode, AutoConfigClient
  File "~/env/lib/python3.6/site-packages/snakebite/client.py", line 1473
    baseTime = min(time * (1L << retries), cap);
                            ^
SyntaxError: invalid syntax

It worked when using Airflow 1.10.2

@mik-laj