Google Analytics search query support

[mhostetter]

I am receiving no search queries in Google Analytics from my webpage using sphinx-immaterial. Full disclosure, I wasn't receiving them using sphinx_rtd_theme either. I was, however, able to see the search terms on the readthedocs.org Admin tab. I don't believe I get any search terms in the Admin tab with sphinx-immaterial.

My question still stands, though. It seems that it should be possible to receive Google Analytics search results as is discussed here with sphinx-immaterial.

I believe I'm configuring Google Analytics the correct way. And I can see that the JS is loaded and is emitting page references (which I can see in my GA dashboard).

html_theme_options = {
    "google_analytics": ["G-***********", "auto"],
}

However, the webpage is not emitting anything when search terms are typed or selected. I did some investigating and comparison with what the Material for Mkdocs webpage does. I found the Material for Mkdocs website does invoke analytics.js when searching and sphinx-immaterial does not, see below.

Thoughts on this? Can we add some JS to invoke the analytics.js after typing a search query (begin scrolling through results) or when clicking on a search result?

Material for Mkdocs

On https://squidfunk.github.io/mkdocs-material/, I typed the search term and nothing was emitted on the network. Once I scrolled down onto the first result, or clicked somewhere, the analytics.js was triggered (see the last line).

Examining the last packet, we see the ?q=a custom search query.

Google Analytics uses the q to parse the search result.

Sphinx Immaterial

I noticed no such ?q= emitted from my website https://galois.readthedocs.io/en/v0.0.25/index.html using sphinx-immaterial.

When arriving on a page, there is a analytics.js trigger just saying where we are.

However, after typing a search query, scrolling down through the results, and then clicking a page there is no additional analytics.js trigger with a ?q= included.

[mhostetter]

Edit: It turns out I am getting search terms from the sphinx_rtd_theme versions of the webpage. https://galois.readthedocs.io/en/v0.0.24/index.html

I confirmed it with the "inspect" network tool

and with the realtime results from the GA dashboard

I think it would be valuable if sphinx-immaterial could emit a ?q= tag when searching. 👍

[jbms]

The normal sphinx search uses a separate webpage (search.html) with a ?q= query parameter to display search results and therefore works by default I suppose.

Sphinx immaterial has a custom search implementation designed to be similar to mkdocs-material that displays results directly on the same page in a drop-down menu and the query string ?q= parameter is not added, just like mkdocs-material. Perhaps you can see what is causing the q= events to be emitted in mkdocs-material using browser dev tools.

[mhostetter]

The normal sphinx search uses a separate webpage (search.html) with a ?q= query parameter to display search results and therefore works by default I suppose.

Yes, that is my understanding too.

Perhaps you can see what is causing the q= events to be emitted in mkdocs-material using browser dev tools.

Yes, I will try. I am very much out of my depth when it comes to web dev, though. I'll report back any successes or failures.

[mhostetter]

I discovered GA search support was added in squidfunk/mkdocs-material@c829555.

I believe this functionality currently corresponds to this code:

sphinx-immaterial/src/partials/integrations/analytics/google.html

Lines 40 to 46 in 6ceccff

	if (document.forms.search) {
	var query = document.forms.search.query
	query.addEventListener("blur", function() {
	if (this.value)
	gtag("event", "search", { search_term: this.value })
	})
	}

@jbms I know you previously said (#34 (comment)):

I have made a lot of changes to the search code, so if there are any upstream changes to the search code, it may take some significant effort to resolve.

Where can I find the files that diverge from upstream regarding search? I'll try to determine if something is different that could be causing the issue.

[mhostetter]

I believe this file in upstream is important too. I cannot find it in our copy.

/~https://github.com/squidfunk/mkdocs-material/blob/f9044530e6eb7467ed86960363967aa53bbcb7dc/material/partials/integrations/analytics/google.html

In fact, it seems the entire material/ directory is not included in our version. Is this intentional?

[jbms]

The material/ directory just contains automatically generated files derived from the other source files in the repository and its dependencies. In this theme we don't check those generated files into the repository; instead they are generated automatically as part of the build process.

The material/partials/integrations/analytics/google.html file is just derived from the src/partials/integrations/analytics/google.html file that you already found.

To see the files that diverge you can do a diff against the src/ sub-directory.

[mhostetter]

I would like to (blindly) try things to resolve this, however I'm struggling "developing" with this theme.

@jbms and @2bndy5 how do you make and test changes to this theme?

Normally, I will install a Python package in "editable" mode pip3 install -e . so that immediate changes in source are seen upon next import. However, I get a permission error when I try that with sphinx-immaterial. Currently, I'm making a change and then running pip3 install ., which takes minutes to build the wheel. Any help/advice would be appreciated.

Editable install errors

matt@DESKTOP-I79NAGP:/mnt/c/Users/matth/repos/sphinx-immaterial$ pip3 install -e .
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///mnt/c/Users/matth/repos/sphinx-immaterial
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: markupsafe in /usr/lib/python3/dist-packages (from sphinx-immaterial==0.2.post1.dev12) (1.1.0)
Requirement already satisfied: sphinx>=4.0 in /home/matt/.local/lib/python3.8/site-packages (from sphinx-immaterial==0.2.post1.dev12) (4.4.0)
Requirement already satisfied: sphinxcontrib-qthelp in /home/matt/.local/lib/python3.8/site-packages (from sphinx>=4.0->sphinx-immaterial==0.2.post1.dev12) (1.0.3)
Requirement already satisfied: Pygments>=2.0 in /home/matt/.local/lib/python3.8/site-packages (from sphinx>=4.0->sphinx-immaterial==0.2.post1.dev12) (2.10.0)
Requirement already satisfied: Jinja2>=2.3 in /usr/lib/python3/dist-packages (from sphinx>=4.0->sphinx-immaterial==0.2.post1.dev12) (2.10.1)
Requirement already satisfied: sphinxcontrib-devhelp in /home/matt/.local/lib/python3.8/site-packages (from sphinx>=4.0->sphinx-immaterial==0.2.post1.dev12) (1.0.2)
Requirement already satisfied: sphinxcontrib-serializinghtml>=1.1.5 in /home/matt/.local/lib/python3.8/site-packages (from sphinx>=4.0->sphinx-immaterial==0.2.post1.dev12) (1.1.5)
Requirement already satisfied: requests>=2.5.0 in /usr/lib/python3/dist-packages (from sphinx>=4.0->sphinx-immaterial==0.2.post1.dev12) (2.22.0)
Requirement already satisfied: importlib-metadata>=4.4 in /home/matt/.local/lib/python3.8/site-packages (from sphinx>=4.0->sphinx-immaterial==0.2.post1.dev12) (4.11.1)
Requirement already satisfied: imagesize in /usr/lib/python3/dist-packages (from sphinx>=4.0->sphinx-immaterial==0.2.post1.dev12) (1.2.0)
Requirement already satisfied: packaging in /usr/lib/python3/dist-packages (from sphinx>=4.0->sphinx-immaterial==0.2.post1.dev12) (20.3)
Requirement already satisfied: alabaster<0.8,>=0.7 in /usr/lib/python3/dist-packages (from sphinx>=4.0->sphinx-immaterial==0.2.post1.dev12) (0.7.8)
Requirement already satisfied: babel>=1.3 in /usr/lib/python3/dist-packages (from sphinx>=4.0->sphinx-immaterial==0.2.post1.dev12) (2.6.0)
Requirement already satisfied: sphinxcontrib-htmlhelp>=2.0.0 in /home/matt/.local/lib/python3.8/site-packages (from sphinx>=4.0->sphinx-immaterial==0.2.post1.dev12) (2.0.0)      
Requirement already satisfied: docutils<0.18,>=0.14 in /usr/lib/python3/dist-packages (from sphinx>=4.0->sphinx-immaterial==0.2.post1.dev12) (0.16)
Requirement already satisfied: snowballstemmer>=1.1 in /usr/lib/python3/dist-packages (from sphinx>=4.0->sphinx-immaterial==0.2.post1.dev12) (2.0.0)
Requirement already satisfied: sphinxcontrib-jsmath in /home/matt/.local/lib/python3.8/site-packages (from sphinx>=4.0->sphinx-immaterial==0.2.post1.dev12) (1.0.1)
Requirement already satisfied: sphinxcontrib-applehelp in /home/matt/.local/lib/python3.8/site-packages (from sphinx>=4.0->sphinx-immaterial==0.2.post1.dev12) (1.0.2)
Requirement already satisfied: zipp>=0.5 in /usr/lib/python3/dist-packages (from importlib-metadata>=4.4->sphinx>=4.0->sphinx-immaterial==0.2.post1.dev12) (1.0.0)
Installing collected packages: sphinx-immaterial
  Running setup.py develop for sphinx-immaterial
    error: subprocess-exited-with-error

    × python setup.py develop did not run successfully.
    │ exit code: 1
    ╰─> [32 lines of output]
        running develop
        /tmp/pip-build-env-5ch1rbpo/overlay/lib/python3.8/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
          warnings.warn(
        WARNING: The user site-packages directory is disabled.
        /tmp/pip-build-env-5ch1rbpo/overlay/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build 
and pip and other standards-based tools.
          warnings.warn(
        error: can't create or remove files in install directory
       
        The following error occurred while trying to add or remove files in the
        installation directory:
       
            [Errno 13] Permission denied: '/usr/lib/python3.8/site-packages/test-easy-install-857.write-test'
       
        The installation directory you specified (via --install-dir, --prefix, or
        the distutils default setting) was:
       
            /usr/lib/python3.8/site-packages/
       
        Perhaps your account does not have write access to this directory?  If the
        installation directory is a system-owned directory, you may need to sign in
        as the administrator or "root" account.  If you do not have administrative
        access to this machine, you may wish to choose a different installation
        directory, preferably one that is listed in your PYTHONPATH environment
        variable.
       
        For information on other options, you may wish to consult the
        documentation at:
       
          https://setuptools.pypa.io/en/latest/deprecated/easy_install.html
       
        Please make the appropriate changes for your system and try again.
       
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> [32 lines of output]
    running develop
    /tmp/pip-build-env-5ch1rbpo/overlay/lib/python3.8/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    WARNING: The user site-packages directory is disabled.
    /tmp/pip-build-env-5ch1rbpo/overlay/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and 
pip and other standards-based tools.
      warnings.warn(
    error: can't create or remove files in install directory
   
    The following error occurred while trying to add or remove files in the
    installation directory:
   
        [Errno 13] Permission denied: '/usr/lib/python3.8/site-packages/test-easy-install-857.write-test'
   
    The installation directory you specified (via --install-dir, --prefix, or
    the distutils default setting) was:
   
        /usr/lib/python3.8/site-packages/
   
    Perhaps your account does not have write access to this directory?  If the
    installation directory is a system-owned directory, you may need to sign in
    as the administrator or "root" account.  If you do not have administrative
    access to this machine, you may wish to choose a different installation
    directory, preferably one that is listed in your PYTHONPATH environment
    variable.
   
    For information on other options, you may wish to consult the
    documentation at:
   
      https://setuptools.pypa.io/en/latest/deprecated/easy_install.html
   
    Please make the appropriate changes for your system and try again.
   
    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

[jbms]

I'm not sure about the permission issues --- maybe there is something we need to fix --- but I'd recommend using a virtualenv, which I think may fix the permission issues.

In addition to using pip install -e ., after doing that, you can run: npm run start. That will incrementally rebuild after any changes to the CSS or Javascript.

You will still need to re-run sphinx-build -a after any changes though to generate your documentation.

[2bndy5]

My workflow has become

1. Build in WSL Ubuntu bash (from windows)

   python setup.py bdist_wheel
   

2. Install the wheel on windows

   pip install dist/sphinx-immaterial.0.x.x-devx.postx.whl
   

3. If I want to make changes without rebuilding, I do it in the site-packages install folder (no -e needed).

All of this I do from within python venv(on Linux and Windows).

python -m venv env
source env/bin/activate

Command is a bit different to activate venv on Windows, but Windows terminal runs faster than Ubuntu bash on Windows. The theme has to be packaged from Linux though, otherwise the built CSS and JS bundles are named incorrectly.

[mhostetter]

Thank you both. I will give that a try.