Add new option to run just new jobs or jobs without history

[nille02]

I have the issue that i have some url.yaml files with hundreds to thousands of jobs. A full run can take hours and i do that in general just once a month or even less.
Before that option i checked for the new ids and run them manual. but it was quite the chore and i looked how i can avoid that.

My only issue is that get_history_data() is quite slow but i guess sqllite is the reason?

[thp]

See comments.

[nille02]

I added the redis version of has_history_data() to it but it just wraps around get_history_data() since redis is already fast.

For this a new cache.db was used and a redis server that runs on a debian stable vm.

Redis          Benchmark: 864 Jobs took 0:00:00.680740
sqllite        Benchmark: 864 Jobs took 0:00:30.835610
sqllite Cached Benchmark: 864 Jobs took 0:00:00.322071

The time is just the loop for all jobs over has/get_history_data()

[thp]

See comments.

[thp]

Works for me. Please also add a changelog entry to the PR.

[nille02]

Just a entry in the CHANGELOG.md? If yes its done.

And everyone i hope you all had a happy new year.

[thp]

Looks good, thanks for the updates!

[thp]

@nille02 Can you fix up the last remaining code style issues?

----------------------------- Captured stdout call -----------------------------
/home/runner/work/urlwatch/urlwatch/lib/urlwatch/command.py:151:36: E226 missing whitespace around arithmetic operator
/home/runner/work/urlwatch/urlwatch/lib/urlwatch/storage.py:608:89: E231 missing whitespace after ','
/home/runner/work/urlwatch/urlwatch/lib/urlwatch/storage.py:609:115: W504 line break after binary operator
/home/runner/work/urlwatch/urlwatch/lib/urlwatch/storage.py:610:121: E261 at least two spaces before inline comment
=========================== short test summary info ============================

After that, this is ready to merge :)

[nille02]

Should be fixed now

[thp]

There's still 3 more:

/home/runner/work/urlwatch/urlwatch/lib/urlwatch/storage.py:608:89: E231 missing whitespace after ','
/home/runner/work/urlwatch/urlwatch/lib/urlwatch/storage.py:608:108: W291 trailing whitespace
/home/runner/work/urlwatch/urlwatch/lib/urlwatch/storage.py:609:114: W291 trailing whitespace

(you can run this locally to verify it's fixed, no need to wait for CI -- use pytest -v after installing dependencies, see /~https://github.com/thp/urlwatch/blob/master/.github/workflows/unit-tests.yml)

[nille02]

Now it should be done.

I did try pytest but i got some different error and i guess its due to windows. So had to skip the documentation test but pep8 was happy now.

[thp]

Thanks for working on this and seeing it through! Merged :)