refactor: add only new artifacts #48
Conversation
| @@ -221,6 +227,27 @@ func (c *Crawler) Visit(ctx context.Context, url string) error { | |||
| return nil | |||
| } | |||
|
|
|||
| // To avoid a large number of requests to the server, we should skip already saved artifacts (if the start date is specified). | |||
| // P.S. We do not need to check for updates, since artifacts are immutable | |||
| // see https://central.sonatype.org/publish/requirements/immutability | |||
There was a problem hiding this comment.
The document seems relevant to the sonatype repository. Is this repository also immutable?
https://mvnrepository.com/repos/central
There was a problem hiding this comment.
I didn't find official info about that. But i think rules should be same.
But i saw answer about that - https://stackoverflow.com/questions/40739939/dropping-a-release-from-public-maven-central
Also indirect evidence is this answer (they had the same sha1 for several artifacts):
instead of changing the file - they release a new version
|
I remember we created a small script to compare two databases, but didn't find it... |
I will do that and write to you |
|
I have bad news... I found problem with this idea - it looks like this date is not date it was added to maven central.
So i close this PR. I have another idea. |
Description
Recently, we’ve been encountering
too many requestserror more frequently (/~https://github.com/aquasecurity/trivy-java-db/actions/runs/12458901714).Increasing the number of parallel goroutines or reducing the delay between requests doesn't resolve the issue.
Therefore, this PR adds functionality to add only new artifacts to the database. The need for adding is determined using the
updateAtfield from thecache/db/metadata.jsonfile (the date minus one day to account for the time needed to build the database).If the
cache/dbdir is empty, the crawler will save all indexes.Test runs:
2024-10-13- /~https://github.com/DmitriyLewen/trivy-java-db/actions/runs/12465006507/job/347900622522024-12-22- /~https://github.com/DmitriyLewen/trivy-java-db/actions/runs/12465457228/job/34791277401