Not getting 10,000 unique package names

[nazanaza2970]

I am using the raw.json file to get the list of package names. Here is the code:

with open("jplabfiles/raw.json", encoding='utf-8') as f:
    data = json.load(f)
package_names = []
for item in data:
    package_names.append(item['name'])
print(len(set(package_names)))

I am getting only 5247 packages. Please let me know if I am doing something wrong.

[LeoDog896]

Interesting.. that script is right; it may be an NPM API problem; I'll see what I can do.

[Zhengqbbb]

1. Looks like the pkg list have many repetition.
2. And it seems that I increase the number of requests, the number of pkg is always 5247.

---

🫠🫠🫠

[LeoDog896]

Since this issue got bumped again, I did a more formal investigation into the NPM registry, and it seems as if the registry itself is reduplicating package names after a certain point. I've sent them out an email, and will report back here when they reply.

[LeoDog896]

They have not responded via email - I've filed a PR/issue? on the registry repository, but I'm afraid this will probably not bring attention. If there's a way to get the entire NPM info dump, that would be great.

I'll try using the Replicate API later (maybe I can also get a bigger list of packages!)