Skip to content
This repository has been archived by the owner on Jan 22, 2024. It is now read-only.

Probable CDN problems! #1624

Closed
hqnicolas opened this issue Mar 30, 2022 · 18 comments
Closed

Probable CDN problems! #1624

hqnicolas opened this issue Mar 30, 2022 · 18 comments

Comments

@hqnicolas
Copy link

#7 13.24 E: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/Packages 404 Not Found
#7 13.24 E: Failed to fetch https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/Packages 404 Not Found

@bermeitinger-b
Copy link

The same happens for the ubuntu2004 path: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/

Example:

E: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/./
cuda-compiler-11-4_11.4.4-1_amd64.deb  404  Not Found [IP: 152.199.20.126 443]

@klueska
Copy link
Contributor

klueska commented Mar 30, 2022

This is a known issue and internal teams at NVIDIA are working to resolve it.

@hqnicolas
Copy link
Author

please someone Call to Lapsus$ Hackers if they have a backup from these folders

@Climbgunks
Copy link

piling on w/ 18.04 -- assuming it's the same issue:

The repository 'https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Release' does not have a Release file.

@gaitanignacio
Copy link

I'm getting the same issue right now on a docker build from an old Dockerfile that I saw working properly before!

Basically:

FROM 763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:1.6.0-gpu-py36-cu110-ubuntu18.04

RUN apt-get update
E: The repository 'https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

@lorriexingfang86
Copy link

I m getting the same error when I do:
apt-get update

for the image:
tensorflow/tensorflow:2.6.1-gpu

@rrlamichhane
Copy link

I'm getting an error as well, it's different than most that have been posted here. This is the error I'm getting, hopefully this will help the debuggers at nvidia resolve it sooner:

Reading package lists...
18:19:34
  W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY F60F4B3D7FA2AF80
18:19:34
  E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Release' is not signed.
18:19:35
  The command '/bin/sh -c apt update && apt install -y cuda-compat-11-4' returned a non-zero code: 100

@santiagoolivar2017
Copy link

Same here while executing
RUN apt-get update --fix-missing apt-get install

I am getting the next error

Reading package lists...
E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1710/x86_64  Release' does not have a Release file.
E: The repository 'https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  Release' does not have a Release file.
The command '/bin/sh -c apt-get update --fix-missing apt-get install returned a non-zero code: 100

@ryan-williams
Copy link

Minimal bash one-liner test, since I haven't seen anyone post it exactly:

docker run --rm nvidia/cuda:11.4.0-base apt-get update
Get:1 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Ign:3 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  InRelease
Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:5 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal/multiverse amd64 Packages [177 kB]
Ign:7 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64  InRelease
Get:8 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Release [696 B]
Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1275 kB]
Get:10 http://archive.ubuntu.com/ubuntu focal/restricted amd64 Packages [33.4 kB]
Get:11 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]
Err:12 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64  Release
  404  Not Found [IP: 152.195.19.142 443]
Get:13 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Release.gpg [836 B]
Get:14 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1160 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1149 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [30.3 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2124 kB]
Get:18 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [51.2 kB]
Get:19 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [26.0 kB]
Get:20 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64  Packages [629 kB]
Get:21 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1708 kB]
Get:22 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [863 kB]
Get:23 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1086 kB]
Get:24 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [25.8 kB]
Reading package lists...
E: The repository 'https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64  Release' does not have a Release file.

@raa2463
Copy link

raa2463 commented Mar 31, 2022

Facing similar issue
E: The repository 'https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64 Release' does not have a Release file.

@MusHusKat
Copy link

Similar issue:

[ERROR] 31/03/2022, 11:16:29 am: E: The repository 'https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Release' no longer has a Release file.

@tgaddair
Copy link

For those looking for a workaround, in case you're just using one of these images as a parent and don't need to install any additional CUDA libraries, you can run the following before apt-get update:

RUN rm /etc/apt/sources.list.d/cuda.list && rm /etc/apt/sources.list.d/nvidia-ml.list

Depending on how you're building your image, you may need to run these commands with sudo.

This was a suggestion from the Nvidia forums, where this issue is also being tracked:

https://forums.developer.nvidia.com/t/error-apt-get-updating-from-nvidia-cuda11-2-1-base-ubuntu20-04/209836/8

@tuong-olli
Copy link

For those looking for a workaround, in case you're just using one of these images as a parent and don't need to install any additional CUDA libraries, you can run the following before apt-get update:

RUN rm /etc/apt/sources.list.d/cuda.list && rm /etc/apt/sources.list.d/nvidia-ml.list

Depending on how you're building your image, you may need to run these commands with sudo.

This was a suggestion from the Nvidia forums, where this issue is also being tracked:

https://forums.developer.nvidia.com/t/error-apt-get-updating-from-nvidia-cuda11-2-1-base-ubuntu20-04/209836/8

It works fine for me. Thanks you!

@jimsparkman
Copy link

Appears repos are back online again.

@santiagoolivar2017
Copy link

Confirmed, working again for me!

@lorriexingfang86
Copy link

confirmed here too. it works now!

@elezar
Copy link
Member

elezar commented Apr 4, 2022

Please see the link from #1009 (comment)

Note that this may be due to a CDN issue with the CUDA downloads repositories and are not specific to NVIDIA docker. Closing this issue.

@elezar elezar closed this as completed Apr 4, 2022
@BITWN
Copy link

BITWN commented Jul 12, 2022

confirmed here too, run the following command before apt-get update works for me!
RUN rm /etc/apt/sources.list.d/cuda.list

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests