Result uploads are failing due to import error (urllib3 v2.0)

[LoopedBard3]

Most of the runs have been mitigated such that they are succeeding, however some are still failing:

• OSX Runtime runs
• OSX Perf runs

Other action items:

• Backport in perf repo
• Backport in runtime branches - waiting to see if we are able to fix the underlying issue by updating ssl on our machines first

Error:

  File "C:\Users\runner\AppData\Roaming\Python\Python37\site-packages\azure\core\pipeline\transport\__init__.py", line 73, in __getattr__
    from ._requests_basic import RequestsTransport
  File "C:\Users\runner\AppData\Roaming\Python\Python37\site-packages\azure\core\pipeline\transport\_requests_basic.py", line 28, in <module>
    import urllib3
  File "C:\Users\runner\AppData\Roaming\Python\Python37\site-packages\urllib3\__init__.py", line 39, in <module>
    "urllib3 v2.0 only supports OpenSSL 1.1.1+, currently "
ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.1.0j  20 Nov 2018. See: /~https://github.com/urllib3/urllib3/issues/2168

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\dotnetbuild\work\AD4809D9\w\A32708FF\e\performance\scripts\benchmarks_ci.py", line 330, in <module>
    __main(sys.argv[1:])
  File "C:\dotnetbuild\work\AD4809D9\w\A32708FF\e\performance\scripts\benchmarks_ci.py", line 318, in __main
    import upload
  File "C:\dotnetbuild\work\AD4809D9\w\A32708FF\e\performance\scripts\upload.py", line 1, in <module>
    from azure.storage.blob import BlobClient, ContentSettings
  File "C:\Users\runner\AppData\Roaming\Python\Python37\site-packages\azure\storage\blob\__init__.py", line 11, in <module>
    from ._blob_client import BlobClient
  File "C:\Users\runner\AppData\Roaming\Python\Python37\site-packages\azure\storage\blob\_blob_client.py", line 23, in <module>
    from ._shared.base_client import StorageAccountHostsMixin, parse_connection_str, parse_query
  File "C:\Users\runner\AppData\Roaming\Python\Python37\site-packages\azure\storage\blob\_shared\base_client.py", line 31, in <module>
    from azure.core.pipeline.transport import RequestsTransport, HttpTransport
  File "C:\Users\runner\AppData\Roaming\Python\Python37\site-packages\azure\core\pipeline\transport\__init__.py", line 77, in __getattr__
    raise ImportError("requests package is not installed")
ImportError: requests package is not installed

[LoopedBard3]

There are still some failures due to this occuring in the iOS runs on both runtime and performance repo runs. Using this issue to track the rest of the fallout failures due to the urllib change.

[kotlarmilos]

• Backport in runtime branches - waiting to see if we are able to fix the underlying issue by updating ssl on our

@LoopedBard3 Any luck with updating SSL on the machines?

[LoopedBard3]

@kotlarmilos I am currently in chat with FR to find the right way to go about updating the ssl certificates. However, all queues should be working if rebased off the latest performance repo.

[kotlarmilos]

Thanks! It seems to be resolved. I've noticed several instances of unscheduled jobs for iOS performance measurements. Here's an example: https://helix.dot.net/api/jobs/2ef88556-41b2-490b-b60b-95cbeeb8d4a3/workitems/SOD%20-%20iOS%20HelloWorld%20Native%20AOT%20.app%20Size?api-version=2019-06-17

I suspect that the reason for the unscheduled jobs is Helix queue osx.1015.amd64.iphone.perf is set for estimated removal date of 2023-05-31. Here is a PR that eventually solves the issue with unscheduled jobs: dotnet/runtime#86574

[LoopedBard3]

After digging in more I have figured out that the estimated removal is not keeping the helix jobs from being run, just clobbering the helix results, as you seem to have figured out. I made a PR to change this behavior to match the windows machine behavior dotnet/runtime#86729. For the unscheduled job, you are most likely missing your access token, which is added by appending the url with '&access_token={token_here}'. The linked job shows waiting so I am not sure what is happening with that, but if you are able to track down other links like that for more recent runs, it should work.

[kotlarmilos]

After digging in more I have figured out that the estimated removal is not keeping the helix jobs from being run, just clobbering the helix results, as you seem to have figured out. I made a PR to change this behavior to match the windows machine behavior dotnet/runtime#86729.

Thank you.

For the unscheduled job, you are most likely missing your access token, which is added by appending the url with '&access_token={token_here}'. The linked job shows waiting so I am not sure what is happening with that, but if you are able to track down other links like that for more recent runs, it should work.

Probably, I will check in the explorer if there are measurements.

[kotlarmilos]

@LoopedBard3 I confirm that the issue has been fixed. Besides the unscheduled status, the measurements have been recorded.

Once you check the following feel free to close this issue.

[ ] Backport in runtime branches - waiting to see if we are able to fix the underlying issue by updating ssl on our machines first