esgf-pyclient review

[agstephens]

• Clone esgf-pyclient into a sandbox environment
• Run the unit tests and check if any need fixing
• Make updates to the requests-cache library interface as specified here: Error with requests_cache dependency #71 (comment)
• Test the above with different versions of the requests-cache library to ensure they work with the old and new interface
• Update the changelog/history
• Create PR and merge to master

[alaniwi]

current failures (path to installation munged here):

_______________________________________________ TestConnection.test_get_shard_list ________________________________________________
Traceback (most recent call last):
  File "/path/to/esgf-pyclient/tests/test_connection.py", line 46, in test_get_shard_list
    assert len(shards['esgf-index1.ceda.ac.uk']) > 1
AssertionError: assert 1 > 1
 +  where 1 = len([('8983', 'solr/datasets')])
____________________________________________________ TestContext.test_distrib _____________________________________________________
Traceback (most recent call last):
  File "/path/to/esgf-pyclient/tests/test_context.py", line 107, in test_distrib
    assert count1 < count2
AssertionError: assert 53725 < 53725
_______________________________________________ TestContext.test_distrib_with_cache _______________________________________________
Traceback (most recent call last):
  File "/path/to/esgf-pyclient/tests/test_context.py", line 120, in test_distrib_with_cache
    assert count1 < count2
AssertionError: assert 53725 < 53725
____________________________________________________ TestContext.test_replica _____________________________________________________
Traceback (most recent call last):
  File "/path/to/esgf-pyclient/tests/test_context.py", line 177, in test_replica
    assert context.hit_count > 2
AssertionError: assert 1 > 2
 +  where 1 = <pyesgf.search.context.DatasetSearchContext object at 0x7f7dbb0fdd30>.hit_count
______________________________________ TestResults.test_batch_size_has_no_impact_on_results _______________________________________
Traceback (most recent call last):
  File "/path/to/esgf-pyclient/tests/test_results.py", line 256, in test_batch_size_has_no_impact_on_results
    assert len(ids_batch_size_50) == len(ids_batch_size_100)
AssertionError: assert 50 == 100
 +  where 50 = len([<pyesgf.search.results.DatasetResult object at 0x7f7dbb02fbe0>, <pyesgf.search.results.DatasetResult object at 0x7f7d...h.results.DatasetResult object at 0x7f7dbb00af60>, <pyesgf.search.results.DatasetResult object at 0x7f7dbb02f550>, ...])
 +  and   100 = len([<pyesgf.search.results.DatasetResult object at 0x7f7dbaef4128>, <pyesgf.search.results.DatasetResult object at 0x7f7d...h.results.DatasetResult object at 0x7f7dbaef4240>, <pyesgf.search.results.DatasetResult object at 0x7f7dbb16ae10>, ...])
======================================================== warnings summary =========================================================
../../../../var/tmp/ve/lib64/python3.6/site-packages/_pytest/config/__init__.py:1184
  /var/tmp/ve/lib64/python3.6/site-packages/_pytest/config/__init__.py:1184: PytestDeprecationWarning: The --strict option is deprecated, use --strict-markers instead.
    _pytest.deprecated.STRICT_OPTION, stacklevel=2

-- Docs: https://docs.pytest.org/en/stable/warnings.html
===================================================== short test summary info =====================================================
FAILED tests/test_connection.py::TestConnection::test_get_shard_list - AssertionError: assert 1 > 1
FAILED tests/test_context.py::TestContext::test_distrib - AssertionError: assert 53725 < 53725
FAILED tests/test_context.py::TestContext::test_distrib_with_cache - AssertionError: assert 53725 < 53725
FAILED tests/test_context.py::TestContext::test_replica - AssertionError: assert 1 > 2
FAILED tests/test_results.py::TestResults::test_batch_size_has_no_impact_on_results - AssertionError: assert 50 == 100
================================= 5 failed, 43 passed, 12 skipped, 1 warning in 358.71s (0:05:58) =================================

This is with some tests skipped - need to check again with credentials supplied (using a DKRZ openid - there is a temporary issue with CEDA ones):

tests/test_ats.py ss.                                                                                                       [  5%]
tests/test_connection.py ..Fs...                                                                                            [ 16%]
tests/test_context.py ...........FF..F.                                                                                     [ 45%]
tests/test_logon.py ssssssss                                                                                                [ 58%]
tests/test_opendap.py ...                                                                                                   [ 63%]
tests/test_results.py .F........s...                                                                                        [ 86%]
tests/test_shard_regex.py .                                                                                                 [ 88%]
tests/test_temporal_search.py ..                                                                                            [ 91%]
tests/test_util.py ....                                                                                                     [ 98%]
tests/test_wget.py .                                                                                                        [100%]

[alaniwi]

test_logon results with DKRZ credentials supplied:

test_logon.py ...FxFF.                                                   [100%]

=================================== FAILURES ===================================
_____________________________ TestLogon.test_logon _____________________________
Traceback (most recent call last):
  File "/path/to//esgf-pyclient/tests/test_logon.py", line 86, in test_logon
    lm.logon(TEST_USER, TEST_PASSWORD, TEST_MYPROXY)
  File "/path/to//esgf-pyclient/pyesgf/logon.py", line 186, in logon
    updateTrustRoots=update_trustroots)
  File "/var/tmp/ve/lib64/python3.6/site-packages/myproxy/client/__init__.py", line 1455, in logon
    **getTrustRootsKw)
  File "/var/tmp/ve/lib64/python3.6/site-packages/myproxy/client/__init__.py", line 1612, in getTrustRoots
    conn.write(self.__class__.GLOBUS_INIT_MSG)
  File "/var/tmp/ve/lib64/python3.6/site-packages/OpenSSL/SSL.py", line 1652, in send
    self._raise_ssl_error(self._ssl, result)
  File "/var/tmp/ve/lib64/python3.6/site-packages/OpenSSL/SSL.py", line 1566, in _raise_ssl_error
    _raise_current_error()
  File "/var/tmp/ve/lib64/python3.6/site-packages/OpenSSL/_util.py", line 57, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]
____________________ TestLogon.test_logon_with_verify_false ____________________
Traceback (most recent call last):
  File "/path/to//esgf-pyclient/tests/test_logon.py", line 96, in test_logon_with_verify_false
    self.test_logon({'verify': False})
  File "/path/to//esgf-pyclient/tests/test_logon.py", line 86, in test_logon
    lm.logon(TEST_USER, TEST_PASSWORD, TEST_MYPROXY)
  File "/path/to//esgf-pyclient/pyesgf/logon.py", line 186, in logon
    updateTrustRoots=update_trustroots)
  File "/var/tmp/ve/lib64/python3.6/site-packages/myproxy/client/__init__.py", line 1455, in logon
    **getTrustRootsKw)
  File "/var/tmp/ve/lib64/python3.6/site-packages/myproxy/client/__init__.py", line 1612, in getTrustRoots
    conn.write(self.__class__.GLOBUS_INIT_MSG)
  File "/var/tmp/ve/lib64/python3.6/site-packages/OpenSSL/SSL.py", line 1652, in send
    self._raise_ssl_error(self._ssl, result)
  File "/var/tmp/ve/lib64/python3.6/site-packages/OpenSSL/SSL.py", line 1566, in _raise_ssl_error
    _raise_current_error()
  File "/var/tmp/ve/lib64/python3.6/site-packages/OpenSSL/_util.py", line 57, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]
____________________ TestLogon.test_logon_with_verify_true _____________________
Traceback (most recent call last):
  File "/path/to//esgf-pyclient/tests/test_logon.py", line 92, in test_logon_with_verify_true
    self.test_logon({'verify': True})
  File "/path/to//esgf-pyclient/tests/test_logon.py", line 86, in test_logon
    lm.logon(TEST_USER, TEST_PASSWORD, TEST_MYPROXY)
  File "/path/to//esgf-pyclient/pyesgf/logon.py", line 186, in logon
    updateTrustRoots=update_trustroots)
  File "/var/tmp/ve/lib64/python3.6/site-packages/myproxy/client/__init__.py", line 1455, in logon
    **getTrustRootsKw)
  File "/var/tmp/ve/lib64/python3.6/site-packages/myproxy/client/__init__.py", line 1612, in getTrustRoots
    conn.write(self.__class__.GLOBUS_INIT_MSG)
  File "/var/tmp/ve/lib64/python3.6/site-packages/OpenSSL/SSL.py", line 1652, in send
    self._raise_ssl_error(self._ssl, result)
  File "/var/tmp/ve/lib64/python3.6/site-packages/OpenSSL/SSL.py", line 1566, in _raise_ssl_error
    _raise_current_error()
  File "/var/tmp/ve/lib64/python3.6/site-packages/OpenSSL/_util.py", line 57, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]
=============================== warnings summary ===============================
../../../../../var/tmp/ve/lib64/python3.6/site-packages/_pytest/config/__init__.py:1184
  /var/tmp/ve/lib64/python3.6/site-packages/_pytest/config/__init__.py:1184: PytestDeprecationWarning: The --strict option is deprecated, use --strict-markers instead.
    _pytest.deprecated.STRICT_OPTION, stacklevel=2

tests/test_logon.py::TestLogon::test_bootstrap
tests/test_logon.py::TestLogon::test_logoff
tests/test_logon.py::TestLogon::test_logon
tests/test_logon.py::TestLogon::test_logon_openid
tests/test_logon.py::TestLogon::test_logon_with_verify_false
tests/test_logon.py::TestLogon::test_logon_with_verify_true
  /var/tmp/ve/lib64/python3.6/site-packages/myproxy/client/utils/openssl.py:67: DeprecationWarning: The SafeConfigParser class has been renamed to ConfigParser in Python 3.2. This alias will be removed in future versions. Use ConfigParser directly instead.
    SafeConfigParser.__init__(self)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ============================
FAILED test_logon.py::TestLogon::test_logon - OpenSSL.SSL.Error: [('SSL routi...
FAILED test_logon.py::TestLogon::test_logon_with_verify_false - OpenSSL.SSL.E...
FAILED test_logon.py::TestLogon::test_logon_with_verify_true - OpenSSL.SSL.Er...
============== 3 failed, 4 passed, 1 xfailed, 7 warnings in 6.38s ==============

[alaniwi]

Following the changes in branch fix_tests, the tests are passing except for those now marked as expected failures relating to facets=* (and independent of caching). This includes passing tests related to caching, and applies regardless of whether requests-cache==0.7.2 or requests-cache==0.5.2 is installed in the virtual environment.

This issue can be closed once the PR for the tests is merged to master - see #73 - leaving for e.g. @agstephens to review.

There is no change required to the code base outside of tests/ because this has already been done.

[larsbuntemeyer]

(using a DKRZ openid - there is a temporary issue with CEDA ones)

This is still an issue for me, e.g., with my DKRZ openid, i can not access CORDEX data hosted at esgf.ceda.ac.uk. It neither works through pyesgf nor the ESGF web interface nor via wget script (i get 302/404 error). I get redirected to esgf-data1.ceda.ac.uk which gives 404.