fir(test/init.py): Ignoring DeprecationWarning from "cassandra.io.asyncioreactor" #1118

Orenef11 · 2021-11-23T17:53:08Z

Replace all places that use old async node (@asyncio.coroutine and yield from) and use a new Python syntax (Only for Python v3.5+)
Save the old code to allow users with Python v2+ to run our tests.
requirements.txt:
- Install futures package only for Python v2
- Use the latest sex package version

Orenef11 · 2021-11-23T17:53:50Z

@absurdfarce Can you review this PR?

absurdfarce · 2021-12-10T06:36:49Z

@Orenef11 So I'm not sure I quite understand the intent here. Are the DeprecationWarnings coming from cassandra.io.asyncioreactor causing tests to fail? Or is the goal here merely to suppress any output that might result from these warnings? If we're actually seeing tests fail because of this problem I agree something needs to be done, but if we're only trying to limit some output logging that might be a different story.

How much noise do these exceptions actually generate?

And is there a case for fixing the root cause of the deprecations rather than suppressing the errors overall?

I guess I'm just a bit nervous about the idea of explicitly suppressing errors reported by tests. :(

Your comments mentioned that these errors were only visible on Python 3.8. I haven't tried a test run with that version of Cython yet... suppose I could just run through that exercise rather than being lazy and asking you all these questions. :)

Orenef11 · 2021-12-12T12:29:57Z

@absurdfarce
I think these are just warning messages. The goal was to reduce those messages from a few hundred or thousand to one for all tests.

If you want, I can try to fix it for Python3.8.

absurdfarce · 2021-12-13T20:49:46Z

@Orenef11 If you'd like to try to resolve them I'm certainly fine with that. I'm trying to repro myself in order to see which deprecation messages are causing the most trouble.. but I don't think I'm doing the same thing you are:

(python-driver-py38) [mersault@linux python-driver]$ nosetests --verbosity=2 -w tests/unit/io                                                                                     
Verify that timer timeouts are honored appropriately ... SKIP: not running asyncio tests; current connection_class is <class 'cassandra.io.libevreactor.LibevConnection'>         
test_timer_cancellation (tests.unit.io.test_asyncioreactor.AsyncioTimerTests) ... SKIP: not running asyncio tests; current connection_class is <class 'cassandra.io.libevreactor.L
ibevConnection'>                                                                                                                                                                  
test_blocking_on_write (tests.unit.io.test_asyncorereactor.AsyncoreConnectionTest) ... ok                                                                                         
test_eagain_on_buffer_size (tests.unit.io.test_asyncorereactor.AsyncoreConnectionTest) ... ok                                                                                     
test_error_message_on_startup (tests.unit.io.test_asyncorereactor.AsyncoreConnectionTest) ... ok                                                                                  
test_ewouldblock_on_buffer_size (tests.unit.io.test_asyncorereactor.AsyncoreConnectionTest) ... ok                                                                                
Validate that all messages are processed with different scenarios: ... ok                                                                                                         
test_partial_header_read (tests.unit.io.test_asyncorereactor.AsyncoreConnectionTest) ... ok                                                                                       
test_partial_message_read (tests.unit.io.test_asyncorereactor.AsyncoreConnectionTest) ... ok                                                                                      
test_partial_send (tests.unit.io.test_asyncorereactor.AsyncoreConnectionTest) ... ok                                                                                              
test_protocol_error (tests.unit.io.test_asyncorereactor.AsyncoreConnectionTest) ... ok                                                                                            
test_socket_error_on_read (tests.unit.io.test_asyncorereactor.AsyncoreConnectionTest) ... ok                                                                                      
test_socket_error_on_write (tests.unit.io.test_asyncorereactor.AsyncoreConnectionTest) ... ok                                                                                     
test_sslwantread_on_buffer_size (tests.unit.io.test_asyncorereactor.AsyncoreConnectionTest) ... ok                                                                                
test_sslwantwrite_on_buffer_size (tests.unit.io.test_asyncorereactor.AsyncoreConnectionTest) ... ok                                                                               
test_successful_connection (tests.unit.io.test_asyncorereactor.AsyncoreConnectionTest) ... ok                                                                                     
Verify that timer timeouts are honored appropriately ... ok                                                                                                                       
Verify that timer cancellation is honored ... ok                                                                                                                                  
Verify that timer timeouts are honored appropriately ... SKIP: Skipping the eventlet tests because it's not installed                                                             
Verify that timer cancellation is honored ... SKIP: Skipping the eventlet tests because it's not installed                                                                        
Verify that timer timeouts are honored appropriately ... SKIP: Skipping the gevent tests because it's not installed                                                               
Verify that timer cancellation is honored ... SKIP: Skipping the gevent tests because it's not installed                                                                          
test_blocking_on_write (tests.unit.io.test_libevreactor.LibevConnectionTest) ... ok                                                                                               
test_eagain_on_buffer_size (tests.unit.io.test_libevreactor.LibevConnectionTest) ... ok                                                                                           
test_error_message_on_startup (tests.unit.io.test_libevreactor.LibevConnectionTest) ... ok                                                                                        
test_ewouldblock_on_buffer_size (tests.unit.io.test_libevreactor.LibevConnectionTest) ... ok                                                                                      
Validate that all messages are processed with different scenarios: ... ok                                                                                                         
test_partial_header_read (tests.unit.io.test_libevreactor.LibevConnectionTest) ... ok                                                                                             
test_partial_message_read (tests.unit.io.test_libevreactor.LibevConnectionTest) ... ok                                                                                            
test_partial_send (tests.unit.io.test_libevreactor.LibevConnectionTest) ... ok
test_protocol_error (tests.unit.io.test_libevreactor.LibevConnectionTest) ... ok
test_socket_error_on_read (tests.unit.io.test_libevreactor.LibevConnectionTest) ... ok
test_socket_error_on_write (tests.unit.io.test_libevreactor.LibevConnectionTest) ... ok
test_sslwantread_on_buffer_size (tests.unit.io.test_libevreactor.LibevConnectionTest) ... ok
test_sslwantwrite_on_buffer_size (tests.unit.io.test_libevreactor.LibevConnectionTest) ... ok
test_successful_connection (tests.unit.io.test_libevreactor.LibevConnectionTest) ... ok
Test for asserting that watchers are closed in LibevConnection ... ok
Verify that timer timeouts are honored appropriately ... ok
Verify that timer cancellation is honored ... ok
Verifiy that _send_options_message() is called in ... ok
Verify that close() disconnects the connector and errors callbacks. ... ok
Verify that __init__() works correctly. ... ok
Verify that handle_read() processes complete messages properly. ... ok
Verify that handle_read() processes incomplete messages properly. ... ok
Verifiy that push() calls transport.write(data). ... ok
Verify that the protocol class notifies the connection ... ok
Verify that the dataReceived() callback writes the data to ... ok
Verify that timer timeouts are honored appropriately ... Unhandled error in Deferred:

Traceback (most recent call last):
Failure: twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused.

ok
Verify that timer cancellation is honored ... ok

Is there a cleaner way to repro the DeprecationWarnings you're seeing?

Orenef11 · 2021-12-13T21:48:57Z

@absurdfarce
I'll try to fix it :-)

Run the test with pytest
Create pytest.ini file in the main root:

[pytest]
python_files = test_*.py *_test.py *_tests.py
log_cli=true
log_cli_level = WARNING
log_level = DEBUG
log_format = %(asctime)s,%(msecs)03d %(name)-15s %(levelname)-5s %(filename)-15s:%(lineno)-4d | %(message)s
log_file_format = %(asctime)s,%(msecs)03d %(process)-7d %(name)-30s %(levelname)-8s %(filename)-20s:%(lineno)-4d | %(test_name)s: %(message)s
log_file= logs/python-driver.log

absurdfarce · 2021-12-13T22:11:35Z

Ah, thanks for that @Orenef11 , that was extremely useful! I see the errors you're referencing now.

Please let me know if you run into any trouble implementing a fix. It looks like you'll have to fix cassandra.io.asyncioreactor and likely update the asynctest dependency as well (since some of the error messages are coming from there as well). We also need to make sure any changes will also run without issue for older Python runtimes, specifically 3.5 (the oldest version we currently support in the 3.x line) as well as 2.7.

Thanks, and again please let me know if you need need any help!

Orenef11 · 2021-12-14T17:12:26Z

@absurdfarce
Please review the PR by checking each commit separately.
The last commit is very big because I renamed the TestCluster to IntergrationTestCluster

Orenef11 · 2021-12-15T09:19:32Z

@absurdfarce
The next step will be to change the test platform from nosetest to pytest :-)

Orenef11 · 2021-12-28T12:02:55Z

ping @absurdfarce
Can you review it?

Orenef11 · 2021-12-28T12:03:33Z

cassandra/io/asyncioreactor.py

@@ -9,24 +14,16 @@


 log = logging.getLogger(__name__)
+is_min_python_version_3_5 = (sys.version_info.major, sys.version_info.micro) >= (3, 5)


Ill change it to system.version_info >= (3, 5)

absurdfarce · 2022-01-04T16:34:06Z

@Orenef11 Apologies, everyone here has been out for holiday/end-of-year reasons. I'll try to take another look at this soon!

Thanks for your patience!

Orenef11 · 2022-01-04T22:21:02Z

@Orenef11 Apologies, everyone here has been out for holiday/end-of-year reasons. I'll try to take another look at this soon!

Thanks for your patience!

NP, when you can please review it.

…yncio.coroutine * Replace all places that use old async node (`@asyncio.coroutine` and `yield from`) and use a new Python syntax (Only for Python v3.5+) * Save the old code to allow users with Python v2+ to run our tests. * requirements.txt: * Install `futures` package only for Python v2 * Use the latest `six` package version

…arning: invalid escape sequence \" warning

…Warning: invalid escape sequence \" warning

…ionTestCluster"

absurdfarce · 2022-01-25T20:55:47Z

@Orenef11 My sincerest apologies for the extended delay in getting back to you on this. I've been swamped with work for some of the other drivers supported by DataStax but (as will hopefully become apparent soon) there's also been a fair amount of work going on behind the scenes for this effort as well.

I was a bit nervous about the extensive modifications to the test framework (like those outlined in this PR) just to remove some deprecation warnings. Perhaps even more importantly I couldn't shake the feeling that the deprecation warning really was trying to tell us something. In the not-too-distant future I'd like to catch the Python driver up with more modern Python runtimes and deprecation warnings such as these were indicating an area that would need to be addressed as part of any such effort. So I got to wondering... what's actually going on here and is it possible to just fix the underlying condition?

The results of that investigation are documented in PYTHON-1290. My conclusion was that given the set of Python versions we're tasked with supporting now it does make sense to fix the underlying issue directly. I've opened another PR to do exactly that.

At this point I think I'd like to close this PR in favor of the underlying fix. What do you think? I would certainly welcome your review of the other PR as well.

Orenef11 · 2022-01-26T07:39:08Z

No problem, my goal is to improve the driver as much as I can :-)

The current PR contains an additional commitment, can I open it in another PR?

Last thing, is it possible to use Pytest and not unittest package?
The change should be very minimal because Pytest can run unittest markes.

absurdfarce · 2022-01-26T16:01:20Z

Thanks (again) @Orenef11! I certainly appreciate all the effort you've put in so far!

Just to make sure I have this right: #1120 is the other PR you mentioned, right?

On the question of changing to pytest... where would you like to make this change? In the tox.ini setup for running tests (i.e. in favor of nose)? Or in the actual test classes themselves?

Orenef11 · 2022-01-26T17:01:19Z

Thanks (again) @Orenef11! I certainly appreciate all the effort you've put in so far!

Just to make sure I have this right: #1120 is the other PR you mentioned, right?

its is other issue...
https://www.flake8rules.com/rules/W605.html

On the question of changing to pytest... where would you like to make this change? In the tox.ini setup for running tests (i.e. in favor of nose)? Or in the actual test classes themselves?

The short fix it change only the tox.init.
The best way is to change all the tests so that they use pytest and not unittest, but the PR will be big.
What do you prefer?

absurdfarce · 2022-02-02T16:50:57Z

This issue has been addressed with the changes in #1119. Closing this one out.

Re: changing how the unittests are run... I'm open to the idea of converting the process running the tests to pytest. I do not want to change the tests themselves to use the pytest API. unittest is part of the Python standard lib so I'm extremely reluctant to move away from that.

absurdfarce self-requested a review December 8, 2021 17:29

Orenef11 force-pushed the igonore_deprecation_warnings branch from 2283cc6 to 8974779 Compare December 14, 2021 17:10

Orenef11 commented Dec 28, 2021

View reviewed changes

Orenef11 force-pushed the igonore_deprecation_warnings branch from 8974779 to 19394c3 Compare January 4, 2022 22:31

Orenef11 added 4 commits January 19, 2022 12:07

fix(tests/integration/standard/test_cluster.py): Fixing "DeprecationW…

e928c03

…arning: invalid escape sequence \" warning

fix(tests/integration/standard/test_metadata.py): Fixing "Deprecation…

f817191

…Warning: invalid escape sequence \" warning

fix(tests/integration/__init__.py): Rename "TestCluster" to "Integrat…

39fcfbb

…ionTestCluster"

Orenef11 force-pushed the igonore_deprecation_warnings branch from 19394c3 to 39fcfbb Compare January 19, 2022 10:07

absurdfarce mentioned this pull request Jan 25, 2022

PYTHON-1290 Convert asyncio reactor away from @asyncio.coroutine #1119

Merged

absurdfarce closed this Feb 2, 2022

datasttax added a commit to datasttax/python-driver that referenced this pull request Jul 28, 2023

Merge pull request datastax#1118 from datastax/remove_unittest2

62cac65

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fir(test/init.py): Ignoring DeprecationWarning from "cassandra.io.asyncioreactor" #1118

fir(test/init.py): Ignoring DeprecationWarning from "cassandra.io.asyncioreactor" #1118

Orenef11 commented Nov 23, 2021 •

edited

Loading

Orenef11 commented Nov 23, 2021

absurdfarce commented Dec 10, 2021

Orenef11 commented Dec 12, 2021

absurdfarce commented Dec 13, 2021

Orenef11 commented Dec 13, 2021

absurdfarce commented Dec 13, 2021

Orenef11 commented Dec 14, 2021 •

edited

Loading

Orenef11 commented Dec 15, 2021

Orenef11 commented Dec 28, 2021

Orenef11 Dec 28, 2021

absurdfarce commented Jan 4, 2022

Orenef11 commented Jan 4, 2022

absurdfarce commented Jan 25, 2022

Orenef11 commented Jan 26, 2022

absurdfarce commented Jan 26, 2022

Orenef11 commented Jan 26, 2022 •

edited

Loading

absurdfarce commented Feb 2, 2022

		@@ -9,24 +14,16 @@


		log = logging.getLogger(__name__)
		is_min_python_version_3_5 = (sys.version_info.major, sys.version_info.micro) >= (3, 5)

fir(test/__init__.py): Ignoring DeprecationWarning from "cassandra.io.asyncioreactor" #1118

fir(test/__init__.py): Ignoring DeprecationWarning from "cassandra.io.asyncioreactor" #1118

Conversation

Orenef11 commented Nov 23, 2021 • edited Loading

Orenef11 commented Nov 23, 2021

absurdfarce commented Dec 10, 2021

Orenef11 commented Dec 12, 2021

absurdfarce commented Dec 13, 2021

Orenef11 commented Dec 13, 2021

absurdfarce commented Dec 13, 2021

Orenef11 commented Dec 14, 2021 • edited Loading

Orenef11 commented Dec 15, 2021

Orenef11 commented Dec 28, 2021

Orenef11 Dec 28, 2021

Choose a reason for hiding this comment

absurdfarce commented Jan 4, 2022

Orenef11 commented Jan 4, 2022

absurdfarce commented Jan 25, 2022

Orenef11 commented Jan 26, 2022

absurdfarce commented Jan 26, 2022

Orenef11 commented Jan 26, 2022 • edited Loading

absurdfarce commented Feb 2, 2022

fir(test/init.py): Ignoring DeprecationWarning from "cassandra.io.asyncioreactor" #1118

fir(test/init.py): Ignoring DeprecationWarning from "cassandra.io.asyncioreactor" #1118

Orenef11 commented Nov 23, 2021 •

edited

Loading

Orenef11 commented Dec 14, 2021 •

edited

Loading

Orenef11 commented Jan 26, 2022 •

edited

Loading