Python 3 Port

Post date: Feb 28, 2019 4:35:57 PM

With Python 2.7 going out of support in 2020, I will be upgrading all the CrossMgr software to Python 3.

This project will include:

  • CrossMgr

  • CrossMgrVideo

  • CrossMgrImpinj

  • CrossMgrAlien

  • TagReadWrite

  • SeriesMgr

  • RaceDB

The plan is to use the "six" module to make the source code compatible with Python 2.7 and Python 3 simultaneously.

This allows me to test the code in both Python 2.8 and Python 3 so that I can verify that the two versions work the same.

After 2020, the plan is to support Python 3 exclusively and drop support for Python 2.7.

Q: What does this mean to me?

A: If you install from windows, over the next few months there will be a new program version with a number 3.x.x. It will install as usual.

The 3.x.x version will be backwards compatible with previous races and series. You can read/write your existing files as usual.

The previous versions will continue to be available until 2020. When the 3.x.x versions are available, bug fixes and new enhancements will only be available in the 3.x.x version.

Version 3.x.x programs and older versions will continue to interoperate. For example, it will be possible to use 3.x.x CrossMgr races with previous versions of SeriesMgr or CrossMgrImpinj, or CrossMgr with RaceDB.

If you are running RaceDB, you can continue to run with Python 2.7 for 2020. In 2021, you will need to upgrade to RaceDB 3.x.x and switch to Python 3. If you have written scripts in Python 2.7, you will have to upgrade them to Python 3. In most cases, this isn't difficult with 2to3.

The 3.x.x versions will be backwards compatible with your RaceDB database as usual. You can continue to use your existing database.

More details:

As the underlying modules that support CrossMgr etc. will soon stop supporting Python 2.7, supporting Python 3 is important so that CrossMgr programs continue to be available for years to come.

The Python 3 conversion is not expected to change performance (maybe slightly faster) and mostly involves small syntactic and organizational changes. However, there are some issues that have come up.

Python 3 only manipulates strings in uncode format, and reading/writing data externally must provide an explicit encoding (usually utf-8). Python 2.7 was more flexible about this and could maintain regular strings and unicode internally (of course, this could cause other problems too).

A big challenge has been in reading/writing data to sockets and converting images to/from graphic representations (png, jpeg, etc.), and there are a number of idiosyncrasies to swear over ;)

For example, in Python 3, json.dumps( ...) returns a unicode string. If you then try to write it to a socket with send(...), it fails as "send "requires "bytes" not "str". A bit of a surprise because one generally writes json to a socket.

The solution is s.send( json.dumps(...).encode() ) which encodes the string to bytes in utf-8. Fortunately, this two step approach also works in Python 2.7. Unfortunately, I can't find a tool that can check this statically, so, lots of testing required cause the code just blows up if this happens.

Another issues is that open(fname, 'rb') opens a file returning bytes, not a str, and open(fname, 'r') returns str not bytes. One has check every open call to check whether it is dealing with binary data (images, etc.) or text data in utf-8. As CrossMgr creates lots of html, ftp, pdf, Excel, ... files there is a lot to check. Again, I have not been able to find a tool to check for this (more testing...)

Finally, you cannot put bytes into a StringIO - you must use a BytesIO. This mostly impacts reading images, but, Python 2.7 has no Bytes object, so coding changes are required.

The good news so far is that this process has required me to do a line-by-line code review as well as run pylint. This has resulted in fixing over 30 bugs (obscure) so far. This should help with those once-in-a-while CrossMgr hangs.

More details to come as the process continues.