Re: [pygame] pygame with SDL2 proposal

[René Dudfield <renesd@xxxxxxxxx>]

Haha. "import pygumm" definitely does sound better. Seriously.

For *me* pypy is a non goal, and CFFI has some drawbacks. I know pypy also has plenty of upsides in certain cases... which has been mentioned a lot, so I won't repeat the good parts about it here.


This was my reasoning...

• CFFI is slower on CPython. CPython is what 99% of people use.
  
• People don't really contribute if something is in python which interfaces in C - and they only know python. This has been proven by other cffi and ctypes projects. Ctypes, and CFFI is a much rarer skill than standard platform tools. C still happens to be very widely known.
  
• CFFI has issues on weird platforms... like iOS.
• Cython, numba, numexpr are available. Numpy is not hard to install any more. These are faster than pypy in throughput, and latency.
  
• Shaders in GLSL are pretty damn amazing if you need cross platform CPU/GPU acceleration.
• pypy has jit warm up, jankiness because of jit and GC, and also doesn't work on weird platforms like iOS.
• Converting everything to shared libraries, and supporting pypy is a whole lot more effort. However, in general I support sending things upstream and separating the C/asm code from the python code. So that it's easier for other people to use them, and there is less maintenance on pygame itself. However, packaging up shared libraries for multiple platforms is a lot of work. If those shared libraries are not already packaged for the 24 platforms, that actually creates maintenance burden as well.
  
• pygame_cffi is slower on pypy than on CPython for many apps where the bottlenecks are already in the C/asm/platform code. Before the pypy project took funding, forked pygame, removed credits and copyright info, they were warned that blit was the bottleneck. So, like in a lot of things, it looks good in their benchmarks, but for real world things it doesn't perform so well - in many cases.
• pypy uses significantly larger amounts of memory. Which is important for some older, or low memory systems.
  
• pypy is slower than CPython where it matters for real time apps. Where pypy is faster, there are faster extensions available for CPython, and more appropriate for game JITs. Pypy has many performance cliffs, where it is really fast for some things after lots of warmup, but then for some magic un-obvious reason you fall into the valley of slow. The baseline pure python code performance is always faster in CPython when the jit can't work. Lower bound performance, and consistent performance turns out to be really important to real time apps. This means you can write a lot more code in python, without falling into the valley of slow.
  

Happy to take patches for pygame to help make it easier for anyone who wants to share code. Rect is mainly an SDL thing, with a whole bunch of niceties mostly at the python integration level. I'm not sure how to separate any of that really. There are pure python implementations of Rect, and perhaps you could use that to implement something in RPython for integration with pypy? But I think perhaps this runs into the case where integrating with C is slower in CFFI when there are lots of calls which take very little time. I'm sure the pypy CFFI people could help you there.

Again, there are definitely some good use cases for pypy, and CFFI. Even in games. Like in your game. But for me, and for the reasons above, I don't think it's a good idea for pygame. At the moment.

CPython has had some really nice speed improvements recently (since 2015). Python 3.6 is good, and 3.7 is looking significantly faster for some benchmarks.
    https://speed.python.org/comparison/?exe=5%2BL%2Bdefault%2C5%2BL%2B2.7&ben=223%2C224%2C225%2C226%2C227%2C228%2C229%2C230%2C231%2C232%2C233%2C234%2C235%2C236%2C237%2C238%2C239%2C287%2C240%2C241%2C242%2C243%2C244%2C245%2C246%2C247%2C248%2C249%2C250%2C251%2C252%2C253%2C254%2C255%2C288%2C256%2C257%2C258%2C259%2C260%2C261%2C262%2C263%2C264%2C265%2C266%2C267%2C268%2C289%2C269%2C270%2C271%2C272%2C273%2C274%2C276%2C275%2C277%2C278%2C279%2C280%2C281%2C282%2C285%2C284%2C283%2C286&env=1&hor=true&bas=none&chart=normal+bars

Especially with all the FASTCALL work.
    https://haypo.github.io/fastcall-issues.html
    http://faster-cpython.readthedocs.io/fat_python.html
Accepted CPython jit API pep.
    https://www.python.org/dev/peps/pep-0523/

I like this trend where people have decided to work together to get things into the CPython project, rather than do forks.

cheers,

On Sun, Mar 19, 2017 at 6:32 AM, bw <stabbingfinger@xxxxxxxxx> wrote:

Hi, thanks for inviting me to opine. :)

I like it. But "import pygumm" looks a lot more handsome than "import pygame2". Seriously.

Having had much recent experience with a SDL2 implementation (pypy, CFFI, plus all the SDL2 libs), I found that I would really like to see in the next generation:

1. CFFI wrapper. It is Python, which any of us can write, and it exposes all of the SDL2 API to programmers, not just what the C developers choose to expose. This would make pygame superbly extensible. And it would allow easy ad hoc inclusion of third-party prebuilt libs, something that is a fairly high bar to hurdle in pygame gen 1.

2. CFFI would also make pygame compatible with pypy, something which is sorely needed in my opinion. One can use optimization tricks to compensate for CPython's speed limit, but one cannot do many things that are computationally intensive without farming out to a third-party opaque library. pypy could let coders do a lot more in Python, when a full blown physics/geometry/numeric/scientific opaque lib is overkill.

3. I had acquired the problem of a slower Rect in pypy, however, because pygame has a pyd and I'm not spiffy enough to port it to anything but Python. If CFFI and pypy are an option worthy of consideration, Rect would be best done in a shared library. Rect is truly a workhorse, and pypy Rects can't compete with pygame Rects under heavy usage of Rect() and inflate and move, etc. In pygame most of my time is spent in blit(). In pypy most of my time is spent in Rect, because the class is 100% Python except for the four-slot CFFI data structure.

See what can be done with CFFI and pypy here:
http://pygame.org/project/2957
https://bitbucket.org/gummbum/gsdl2

I made a number of releases and games And a few drop-in tests "import gsdl2 as pygame", but it ain't perfect. However, I wasn't trying hard for software rendering compatibility, I was shooting for the new texture features. As expected, I discovered that all of the software-blit optimization tricks that I solved in gummworld2 were no longer needed.

Sounds like a few of you have already put a lot of thought and discussion into the next gen, and I understand this is something completely different. I certainly don't propose gsdl2 as a new code base. But it would be grand if choose to use CFFI. I have no stake in seeing my code transform into a butterfly, but if parts of it are useful then good.

Also, an acceptable alternative might be to provide CFFI-compatible shared libs so that people who want to use pypy have that choice, even if they do not have the nice pygame API available. If so, anybody could write a pygame-like API just like I did, and it doesn't need to be maintained by the pygame core developers.

On 3/18/2017 5:02 AM, René Dudfield wrote:

Hello,

so, I've spent some more time reading source code, and investigating SDL2 and the options a lot.

I *think* this plan below could be the best way forward for us.

tldr;
 * have a single source SDL1, and SDL2 code base with a compile time option. (like we have single source py2 and py3).
 * move bitbucket/pygame repo to github/pygame.


Rather than reimplementing everything, and introducing lots of compatibility bugs, Instead I think a gradual approach would work better. I'm starting to think that perhaps pygame_sdl2 by Tom, is not such a great way forward (for the pygame project, I think it was necessary and good for Ren'Py).

A big breaking change is kind of silly for how little resources we have. Python 3 managed to pull it off... after a decade, and with massive resources having poured into it. And it only took off once they put compatibility code back in. Here's the thread where some discussion has been happening.
https://bitbucket.org/pygame/pygame/issues/174/pygame-20-the-sdl2-edition


What we do have is some patches to get the current code base working with SDL2.
https://bitbucket.org/llindstrom/pygame-1.10-patch

I think it should be possible with some work to improve an SDL2 compatibility layer, so that pygame code base can work with either (as a compile time option). Then newer APIs can be introduced in time, in a non break-every-program kind of way. Also, it's been proven that 'hardware' blitting does not need to break SDL1 API compatibility to use hardware accelerated APIs (Angle, SDL_gpu, [insert 4 or 5 other projects], ...).

Having a pygame2, or whatever separate repo would also make maintenance harder. Since for the foreseeable future, we will likely need to do maintenance releases for this anyway (at least I want to!, and I know other users of pygame will).

---

For pip uploads, they would continue to be for SDL1, until we can finish the SDL2 changes, and it works better. There would be no new additions until compatibility is mostly there.

The work would progress by slowly adding in compatibility changes whilst keeping pygame working. By keeping the SDL2 patch set as is, and slowly reducing the patch set until it is size zero. So we always have a pygame with sdl2, and a pygame with sdl1 that is producible. Eventually the patch set will disappear.

---

A pygame2 module would just cause confusion amongst users, and that really boring 'pygame 1 or pygame 2' type debate would go on, and on like the "python 2, verses python 3" one did. For me, just avoiding that discussion for the next decade would be worth it.

Then there would need to be two sets of documentation on the website. And two sets of tutorials... and... we'd have 2 of everything to *do*, and 2 of everything for people to look at.

Finally, "import pygame2" looks kind of weird. I've grown used to "import pygame".

...



I'm keen to get moving on this. I've been trying to get feedback on the 'pygame sdl2 addition' issue for the last month, and the issue has been open since 2013. So I'd like to put a time limit on this decision for one more week.

I'd really like to hear back from contributors, and users (everyone).




ps. Interestingly SDL_gfx has an SDL2 release now. http://www.ferzkopp.net/wordpress/2016/01/02/sdl_gfx-sdl2_gfx/