Elysian Shadows

This is my story of learning the hard way that python threading blows, and that its (almost) completely useless..

About 5 months ago I was hired into a crypto exchange, a fairly large one. ( cryptsy.com ) And we got to work on a new cloud mining thing that was real. ( mintsy.co ) ( Majority of 'cloud mining' are ponzi schemes.. )

So I am the dev for the proxy which is what controls all miners connected to us, and matches them up to which pool they should mine at and where. Seems fairly trivial and I got the first one running in a day no problem since python's threading was super awesome and sharing memory was no problem at all. Last week we started stress testing (after months of my proxy running with little load on it) and it got fucking demolished..

Messages coming from the pool were minutes delayed sending to the miner, and I had no clue why. After doing more research on pythons threading and how it works, I found this thing called the GIL. Or 'Global Interpreter Lock'. Basically, pythons threading runs one interpreter and thus only interprets one thing at a time, and threads are NOT parallel. So I would have been better off in a single thread on a single super power core rather than on a dual cpu multi core server. Infact, more cpus and more cores MAKES PYTHON MULTITHREADING EVEN SLOWER. It makes no sense whatsoever..

After a code-a-thon over this last weekend, I rewrote the entire proxy using python's multiprocess. Which really is a duplicate of pythons threading, except now you cant share memory (obviously, its in a different memory space..) BUT python has these awesome things called queues which you can use to pass information across different processes. Anyways, in the end I ended up launching every miner and every pool connection as a separate process. And suddenly the dev box looked like this http://puu.sh/fohMv/8212aec90b.png

My drunk rant about python threading which caused much headache over a couple days of not knowing what the fuck was going on, and being forced to not sleep and rewrite everything from scratch.

Obvious question, were you forced to write said code in python, or was this a personal choice? That said, I've read about PGL before, sorry you ran into this! Real header palmer IMO.

I thought that was a known fact. Your explanation is almost true, but things are much messier than that. Your solution, though, was correct: just use multiple processes instead of multiple threads. Python's stdlib makes it relatively easy to move from one to the other.

If you are really curious, this is an excellent presentation:

https://www.youtube.com/watch?v=Obt-vMVdM8s

If you are into python, I actually recommend all talks by this nice fellow, you can find most of them here:

http://www.dabeaz.com/

bbguimaraes wrote:I thought that was a known fact. Your explanation is almost true, but things are much messier than that. Your solution, though, was correct: just use multiple processes instead of multiple threads. Python's stdlib makes it relatively easy to move from one to the other.

If you are really curious, this is an excellent presentation:

https://www.youtube.com/watch?v=Obt-vMVdM8s

If you are into python, I actually recommend all talks by this nice fellow, you can find most of them here:

http://www.dabeaz.com/

Yea thats actually the video I watched to learn more about it initially, Python is a beautiful language and its fun to write when it works. It wasn't too bad moving it, but along with transitioning to processes I really wanted to clean up my shit thats piled on top of everything since my first poc version. So a rewrite helped me out quite a bit. I just demolish the cpu's now, working on taking it down on load when in reality it isn't doing anything super hard just really fast.

I ran into this same problem on a MUD I was doing in python, you described it succinctly. I was really disappointed with python at the time and thought about giving up and switching to a different language. I ended up getting over that hurdle but losing interest anyway. I'll have to check out that video and see if I can improve my method any. It was rather cludged together admittedly.