/ python

Wrapped under the hood - 1

வாங்கற சம்பளத்துக்கு இன்னிக்கு உருப்படியா என்ன கிழிச்ச? - A question that I have often asked myself in the past. This one is so fresh in memory from talking to my first and best-thus-far manager, that I use it as an anecdote rather often.

That's something that knocks on my doors as much and it did a while back. To be honest with me, our debt clearance was way overdue this time around. Dismayed indeed I have been with this, let me face it. But then, that's how it works, given the "move fast, break things" nature; something I despise with all my heart.

When it came to this edition of the debt payment, there were certain focal points, as I saw over the last month.

First things first, the task was to ensure seamless organization with code. Be it shell scripts*, or the API that drove content to our apps, this is something that helps with sanity in the fast pace.

Second, to ensure communications happen in a near asynchronous manner, often. While straightforward, the lack of smaller reviews was a major blocker. Short-hauls work fine with such ad-hoc ways but long-hauls are rather demanding.

[I plan to write in brief about these two soon, once I have some time at my disposal. Interesting concerns these two are and need some discussion, after all.]

The third, and something that has seen a handful of experiments was real-time feeds. And where else does one start but the place where the

The core data pipeline that pulls data into Tenreads is special to my heart for more reasons than one. It was the first set of components I wrote for Tenreads and one that tested me most, during my early days with it. The first version of the pipeline, I would in retrospect, call out for its utter failure of a design. But, as they say, if your first version isn't shittier from your present shoes, you haven't learned a thing.

And boy, did I learn. Among components that push data to us, the pipeline that scouts for content is the point zero. And natural was the focus to shave time off as much as possible here. Over the last couple of years, I've spent time on experiments with varying stacks for this one part, most. From rewriting in C++ for efficiency to Golang and Postgres for ease, this one has seen a lot.

But, thanks to Guido and the rest of the awesome contributors, Python3.6's rebooted asyncio won.

While it is not a lot of change as such, the current ones give some peace/happiness. It took a few weeks of mulling around to bring it to place. Though it is a fair distance ahead left to call it ready, where it stands, is solid ground to build on.

This one is as much architectural as it is a stack overhaul. From using a handful of libraries to stripping it to three in all, this is a lot of savings from all sides. One small step, as an old man once said, while he was out of the world.

And what made me happier is the new tests; a 512MB box running three copies (five right now) of this continuous runner. This, working just fine along with a data API and a Mongo instance for datastore.

And with CRISPR like edits, your Crisp will be crisper as you deserve; real-time, cleaner, better. Oh, we rebranded to Crisp, did I tell you. Follow through here, for details.