Well, it’s that time of the month again. Windows restarted my machine overnight, just when I needed it least. How can a reboot of my machine actually effect a task I have going on? Well, because it’s because I’m a dumbass and have not yet moved over to tmux or gnu screen (or whatever) to manage my shell sessions through a server that can keep them alive even after I log out… or my terminal session running a script is force-closed because of Windows restarting my machine, precisely as I had instructed it to, because keeping your machine patched up-to-date all of the time is more important than the rare cases when a forced Windows reboot will actually screw you over.
Even this morning, the screw-over isn’t so bad. Just re-run scripts manually, like any given morning what-went-wrong-today snafu. Or is that tarfu? Yeah, I’m tarfu, and today is about moving it back to merely snafu status WHILE I prepare for the meeting that’s later today. Gotta roll SOMETHING out, and I must use that as the fire I light under my own butt to keep the SEO tracking system moving forward.
These are the gestational and formative days of a lightweight datamastering system that leverages the cheapest, most consumer-oriented megaglobal cloud infrastructure that is likely to outlast any of us. What’s further, we will not require any additional cost than that of getting onto the Internet and having a Google (gmail) account, meaning that there are not any extra bills to pay to keep such a system running.
That’s as far as the data. As for where the processing occurs, well that’s another lightweight part of the system. It could be almost any tiny Linux-like general purpose computer (cloud, Raspberry Pi, etc.) running almost anywhere (on the Net). It “checks in” with the data files on the Google cloud to see if there’s work to be done. If there is, it does the work and updates the data file, leaving logging information in that same said file that it has done so, so that no other systems checking in on the file bother to do so.
This reminds me that I have one more level of file locking that I have to implement. All this stuff is now mixed together as one big system, isn’t it? Yep! The routines for checking in on a GSheet whether it needs processing or not, and then moving on if the file has already been processed means that there needs to also exist a “being processed” status. Right now, using only a datestamp per job-needing-running, it generally only reads yesterday’s date or today’s date. Things that still read yesterday still need to be run, while things with today’s date have already been run. This is beautifully simple, easy and obvious to look at (as data in the logs tab) and already implemented this way. However, if more than one “pipulating” server is in the picture (spreading the work around for scaling), then two different servers could collide and try to process the same data file at once.
Okay, so just one more column per job (even that is a lot to add)… oh wait, unless there is a value we can put in that same cell… sure, there are no datatypes here! Obvious now that I’ve thought of it. But the value in the lock-cell can be a datestamp (and in RARE conditions, a timestamp) or other tokens, like “pipulating”. A little flashy and pretentious, but I am inventing the system here, so let’s keep that Github repo that has a following alive, once I get the green light to open source some of my work.
I’ve been doing stuff like this for years before my current employer, and what I’m doing new is working out a few of the details of “un-framework-itizing” this whole pipulation thing. It’s a few supporting libraries that make some cool stuff more easily possible WITHING OTHERWISE NORMAL PYTHON code. In other words, this is not one of those Ruby on Rails / Django-like things requiring you to take up a whole other language just to understand the configuration files you’re looking at. Rather, it’s like Flask, making you question whether it really is a framework at all, and not just Werkzeug, Requests and a few other libraries thrown together with a few instructions of how to connect the dots. Yup, what I’m doing should be a lot like import flask.
That really makes this stuff easy to brand, doesn’t it? import antigravity. Just pushed out a Tweet to that effect. Hmmm, now it’s a matter of re-focusing distractions into doing the right things for the right reasons. You can condition yourself to use your distractions as ways to corral your mind back into the directions it has to be going to move your life forward in meaningful ways that you will be happy you did when you look back to these days in the future.
Every moment you’re living is filled with those important little details. And now, I realize I was meant to push out my on-the-subway Note 8 stylus experimentation on my Medium channel, where I push out excerpts from my daily journal wiring. And in a moment that fleeting and abstract and simple in my head, my identity is born. An artistic guy gone deeply technical with a persistent “newb” perspective on things, trying to do a great job as a dad, an SEO in New York City, and contribute back to the world in positive ways loosely modeled on Linus Torvalds and Guido van Rossum, but in ways more aligned with my own personal abilities and interests.
Yup. Long elevator pitch. Gotta pair it down, but I actually feel the need to layer on that a core part of my schtick is idea-capture, and the both meticulously deliberate and (seemingly) wildly out of control way of moving the kernel of an idea forward to a fully realized life improvement. Yeah, there’s a touch of life hacks and learn to code in my schtick. But more-so, it’s probably ab out free-flowing story capture and gradual refinement in an unabashedly transparent way, finally now being made conveniently possible in our lives, through video, photos, drawings, writing, and all the variations thereof which inspire so much community.
And this is mostly just me keeping my foot in this particular Medium.