Workflow

The saying rightly goes, “Change is the law of life, and those who just look at the present and past, will miss the future.” Project management is the landscape that has been ever-changing with enterprise solutions. With new technological advancements in place, project management evolves with time, and businesses must…

{Core} z/OSMF workflows are not new on the mainframe landscape. However, their adoption by the community is not as widespread as they deserve. Let me show you what z/OSMF workflows can do for you when installing The Open Mainframe Project’s Zowe z/OS components for the first time. Zowe Workflows In my previous…

How to choose the right data scheduling tool? This is a question for all data engineers. Xu Haihui, a software development engineer at China Mobile (Suzhou) Software Technology, said: “As the old saying goes, ‘If you want to do something well, you must first make use of the tools’. If…

At XWBank, a large number of task instances are generated every day, with real-time tasks making up the majority. To better handle the task instances, XWBank chose Apache DolphinScheduler to solve this challenge after a comprehensive consideration. Today, several XWBank projects have applied real-time and quasi-real-time batch processing and offline…

Ticket management is one of those boring topics that comes up every now and again in OPS circles. A lot of teams that I’ve chatted with try to model their ticket management process after the development process using Sprints/Scrum. I’ve found Scrum to be limiting in an Operations setting. The…

3 simple ideas to minimize noise and focus on your work or to-do list How crochet taught me more isn’t always better Whilst living in long-term tenancies, surrounded by all my lovely things I had this huge box of wool. My guilty pleasure was frequently adding more yarn to it, despite knowing I probably have enough for the…

This article is meant for someone who is looking to deploy their ML, ETL or any other task pipeline using dagster. Or if you are still in process of choosing a production level workflow orchestration tool, I’d strongly suggest dagster as one of the options. You can read more about…

弄了一個工作自動化兼排程的生資平台 這個平台長大到現在，終於也可以跟其他pipeline framework匹敵了!! (自己在說) 意外超車!? 總之終於也有了咱們自家定義tasks/pipeline的設定檔啦!! 雖然市面上已經有像是CWL (Common workflow language)或是WDL (Workflow definition language)，CWL是一種open standard，而WDL則是Broad Institute開發的。 但! 要寫程式去讀懂CWL與WDL的規格，還是太多工了，實在不是我一個人能做得了的(x 尤其是我自己就有開發出自己的pipeline排程系統XD 因為原本就有提供programming interface可以讓使用者去定義工作，現在只要想辦法弄一個格式，與我的programming interface串接起來就可以。 為了讓使用者寫設定檔即可定義工作的這個概念可以實現