Published inGraphlet AI·PinnedOpen Source Software and PaaS: Crisis and AdaptationStructural changes in the open source ecosystem are occurring as a result of the rise of cloud computing Platforms as a Service (PaaS) that are disrupting the cycle of investment that sustains the ecosystem. The shift to cloud computing reduces the number of companies making direct investments in open source…Open Source5 min readOpen Source5 min read
Published inGraphlet AI·PinnedMember-onlyDeep Discovery is hiring networkistas!AI + Networks vs. Corruption I’ve recently joined Jeffrey Stein in founding Deep Discovery in order to fight corruption, which is a massive problem globally and here at home. We aren’t just making money, we’re a company for good. …Data Science3 min readData Science3 min read
Published inGraphlet AI·PinnedPython and Parquet PerformanceIn Pandas, PyArrow, fastparquet, AWS Data Wrangler, PySpark and Dask — This post outlines how to use all common Python libraries to read and write Parquet format while taking advantage of columnar storage, columnar compression and data partitioning. Used together, these three optimizations can dramatically accelerate I/O for your Python applications compared to CSV, JSON, HDF or other row-based formats. …Python13 min readPython13 min read
Feb 9A Distributed Dask Quickstart… that makes Pandas faster!The Dask Quickstart easy way has never worked for me! Nor has the first file I loaded in an unpartitioned state ever processed quickly. …Python5 min readPython5 min read
Published inGraphlet AI·Feb 6On Persisting SSH Connections OvernightEver want to leave an SSH connection on overnight? I do this with AWS EC2 machines all the time and get grumpy when it doesn’t work. So here goes for a two-part solution: ServerAliveInterval and tmux with named sessions. SSH Connection ServerAliveInterval Host * ServerAliveInterval 240 This pings the server every four minutes…Bash2 min readBash2 min read
Published inGraphlet AI·Feb 4Add a random ID column to a pandas DataFrame using NumpyI needed to generate random IDs to partition some data for Dask when writing a Parquet file from pandas for a less expensive operation where multiple cores were not required. …Python2 min readPython2 min read
Published inDeep Discovery·Nov 16, 2021Business Graphs for KYC ChecksNetwork Risk Scores > Traditional Watch Lists How clear a picture of financial risk do sanction lists provide? Not very clear, it turns out. …Kyc3 min readKyc3 min read
Published inDeep Discovery·Aug 10, 2021Member-onlySome records have more signal than others…In the world of data labeling for financial crime AI… this is the equivalent of a home run.Know Your Customer1 min readKnow Your Customer1 min read
Published inDeep Discovery·May 4, 2021Member-onlyA Letter from our Founders“Three things cannot long stay hidden: the sun, the moon and the truth.” ―Buddha At Deep Discovery, we are in the business of discovering secrets. The secrets that undermine the health of civilization. Secrets hiding in plain sight in the vast ocean of data across the open Internet. …Kyc2 min readKyc2 min read
Published inGraphlet AI·Jan 28, 2021Member-onlyUsing the Python ML Stack inside PySparkWorking around serialization problems in PySpark If you’ve ever tried to use libraries like numpy, scipy, scikit-learn, transformers, PyTorch or Tensorflow in PySpark, you know how difficult this can be due to serialization errors as Spark’s Java core tries to distribute work across the cluster. …Python2 min readPython2 min read
