LynxKite Highlights from 2022

12 Dec 2022 — Daniel Darabos, Chief Software Architect

While 2022 brought much misfortune and tragedy around the world, it was kind to our small project. Our engineering team has doubled in size. We have not just worked off technical debt, but turned it into its opposite. (“Technical credit”?) And we have added some stunning new features in the 5.x versions.

Let’s take a closer look!

Technical debt to technical credit

Lynx has always taken maintenance work seriously. But taking something seriously is not always enough. The best time to make large systemic improvements is when these improvements become the keys to reaching a goal more easily. Then the time investment for maintenance pays off immediately, it’s clear what we need, and everyone is motivated.

This happened several times this year:

• We have switched to a faster, more standardized storage format for graph data in #237. Back in 2014 we chose SequenceFiles for data storage. I can’t recall why we didn’t opt for Parquet instead. Perhaps it wasn’t fully supported in Spark 0.9. Perhaps we didn’t realize its advantages yet. (Parquet 1.0 was released just a year earlier.) But better late than never: we’re on Parquet now.

• We replaced the system for launching LynxKite. (#269) LynxKite is now a single JAR that you run with spark-submit. LynxKite uses Apache Spark for distributed computing and the Play Framework for providing a web interface. Both of these have a system for launching your application for you. We had to pick one to launch LynxKite and replicate what the other launcher does in our own code. We picked Play in 2014. We understood Spark’s startup better and Play’s generated startup script was flexible. Our modifications to the script grew into an elaborate system of its own. (call_spark_submit.sh) That is gone now. We flipped the startup – we trust spark-submit to start LynxKite and we set up Play ourselves. This lets LynxKite behave like any Spark application. It easily runs in highly customized Spark environments now.

• We’ve switched our build and CI to Earthly. #296 The Makefile is not deleted yet. But Earthly can manage not just the build steps, but the environment and the Docker images too. We share our build cache in the office and with GitHub Actions. The whole setup is simpler and more reliable. (This builds on our work from last year to use Conda for managing all dependencies.)

• We’re switching UI testing automation from Protractor to Playwright. #307 LynxKite is an AngularJS application. (Another choice that made sense in 2014.) Protractor is a dedicated test framework for AngularJS. It’s so deprecated at this point that it’s hard to even run it, never mind reliably or with a current browser. I’m happy to report that Playwright is faster. It has already caught bugs that Protractor’s auto-waiting kept hidden. And the migration is not even hard.

• We’ve revamped demo.lynxkite.com. We can manage and update it easier now and it has that nice new software smell.

New features in LynxKite 5.x

An alternating focus between maintenance and features (a “tick-tock” cycle) is common in software development. You might think 2022 was a “tick” year for LynxKite. But it wasn’t! Check out our new features:

• Run LynxKite scripts without a LynxKite server. #294 Until now you had to run a LynxKite server to be able to use the LynxKite Python API. But now if you have a SparkSession in Python, you can leave it up to the LynxKite library to start a short-lived LynxKite process to execute your code. Perfect for automated batch pipelines!
• Databricks support in LynxKite Enterprise. You can run LynxKite as a job cluster. It’s easy to set up, and once the job is created you can turn it on/off with a click. You can also run LynxKite from a Databricks notebook.
• R support. #292 Python took over most of data science. But not all! There are some fantastic R libraries especially in health sciences that have no Python equivalent. Also some fantastic people who prefer R. Now you can use R in LynxKite via the “Create graph in R” and “Compute in R” boxes.
• Custom plots in Python and R. #297 LynxKite supports static images and interactive HTML charts. Time to present your data science results as abstract 3D artwork!
• GPU support. #241 Many graph algorithms now run on CUDA GPUs.
• Katana Graph support in LynxKite Enterprise. Fast algorithms and Cypher queries for truly enormous graphs.
• A more efficient Neo4j import. #268 For the case when you have a Neo4j that Neo4j itself can’t handle.
• BigQuery support. #245
• Zero-copy import. #261 Importing a table in LynxKite means saving it as a Parquet file in LynxKite’s storage. That’s kind of silly when your source data is already a Parquet file!
• Delegate user permissions for HDFS access in LynxKite Enterprise. Useful if you use Kerberos both for LynxKite logins and file access permissions.

Plans for next year

All of these big additions were motivated by the needs of users. In 2023 we will continue to listen to your feedback and focus on what makes the most difference for users.

That sounds a bit like “we have no plans”, doesn’t it? Let me list the areas that are main strengths of LynxKite and what sort of improvements we are considering in each area:

• Integrations. Many tools you find in LynxKite are also available elsewhere. But LynxKite brings everything together into one big pot. We have long wanted to add integration with Gremlin graph tools like JanusGraph or Azure Cosmos DB. We’re also looking at compatibility with some tools from the KNIME ecosystem.

• Performance. GPU algorithms are already fantastically fast. But the data is currently moved between the GPU and the CPU for each algorithm. Instead, we could keep the data on the GPU and only remove it when necessary.

• Interactive exploration. We are considering many improvements to our visualization engine. Streamlined controls, embedding in Jupyter Notebooks, and better support for heterogeneous graph (such as knowledge graphs) are all in the cards.

We can’t wait for your feedback in 2023. Let’s toast to another successful year for LynxKite!