There are two ways for scheduling/hosting the Spark executors.

Apache Spark includes the Hadoop client libraries. This gives LynxKite the capability to access file systems supported by Hadoop, such as HDFS, S3, and GCS. Data files can be imported from and exported to these file systems. JDBC is used to import from and export to SQL databases. JDBC drivers for some popular database vendors are included. To add more JDBC drivers, copy them into the directory specified by EXTRA_JARS. (As explained in The .kiterc file.)

LynxKite needs some storage space on a filesystem to store internal graph data. This filesystem has to be accessible by all workers. Because of this requirement, in most cases this needs to be a distributed file system with fast access from the Spark workers. Ideal setups include using S3 when running on Amazon EMR or using the HDFS of the same cluster when running using Hadoop YARN. On single machine installations one case simply use local hard disk for this purpose.

Notably the LynxKite process runs on a single machine. This is a design decision on the part of Apache Spark. While it introduces a single point of failure, it does not create a scaling bottleneck, since all the computation work is performed by the Spark Workers.

LynxKite is able to complete operations even with a very small amount of memory compared to the size of the data. So there are no hard data size dependent RAM requirements. Of course, operations will be much faster if the data can be kept in memory continuously. To have an idea of how much memory is needed for that depending on your graph data, see the sections below.

The RAM Formula

How much RAM does LynxKite need in the cluster to be able to work with a data set in memory? The following formula is easy to remember and gives a rough estimation.

To import a CSV of L lines and N columns as vertices, LynxKite needs B_v bytes.

B_v = L × 200 × (N + 1)

Each column needs 200 bytes, and one more column, the vertex ID, is generated.

To import the same CSV of L lines and N columns as edges, LynxKite needs B_e bytes.

B_e = L × 200 × (N + 3)

The edges also get an ID, plus they store the source and destination vertex ID.

These formulas assume columns of around 10 characters. If a column contains significantly more characters, add 2 bytes per character. For a less accurate estimate, we can say that LynxKite needs X × 20 — X × 30 bytes of RAM for working with data that is X bytes on disk.

A Simple Example

Let us say we have a CSV with two columns, src and dst: N = 2. It contains 100 million lines: L = 100,000,000. To import this graph, we need Be = 100,000,000,000 which is 100 GB of RAM.

Getting from the amount of RAM to the number of machines is mostly trivial. There is some overhead, but the formula is loose enough to accommodate that. So if we have 32 GB machines and need 100 GB of RAM, we can just say we need 100 / 32 = 3.125 rounded up to 4 machines. This is the number of worker machines. One extra machine is needed which serves as the master and the web server.

If you need B bytes of RAM and have C bytes of RAM per machine, you need M machines (rounded up).

M ≥ B / C + 1

See the The 32 GB rule for more tips.

LynxKite uses a lot of storage for processing and hosting data.

Storage capacity for data hosted by LynxKite

As a rule of thumb it is recommended to plan at least with 10 times of the storage capacity of the original data. So if you have 100 GB data to analyse in LynxKite have at least 500 GB set aside for permanent storage.

Storage capacity for temporary data

During computation Spark creates temporary files. These files are cleaned up by Spark (once they are not needed) or after restarting LynxKite. The size of these files depend on a lot of things but it is generally recommended to set aside at least 200 GB for them on each Spark executor.

See Configure directories for more details.

The main log of LynxKite is the application log. LynxKite creates a new log file every day (except for those days when it is not used). It automatically deletes the log files older than 60 days.

The application logs the following activities (among other things):

Admin users can browse and download these logs using the <LynxKite_URL>/#/logs URL.

Every time LynxKite is started a log is created for STDERR and STDOUT with the process ID of the application. These logs generally do not contain much information unless something went wrong during the startup process.

It is recommended to have at least 16 GB RAM per machine. It is optimal to have around 4 GB RAM per CPU core. It is common to use 32 GB RAM machines with 8 CPU cores. LynxKite needs a lot of storage for computed data, so we recommend to have a sufficiently large distributed file system (e.g., HDFS). The needed capacity depends heavily on the way LynxKite is used, but it is recommended to plan with at least 5x of the initial data size. For temporary computation files it is recommended to use a smaller but faster drive (e.g. a partition on an SSD with at least 100 GB).

LynxKite runs on any modern Linux distributions. We recommend using Ubuntu Server 14.04 LTS or CentOS/Redhat 6.7, because LynxKite has been tested on those systems.

You need Java 8 on the system.

You need write permission to the directory configured as KITE_DATA_DIR. On HDFS a home directory is also required for the user that starts LynxKite. Example commands to create these directories:

By default LynxKite runs on port 2200. If this is used by another process, move LynxKite to a different port (by changing the KITE_HTTP_PORT setting in .kiterc).

The Java Virtual Machine can be configured to use practically any amount of RAM on a 64 bit system. However its efficiency breaks down after 32 GB. Therefore a JVM with 40GB RAM is actually worse than a JVM with 32 GB RAM. For that reason it is not recommended to set the EXECUTOR_MEMORY above 32 GB. It makes more sense to create multiple executors with less amount of RAM per executor.

You need to create a site configuration to be able to run LynxKite. If you don’t have a config file created yet, running the runner script will warn you and point you to a template config file. Just copy that template to your home directory with name .kiterc and edit the settings as necessary. The settings are documented inside the template file and in the The .kiterc file section. This information is also available in the template file as comments.

Configuration options can also be set using environment variables. This also works when running via the standard docker image.

If an option is set both in an environment var and in .kiterc then the environment variable’s value takes precedence.

This is a best practices section on how to configure various directories properly. The default configuration provided by Hadoop may be suboptimal.

LynxKite can be started with the lynxkite command in the bin directory of the LynxKite installation. For example: kite_2.6.2/bin/lynxkite start. Use the start / stop / restart options if you want to run it as a daemon. Or, you can use the interactive command option, which starts up the program tied to your current shell session, so its output will appear in the shell and pressing Ctrl-C or closing the shell session will terminate the program.

To enable HTTPS, a certificate is required. For testing purposes a self-signed certificate is included (conf/localhost.self-signed.cert). Its password is “keystore-password”. This certificate does not provide proper protection. (Lacking a trusted CA, the self-signed certificate can easily be spoofed.) Using it will trigger a certificate warning in the user’s browser. It is recommended to install a real certificate in its place.

Please refer to the HTTPS setup in the .kiterc file for more details.

Once LynxKite is up and running, take a moment to think about backups. The solution is entirely dependent on the site and purpose of the installation. But consider that LynxKite deals with the following data:

There is a backup tool for Hadoop deployments shipped with LynxKite under tools/kite_meta_hdfs_backup.sh. It is recommended to create a cron job running this tool on a daily basis (or more frequently if needed). The tool creates a copy of the full KITE_META_DIR and the .kiterc file in HDFS.

To restore an earlier version of LynxKite simply copy the .kiterc file and the meta directory back from HDFS to overwrite the current versions of them, then restart LynxKite.

You can provide extra jar files that will be added to the CLASSPATH of LynxKite server if needed, as specified here.

LynxKite is compatible with HDFS running in High Availability mode. In this case the HDFS prefixes i.e. the KITE_DATA_DIR variable in The .kiterc file and the Prefix definitions need to use the appropriate name service defined in the hdfs-site.xml configuration file of Hadoop (e.g. hdfs://nameservice1/user/my_user/my_dir). Make sure that the file is available in the YARN_CONF_DIR. Please refer to the related Hadoop documentation for more details.

When trying to start LynxKite on YARN, you may hit the YARN per-application memory limit. LynxKite will log an error message, such as:

See YARN memory limit for the fix.

This can be due to many reasons such as various logs, Spark shuffle files, the LynxKite data files etc. See Configure directories for more details and fixes.

This can be happen due to a huge set of possible reasons. Logged in administrator users can get a thread dump that reflects LynxKite’s internal state via the url `<LynxKiteUrl>/getThreadDump. (E.g., http://localhost:2200/getThreadDump). This will come in handy when debugging the issue.

This file contains various configurations used when launching a LynxKite instance.

Settings in this file can be overriden using environment variables. See section Using environment variables for more details.

To specify which cluster to use choose one of the following options:

Specify the directory where LynxKite stores metadata about projects. The directory must be on the local file system. Do not forget to set up backups.

Specify the directory of the data where graph data is stored. Should be on a distributed fs (HDFS, S3, etc) unless this is a local setup. Please note that giving the full URI (e.g., file:/…​) is mandatory.

LynxKite can use a faster ephemeral file system for data storage, with reading from the main file system specified by KITE_DATA_DIR as a fallback. The primary use case for this is using SSD-backed HDFS on EC2 while using S3 for the permanent storage.

If KITE_EPHEMERAL_DATA_DIR is set, all data writes go to this directory. KITE_DATA_DIR will be used only for uploaded files.

Optionally set the configuration file to specify what file paths users are allowed to access using what prefixes. Use kite_x.y.z/conf/prefix_definitions_template.txt as a template when creating your custom prefix configuration file. Make sure you put the config file to a LynxKite version independent location (not inside the kite_x.y.z directory, e.g. $HOME/kite_conf). Then set KITE_PREFIX_DEFINITIONS to point to your newly created config file (e.g. the recommended $HOME/kite_conf/prefix_definitions.txt).

Normally, LynxKite can only access the file system via its prefix mechanism, that is, using either the DATA$ prefix (as in DATA$/dir/file.csv) or any other prefixes set up in a prefix definition file. This behavior can be changed by setting KITE_ALLOW_NON_PREFIXED_PATHS to true: then it becomes possible to specify any valid paths such as file:/home/user/kite_data/dir/data.csv or s3n://awskey:awspassword@somedir/somefile. Be careful: all LynxKite users will be able to read and write local and cluster files with the credentials of the LynxKite process. Only allow this in restricted environments.

Specify the YARN configuration directory. It is needed if you want to run against YARN.

By default, LynxKite specifies 15% of executor memory to be the amount allocated for overhead in YARN executor containers. You can set a higher value if YARN is killing your executors for exceeding size, but the executors themselves are not reporting out of memory errors.

YARN offers the possibility of using resource pools. You can specify an existing resource pool and allocate LynxKite to it.

Specify how much memory is available for LynxKite on a single worker machine. Ignored for local setups, use KITE_MASTER_MEMORY_MB for that case.

Specify the number of executors. For standalone cluster it defaults to as many as possible. For a YARN setup, this options is mandatory.

Specify the number of cores per executor LynxKite should use.

Specify how much memory is available for LynxKite on the master machine in megabytes. For local setups this also determines executor memory and EXECUTOR_MEMORY is ignored in that case.

Specify the port for the LynxKite HTTP server to listen on. Must be >=1000.

By default, LynxKite can only be accessed from localhost for security reasons. (That is, address 127.0.0.1). Uncomment the following line if LynxKite should be accessible from other IP addresses as well.

Specify the HTTP port for the watchdog. If this is set, the startup script will start a watchdog as well which will automatically restart the LynxKite server if it detects any problem.

Uncomment this to start an internal watchdog thread inside LynxKite’s driver JVM. This watchdog will kill LynxKite if health checks are failing continuously for the given amount of time.

The LynxKite local temp directory is a local path that exists on all workers and the master and will be used for storing temporary Spark/Hadoop files. This directory can potentially use a lot of space, so if you have a small root filesystem and an extra large drive mounted somewhere then you need to point this to somewhere on the large drive. On the other hand performance of this drive has a significant effect on overall speed, so using an SSD is a nice option here.

Configure the directory LynxKite uses for logging. Defaults to logs under the LynxKite installation directory if not specified.

The LynxKite extra JARS is a colon (:) delimited list of JAR files that should be loaded on the LynxKite CLASSPATH. (It will be loaded on the master and distributed to the workers.)

One typical use case is to configure additional JDBC drivers. To do that, all you need to do is to add the jar file here.

You can enable an interactive Scala interpreter able to access LynxKite internals by using the below exports. You can access the interpreter by SSHing from the host running LynxKite as: ssh ${KITE_AMMONITE_USER}@localhost -p ${KITE_AMMONITE_PORT} and use KITE_AMMONITE_PASSWD as password.

If you do this and do not trust all users who can SSH into this machine (the typical case!) then make sure to modify the file system permissions of .kiterc to be only readable by the user running LynxKite and change the password below.

Options needed if you want to use authentication and HTTPS.

Just use the following configurations with default values for a simple, fake certificate setup.

On a Kerberos-secured Hadoop cluster, set the KERBEROS_PRINCIPAL and KERBEROS_KEYTAB variables. The principal acts like a user name and the keytab file acts like a password.

The setting KITE_MAX_ALLOWED_FILESYSTEM_LIFESPAN_MS forces LynxKite to create a new Hadoop Filesystem object every time KITE_MAX_ALLOWED_FILESYSTEM_LIFESPAN_MS milliseconds have passed. This mechanism prevents certain errors related to Hadoop delegation token expiration. A good value for this is half the time specified in the system’s dfs.namenode.delegation.token.renew-interval setting. That defaults to 1 day, so our default is 12 hours. See this explanation for details.

Uncomment the below lines to export LynxKite’s Spark metrics into a Graphite-compatible monitoring system. You can use this together with tools/monitoring/restart_monitoring_master.sh.

Specify any command line arguments for the jvm that runs the LynxKite driver. (e.g., EXTRA_DRIVER_OPTIONS='-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=50000' will run a local LynxKite in a such a mode that you can attach to it with a debugger.)

Set thread stack size for thread in the executor process. By default, jvm will assign 1 megabyte for each thread, but that is too low for some rdd dependencies. The suffixes M and k can be used to specify megabytes or kilobytes respectively; e.g., 1M or 1500k. This setting does not have an effect in local mode.

Set thread stack size for driver threads. By default, jvm will assign 1 megabyte for each thread, but that is too low for some rdd dependencies. The suffixes M and k can be used to specify megabytes or kilobytes respectively; e.g., 1M or 1500k.

Maximize the number of parallel threads. LynxKite, Spark domain and Sphynx will all use this parameter.

Specify the length of the protected time period in days, while the cleaner does not delete data files. It can be integer or double value.

Anonymous data collection is opt-in by default. This setting can be used to force it on or off on an instance. Accepted values are: optional, always, never

Specify where Sphynx (the single-node server) is running.

Certificate directory for Sphynx, if an encrypted connection is desired.

Specify the PID file for Sphynx.

Specify the directory where Sphynx stores graph data.

Specify the directory where Sphynx stores graph data using vertex ids from the Spark world.

Enable CUDA if you have a compatible GPU and want to use it to process graphs.

Unrestricted Python in LynxKite is a very powerful tool. Since Python execution has full access to the network and file-system, it must be explicitly enabled by the administrator.

Unrestricted R execution in LynxKite is a very powerful tool. Since R execution has full access to the network and file-system, it must be explicitly enabled by the administrator.

A simple chroot-based sandbox for executing users' Python code can improve security in a multi-user environment. To be able to mount directories as read-only, this requires running LynxKite as root. If it’s running in a Docker container, that container must be started with the --privileged flag.

Sphynx can keep entities in memory for high-performance computation. This setting configures how much memory to allocate for this purpose.

The prefix definitions configuration file is used to specify what file paths users are allowed to access using what prefixes. It also serves as a means to provide user access control to certain prefixes if desired.

Special prefixes can be defined here that expand to directories or paths. Suppose, for example, that the administrator specifies DIR="file:/home/user/kite/" in this file. Then whenever the user references DIR$subdir/data.txt (s)he will access the file:/home/users/kite/subdir/data.txt. Moreover, there is no other way a user can refer to that file (unless there are other prefixes defined here that make this possible).

Format:

Lines that begin with a hashmark (#) are comments and ignored.

Prefix symbols should only contain capital letters, underscore, and digits; the first character cannot be a digit. The dollar sign is NOT part of the prefix symbol.

The paths that they define should either be empty, or end in an at sign (@) or end in a slash (/). The defined paths should always start with a file system specification (e.g. s3n: or file:) unless they are empty. The quotation marks are not part of the path, but they should still be supplied.

Local prefixes (the ones starting with file:/) only work on single-machine LynxKite installations. On distributed LynxKite installations, you will need to use a distributed file system, like HDFS or S3.

Examples:

A specification of EMPTY such as the one above gives the user full access to anything; e.g., they can say EMPTY$s3n://key:password@somedir/somefile or EMPTY$file://etc/passwd.