Not all the buttons listed here are accessible at all times, please see the details below on when each function is available.

Instead of clicking on the search bar, you can use the / button. After finding the coveted box, you can press Enter to place the box under your mouse. You can place multiple boxes without leaving the search bar.

Boxes and connected box sequences can be copy-pasted, even to different workspaces and LynxKite instances. A limitation here is that the custom boxes are not copied, so they have to be present on the target instance too.

The copy-paste mechanism is implemented via serializing to YAML, a human-readable and editable textual format, so you can even save box sequences to text files or share them via email. Such a YAML-file (if it has a .yaml extension) can also simply be drag-and-dropped into a LynxKite workspace.

Hold SHIFT while moving a box to align it to a grid.

What if you wanted to compute PageRank for the communities in the graph?

If you want to apply a box to a segmentation, first add the box as normal. Then in the box parameters popup adjust the special Apply to parameter to pick the segmentation. This special parameter is added for all graph-typed inputs, making it possible to work with segmentations (and the segmentations of those segmentations, etc.).

Parametric parameters can reference workspace parameters.

For example, consider a workspace with two Import CSV boxes, one importing accounts-2017.csv and the other importing transactions-2017.csv. You could add a workspace parameter called date with default value 2017. Make the file name parameter of the import boxes parametric by clicking the icon to the right of the parameter input. Change the file name parameters to accounts-$date.csv and transactions-$date.csv. Now 2017 will be substituted for $date, importing the same files as before.

One benefit of this is that you can change the date in a single place (on the anchor box) instead of having to update multiple boxes when the time comes.

Another benefit is that if your workspace is used as a custom box in another workspace, the workspace parameters are specified by the user. Parametric parameters allow you to pass these user-specified parameters on to boxes in the workspace.

Even complex parameters, like a list of vertex attributes, can be toggled to become parametric. In this case the original input field is replaced by a simple text field.

Parametric parameters are evaluated using Scala string interpolation. This means that Scala expressions can be embedded in these parameters. For example, you could write accounts-${date.toInt + 1}.csv.

Unexpected parameters are parameters that have been set at some point on the box, but are no longer recognized.

The list of parameters for many boxes is determined dynamically. For example in Aggregate on neighbors there is one parameter for each vertex attribute. If you have configured an aggregation for attribute X but then changed the input to no longer have an attribute called X, then the parameter that sets aggregation on X becomes an unexpected parameter.

Unexpected parameters are treated as errors. You can click the icon to the right to remove the unexpected parameter. Or you can change the input so that the parameter becomes recognized again.

Click the icon in the popup header to access the box metadata. Click the icon to return to the parameter editor.

The toolbar at the top of the state popup always contains a icon, for saving the state as a snapshot. The snapshot will be saved outside of the workspace, in the directory tree. Snapshots are independent of the workspaces from which they were saved. Use them to share final results, or record intermediate results for comparison.

To save a snapshot you have to specify the full path of the snapshot. A full path in the LynxKite directory system has the following form: top_folder/subfolder_1/subfolder_2/…​/subfolder_n/name
Keep in mind that there is no leading slash at the beginning of the path.

Snapshots can be loaded back into a workspace with an Import snapshot box.

Boxes like Graph visualization, SQL1, Custom plot are essential for looking at your data. It is very natural to want to take a quick look at the data in the middle of a complex workspace.

One option is to quickly create and attach a Graph visualization box, see what the graph looks like at that point, and then delete the box. Instruments are effectively the same, except that no temporary box is added to the workspace. This means instruments can be used even on read-only workspaces.

The instrument buttons are in the popup toolbar. For example, in the last screenshot the buttons for SQL and Visualize are visible, corresponding to the SQL1 and Graph visualization boxes. If you click on SQL, the popup contents are replaced by the box parameters of the SQL1 box at the top and the output state of the SQL1 box at the bottom.

The output state of the instrument once again has a toolbar for snapshotting and applying instruments. This makes it possible to apply one instrument after the other:

Instruments are not saved into the workspace. But they are built from regular boxes, so the same calculations can always be reproduced using conventional boxes.

We use the word graph for a rich type that represents the base graph and its segmentations in one bundle. The popup for a graph shows basic information about the base graph, such as the number of vertices and edges. It lists the attributes and segmentations. Graph attribute values are displayed, attribute histograms are available on click, and segmentations can be opened to dig deeper.

The Graphs chapter gives a more in-depth description of graphs.

Tables are the same in LynxKite as in relational databases and spreadsheet programs: they are a matrix of columns and rows. Tables are the input and output of SQL queries. Graphs can be built from tables via Use table as vertices, Use table as edges, and similar operations.

The plot state is a data visualization created via the Custom plot box, or one of the built-in plotting boxes.

Export boxes, such as Export to CSV, allow you to configure an export operation. The output of these boxes is an export state. It is the export state which actually allows triggering the often resource-intensive computation of creating the output files.

This two-step process avoids accidental exports while editing the workspace. It also provides metadata information about the output, for example a file path. To trigger the export, click on the icon.

Your custom box now has inputs and outputs and can provide useful functionality. Custom boxes can also take parameters. This is configured through the Anchor box of the workspace of the custom box.

You can set the name, type, and default value of the parameters. The following parameter types are supported:

To make use of the custom box’s parameters in the workspace of the custom box, you need to access them from Parametric parameters. Regardless of their type, all the parameters are seen as Strings from the Scala code of the parametric parameters. Use .toInt, .toDouble, .toBoolean on them if you need to do more than simple string substitution.

Once a workspace has been configured as a wizard, clicking it in the workspace browser takes you to the wizard view.

If the In progress setting is disabled in the Anchor box, opening the wizard creates a copy of it. This way multiple users can work off of the same wizard without interfering with each other. The copies will be created with the In progress setting enabled. Opening these copies then will not create further copies.

See our section on Wizards in the workspace browser for more about how wizards look from outside of the workspace.

The simplest way of using Scala to derive attributes is to just provide a one-liner expression in Derive vertex attribute or Derive edge attribute. The examples below are for deriving vertex attributes. The only difference from deriving edge attributes is the way vertex attributes can be accessed.

A simple example:

will generate a constant numeric attribute of value 42.0. You can also use values of other attributes in the expression:

assuming that there is already an age attribute defined. LynxKite can also accept a list of Scala expressions:

In this case, the value of the last expression will be taken as the value of the derived attribute. More complex code can be structured using functions:

LynxKite uses Scala data types internally, so there is no need for type conversion between LynxKite and the derivations script. However, to support persistence, the available types for both input (the type of vertex and edge attributes the script can use) and result are restricted to the following.

Values of other types need to be manually converted before returning from the Scala script. For input types, you can use, for example, either of Convert vertex attribute to String or Convert vertex attribute to number.

Sometimes a vertex (or an edge) does not have any value for a particular attribute. For example, in a Facebook graph, the user’s hometown might or might not be given. In such a case, we say that this attribute is undefined for that particular vertex (or edge). Usually, an undefined value represents the fact that the information is unknown. Indeed, some algorithms (e.g., [predict-attribute-by-viral-modeling]) work on undefined attribute values, and their job is to fill them in with reasonable estimates.

Note that an empty string and an undefined value are two different concepts. Suppose, for example, that a person’s name is represented by three attributes: FirstName, MiddleName, and LastName. In this case, MiddleName could be the empty string (meaning that the person in question has no middle name), or it could be undefined (meaning that their middle name is not known). Thus, the empty string is treated as an ordinary String attribute.

Differences between undefined and defined values:

Fill vertex attributes with constant default values can be used to replace undefined values with a constant. By replacing them with a special value, they can be made part of histograms or filters.

The machine learning algorithm used to create this model.

The name of the attribute that this model is trained to predict. (The dependent variable.)

This will not appear for unsupervised machine learning models.

Details about the pre-processing scaling step applied to the features before training. The two phases are centering and scaling. The first phase (centering) centers the data with mean before scaling, i.e., the mean is subtracted from all elements. The data set acquired this way has a mean of 0. The second phase (scaling) is acquired by dividing all the elements by the standard deviation. The means and deviations in these steps are computed columnwise.

Suppose we have an original data item (a, b). After these two steps, the data item that is used for the training will be ((a-m1)/d1, (b-m2)/d2), where m1 and d1 are the mean and the standard deviation for the first column (the a’s) and m2 and d2 are the mean and the standard deviation for the second column (the b’s).

Note that both steps are optional: it depends on the model, whether they are applied or not.

The list of the feature attributes that this model uses for predictions. (The independent variables.)

For decision tree classification model:

For linear regression and logistic regression models:

For linear regression model:

For logistic regression model:

For KMeans clustering model:

This button is used to select a new set of centers. The vertices placed there will be ones that satisfy all the currently set restrictions (see below). The available options are:

Restrictions narrow down the potential set of candidates that will be chosen when you click on the Pick button. They have the same syntax as filters. (See Filter by attributes.) There are two ways to specify them:

After picking one set of centers with the Pick button the button is replaced by the Next button. Clicking this button will iterate over samples that match the conditions. The samples will show up in a deterministic order. You can skip to an arbitrary sample by clicking on the button. There you can manually enter a position in the sequence and pick it by clicking on Pick by offset.

Bucketed view by default comes in absolute edge density mode. Absolute edge density means the thickness of an edge going from bucket A to bucket B corresponds to the number of edges going from a vertex in bucket A to a vertex in bucket B (or in the weighted case: to the sum of the weights on such edges). This makes the edges going between large buckets typically much thicker than those going between smaller buckets.

Relative edge density, on the other hand, is calculated by dividing the number of edges between bucket A and bucket B by [size of bucket A] × [size of bucket B]. This way, the individual bucket sizes aren’t reflected on the thickness of the edges.

For very large graphs the bucketed view numbers are extrapolated from a sample. Precise calculation would not produce a visible change in the visualization, so most often it is not necessary. It can be desirable however if the numbers from the visualization are to be used in a report.

Click the "approximate counts" option to switch it to "precise counts".

To be able to connect to a database LynxKite requires the JDBC drivers for the database to be installed. LynxKite comes with the JDBC drivers for MySQL, PostgreSQL, and SQLite pre-installed. For accessing other databases you will need to acquire the driver from the vendor. The driver is a jar file. You have to add the full path of the jar file to KITE_EXTRA_JARS in .kiterc and restart LynxKite.

The database for import/export operations is specified via a connection URL. The driver is responsible for interpreting the connection URL. Please consult the documentation for the JDBC driver for the connection URL syntax.

If you are in a controlled network environment, make sure that the LynxKite application and all the Spark executors are allowed to connect to the database server.

These operations import external data to LynxKite. Example: Import CSV.

These operations can build graphs - without importing data to LynxKite. Example: Create example graph.

These operations create subgraphs - a graph formed from a subset of the vertices and edges of the original graph. Example: Filter by attributes.

These operations create Segmentations. Example: Find connected components.

These operations modify Segmentations. Example: Copy edges to base graph.

The operations in this category can change the overall graph structure by adding or discarding vertices and/or edges. Examples: Add reversed edges, and Merge vertices by attribute.

The operations in this category manipulate global graph attributes. For example, Correlate two attributes computes the Pearson-correlation coefficient of two attributes, and stores the result in a graph attribute.

These operations manipulate (create, discard, convert etc.) vertex attributes. These operations perform their task without looking at other edges or vertices and they are not available if the graph has no vertices. Example: Add constant vertex attribute.

These operations are similar to vertex attribute operations, but they manipulate edge attributes. They are not available if the graph has no edges. Example: Add random edge attribute.

These operations compute vertex attributes from attributes of their neighboring elements. They only differ in how we define "neighboring elements". For example, in operation Aggregate to segmentation, these neighboring elements are all the vertices that belong to the same segment (the segment being the vertex whose attribute this operation computes). Another example is Aggregate edge attribute to vertices; in this case the "neighboring elements" are the edges that leave or enter the vertex. Yet another example is Aggregate on neighbors; the "neighboring elements" here are the other vertices connected to the vertex.

Graph computation operations are similar to the vertex (or edge) attribute operations inasmuch as they compute new attributes for each vertex (or edge). However, they are somewhat more complex, since they are not restricted to that single vertex (or edge) in their computation. For example, Compute degree creates a vertex attribute that depends on how many neighbors a given vertex has, so it depends on the neighborhood of the vertex. A more complex example is Compute PageRank, which is not even restricted on the immediate neighborhood of a vertex: it depends on the entire graph. One might say that this category is about metrics that describe the graph structure in some way.

These operations perform machine learning. A machine learning model is trained on a set of data, and it can perform prediction or classification on a new set of data. For example, a logistic regression model can be trained by the operation Train a logistic regression model and it can classify new data with the operation Classify with model.

Utility features to efficiently manage workfows. Examples: Users can add a Comment or create a Graph union.

Utility features to manage and personalize graphs by manipulating (discarding, copying, renaming, etc.) attributes and segmentations. Example: Rename edge attributes.

Visualization features. Examples: users can create charts with Custom plot, or visualize a subset of the graph with Graph visualization.

These operations export data from LynxKite. Example: Export to CSV.

Users can add previously created custom boxes or Built-ins to their workflow by selecting them from the Custom box menu.

Adds an attribute with a fixed value to every edge.

Adds an attribute with a fixed value to every vertex.

Warning! Experimental operation.

Creates a graph with given amount of vertices and average degrees. The edges will follow a power-law - also known as scale-free - distribution and have high clustering. Vertices get two edge attributes called "radial" and "angular" that can later be used for edge strength evaluation or link prediction. The algorithm is based on Popularity versus Similarity in Growing Networks and Network Mapping by Replaying Hyperbolic Growth.

The edges are generated by simulating hyperbolic growth. Vertices are added one by one and at the time of each addition new edges are created in two ways. First, the new vertex is added and it creates edges from itself to older vertices - "external" edges. Then some new edges are added between older vertices - "internal" edges. This way the average amount of edges added per vertex will be slightly more than externalDegree + internalDegree.

Generates a new random numeric attribute with the specified distribution, which can be either (1) a Standard Normal (i.e., Gaussian) distribution with a mean of 0 and a standard deviation of 1, or (2) a Standard Uniform distribution where values fall between 0 and 1.

Generates a new random numeric attribute with the specified distribution, which can be either (1) a Standard Normal (i.e., Gaussian) distribution with a mean of 0 and a standard deviation of 1, or (2) a Standard Uniform distribution where values fall between 0 and 1.

Creates a new vertex attribute that is the rank of the vertex when ordered by the key attribute. Rank 0 will be the vertex with the highest or lowest key attribute value (depending on the direction of the ordering). String attributes will be ranked alphabetically.

This operation makes it easy to find the top (or bottom) N vertices by an attribute. First, create the ranking attribute. Then filter by this attribute.

For every A → B edge adds a new B → A edge, copying over the attributes of the original. Thus this operation will double the number of edges in the graph.

Using this operation you end up with a graph with symmetric edges: if A → B exists then B → A also exists. This is the closest you can get to an "undirected" graph.

Optionally, a new edge attribute (a 'distinguishing attribute') will be created so that you can tell the original edges from the new edges after the operation. Edges where this attribute is 0 are original edges; edges where this attribute is 1 are new edges.

Aggregates edge attributes across the entire graph into one graph attribute for each attribute. For example you could use it to calculate the average call duration across an entire call dataset.

Aggregates an attribute on all the edges going in or out of vertices. For example it can calculate the average duration of calls for each person in a call dataset.

Aggregates vertex attributes across all the segments that a vertex in the base graph belongs to. For example, it can calculate the average size of cliques a person belongs to.

Aggregates across the vertices that are connected to each vertex. You can use the Aggregate on parameter to define how exactly this aggregation will take place: choosing one of the 'edges' settings can result in a neighboring vertex being taken into account several times (depending on the number of edges between the vertex and its neighboring vertex); whereas choosing one of the 'neighbors' settings will result in each neighboring vertex being taken into account once.

For example, it can calculate the average age of the friends of each person.

Aggregates vertex attributes across all the vertices that belong to a segment. For example, it can calculate the average age of each clique.

Aggregates vertex attributes across the entire graph into one graph attribute for each attribute. For example you could use it to calculate the average age across an entire dataset of people.

This special box represents the workspace itself. There is always exactly one instance of it. It allows you to control workspace-wide settings as parameters on this box. It can also serve to anchor your workspace with a high-level description.

Scalable algorithm to calculate the approximate local clustering coefficient attribute for every vertex. It quantifies how close the vertex’s neighbors are to being a clique. In practice a high (close to 1.0) clustering coefficient means that the neighbors of a vertex are highly interconnected, 0.0 means there are no edges between the neighbors of the vertex.

Scalable algorithm to calculate the approximate overlap size of vertex neighborhoods along the edges. If an A → B edge has an embeddedness of N, it means A and B have N common neighbors. The approximate embeddedness is undefined for loop edges.

Bundles the chosen number and Vector[number] attributes into one Vector attribute. By default, LynxKite puts the numeric attributes after each other in alphabetical order and then concatenates the Vector attributes to the resulting Vector in alphabetical order as well. The resulting attribute is undefined where any of the input attributes is undefined.

For example, if you bundle the age, favorite_day and income attributes into a Vector attribute called everything, you end up with the following attributes.

Validates that the segments of the segmentation are in fact cliques.

Creates a new invalid_cliques graph attribute, which lists non-clique segment IDs up to a certain number.

Creates classifications from a model and vertex attributes of the graph. For the classifications with nominal outputs, an additional probability is created to represent the corresponding outcome probability.

Finds a coloring of the vertices of the graph with no two neighbors with the same color. The colors are represented by numbers. Tries to find a coloring with few colors.

Vertex coloring is used in scheduling problems to distribute resources among parties which simultaneously and asynchronously request them. https://en.wikipedia.org/wiki/Graph_coloring

Creates a new segmentation from the selected existing segmentations. Each new segment corresponds to one original segment from each of the original segmentations, and the new segment is the intersection of all the corresponding segments. We keep non-empty resulting segments only. Edges between segmentations are discarded.

If you have segmentations A and B with two segments each, such as:

then the combined segmentation will have four segments:

Adds a comment to the workspace. As with any box, you can freely place your comment anywhere on the workspace. Adding comments does not have any effect on the computation but can potentially make your workflow easier to understand for others — or even for your future self.

Markdown can be used to present formatted text or embed links and images.

Compares the edge sets of two segmentations and computes precision and recall. In order to make this work, the edges of the both segmentation graphs should be matchable against each other. Therefore, this operation only allows comparing segmentations which were created using the Use base graph as segmentation operation from the same graph. (More precisely, a one to one correspondence is needed between the vertices of both segmentations and the base graph.)

You can use this operation for example to evaluate different colocation results against a reference result.

Assortativity is the correlation in the values of an attribute along the edges of the graph. A high assortativity means connected vertices often have similar attribute values.

Uses the NetworKit implementation.

Calculates a centrality metric for every vertex. Higher centrality means that the vertex is more embedded in the graph. Multiple different centrality measures have been defined in the literature. You can choose the specific centrality measure as a parameter to this operation.

Calculates the local clustering coefficient attribute for every vertex. It quantifies how close the vertex’s neighbors are to being a clique. In practice a high (close to 1.0) clustering coefficient means that the neighbors of a vertex are highly interconnected, 0.0 means there are no edges between the neighbors of the vertex.

Computes a scalar for a non-overlapping segmentation. Coverage is the fraction of edges that connect vertices within the same segment.

Uses the NetworKit implementation.

For every vertex, this operation calculates either the number of edges it is connected to or the number of neighboring vertices it is connected to. You can use the Count parameter to control this calculation: choosing one of the 'edges' settings can result in a neighboring vertex being counted several times (depending on the number of edges between the vertex and the neighboring vertex); whereas choosing one of the 'neighbors' settings will result in each neighboring vertex counted once.

The diameter of a graph is the maximal shortest-distance path length between two vertices. All vertex pairs are at most this far from each other.

Uses the NetworKit implementation.

Calculates the extent to which two people’s mutual friends are not themselves well-connected. The dispersion attribute for an A → B edge is the number of pairs of nodes that are both connected to A and B but are not directly connected to each other.

Dispersion ignores edge directions.

It is a useful signal for identifying romantic partnerships — connections with high dispersion — according to Romantic Partnerships and the Dispersion of Social Ties: A Network Analysis of Relationship Status on Facebook.

A normalized dispersion metric is also generated by this operation. This is normalized against the embeddedness of the edge with the formula recommended in the cited article. (disp(u,v)^0.61/(emb(u,v)+5)) It does not necessarily fall in the (0,1) range.

Calculates the length of the shortest path from a given set of vertices for every vertex. To use this operation, a set of starting v_i vertices has to be specified, each with a starting distance sd(v_i). Edges represent a unit distance by default, but this can be overridden using an attribute. This operation will compute for each vertex v_i the smallest distance from a starting vertex, also counting the starting distance of the starting vertex: d(v_i) = min_j(sd(v_j) + D(s_j, v_i, I)) where D(x, y, I) is the length of the shortest path between x and y using at most I edges.

For example, vertices can be cities and edges can be flights with a given cost between the cities. Given a set of starting cities, which might as well be only one city, this operation can compute the lowest cost for reaching each city with a given maximum number of flight changes. In addition to that, an optional base cost can be specified for each starting city, which will be counted into each path starting from that city. For example, that could be the price of getting to the given city by train.

If a city can be reached from more than one of the starting cities, then still only one cost value is computed: the one from the starting city where the route has the lowest cost. If a starting city can be reached from another starting city in a cheaper way than the starting cost, then the assigned cost of that city will be the cheaper cost.

Computes a scalar for a non-overlapping segmentation. Edge cut is the total weight of the edges going between different segments.

Uses the NetworKit implementation.

The effective diameter is a distance within which a given portion of vertices can be found from each other.

For example, at most six degrees of separation are between most people on Earth. There may be hermits and lost tribes that would push the true diameter above 6, but they are a minority. If we ignore 1% of the population and find that the remaining 99% have a true diameter of 6, we can say that the graph has an effective diameter of 6.

Uses the exact and estimated NetworKit implementations.

Edge embeddedness is the overlap size of vertex neighborhoods along the edges. If an A → B edge has an embeddedness of N, it means A and B have N common neighbors.

Computes the hub dominance metric for each segment in a segmentation. The hub dominance of a segment is the highest internal degree in the segment divided by the highest possible internal degree. (The segment size minus one.)

If a segment has a vertex that is connected to all other vertices in that segment then its hub dominance will be 1. This metric is useful for comparing the structures that make up the different segments in a segmentation.

For further analysis and theory see Characterizing the community structure of complex networks by Lancichinetti et al.

Uses the NetworKit implementation.

Warning! Experimental operation.

Adds edge attribute hyperbolic edge probability based on hyperbolic distances between vertices. This indicates how likely that edge would be to exist if the input graph was probability x similarity-grown. On a general level it is a metric of edge strength. Probabilities are guaranteed to be 0 =< p =< 1 . Vertices must have two number vertex attributes to be used as radial and angular coordinates. The algorithm is based on Network Mapping by Replaying Hyperbolic Growth.

Executes custom Python code to define new vertex, edge, or graph attributes, to transform a table, or to create plots and interactive visualizations.

Computing new attributes

The following example computes two new vertex attributes (with_title and age_squared), two new edge attributes (score and names), and two new graph_attributes (hello and average_age). (You can try it on the example graph which has the attributes used in this code.)

graph_attributes is a SimpleNamespace that makes it easy to get and set graph attributes.

vs (for "vertices") and es (for "edges") are both Pandas DataFrames. You can write natural Python code and use the usual APIs and packages to compute new attributes. Pandas and Numpy are already imported as pd and np. es can have src and dst columns which are the indexes of the source and destination vertex for each edge. These can be used to index into vs as in the example.

Assign the new columns to these same DataFrames to output new vertex or edge attributes.

When you write this Python code, the input data may not be available yet. And you may want to keep building on the output of the box without having to wait for the Python code to execute. To make this possible, LynxKite has to know the inputs and outputs of your code without executing it. You can specify them through the Inputs and Outputs parameters. For outputs you must also declare their types.

The currently supported types for outputs are:

In the first example we would set:

Alternatively, you can let LynxKite infer the inputs and outputs from the code. In this case you still need to specify the type of the outputs, but you can do so in the code using a type annotation. For example:

(This inference is based on simple regular expression parsing of the code and does not cover all possibilities. Please list the inputs and outputs explicitly if the inference fails.)

Working with vectors

Vector-typed attributes are still stored as single columns in the vs and es DataFrames.

To output a vector-typed attribute use v.tolist():

On the example graph this would output:

To use this attribute in another Python box use np.stack(v):

Working with tables

If the input of the box is a table, the output will be a transformed table. It works the same way as with graphs, but with a single DataFrame called df:

With graphs you can only change attributes, but cannot add or remove nodes and edges. There is no such limitation with tables.

It’s also possible to output a table when your input is a graph.

Creating plots

Instead of outputting a modified graph or table, you can choose to output a plot created in Python. To do this, set the outputs to matplotlib or html.

With the matplotlib setting you can use Matplotlib or any library backed by Matplotlib to create your plots. For example you can just write vs.plot(). The plot is produced as an SVG, so it will look crisp in any resolution.

The html setting can be used with Python packages that produce custom HTML visualizations. You can use libraries like Deck.gl, Plotly, or py3Dmol to create interactive views of maps, molecules, and anything else.

How exactly you get the HTML depends on the library. Here are a few examples:

The HTML output also allows you to safely embed any custom HTML you may produce. Just assign the HTML content as a string to the html variable.

Executes custom R code to define new vertex, edge, or graph attributes, to transform a table, or to create plots and interactive visualizations.

Triggers the computations for all entities associated with its input.

Computes a scalar for a non-overlapping segmentation. If the vertices were connected randomly while preserving the degrees, a certain fraction of all edges would fall within each segment. We subtract this from the observed fraction of edges that fall within the segments. Modularity is the total observed difference.

A modularity of 0 means the relationship between internal edges and external edges is consistent with randomly selected edges or segments. A positive modularity means more internal edges than would be expected by chance. A negative modularity means less internal edges than would be expected by chance.

Uses the NetworKit implementation.

Calculates PageRank for every vertex. PageRank is calculated by simulating random walks on the graph. Its PageRank reflects the likelihood that the walk leads to a specific vertex.

Let’s imagine a social graph with information flowing along the edges. In this case high PageRank means that the vertex is more likely to be the target of the information.

Similarly, it may be useful to identify information sources in the reversed graph. Simply reverse the edges before running the operation to calculate the reverse PageRank.

Runs on a GPU using RAPIDS cuGraph if KITE_ENABLE_CUDA=yes is set.

Computes the conductance of each segment in a non-overlapping segmentation. The conductance of a segment is the number of edges going between the segment and the rest of the graph divided by sum of the degrees in the segment or the rest of the graph (whichever is smaller).

A high conductance value indicates a segment that is strongly connected to the rest of the graph. A value over 0.5 means more edges going out of the segment than edges inside it.

See Experiments on Density-Constrained Graph Clustering by Görke et al for details and analysis.

Uses the NetworKit implementation.

Computes the density of each segment in a non-overlapping segmentation. The density of a segment is the number of internal edges divided by the number of possible internal edges.

Uses the NetworKit implementation.

Computes the expansion of each segment in a non-overlapping segmentation. The expansion of a segment is the number of edges going between the segment and the rest of the graph divided by the number of vertices in the segment or in the rest of the graph (whichever is smaller).

A high expansion value indicates a segment that is strongly connected to the rest of the graph. A value over 1 means the vertices in this segment have more than one external neighbor on average.

See Experiments on Density-Constrained Graph Clustering by Görke et al for details and analysis.

Uses the NetworKit implementation.

Computes the fragmentation of each segment in a non-overlapping segmentation. The fragmentation of a segment is one minus the ratio of the size of its largest component and the whole segment.

A segment that is entirely connected will have a fragmentation of zero. If the fragmentation approaches one, it will be made up of smaller and smaller components.

Uses the NetworKit implementation.

Computes the stability of each segment in a non-overlapping segmentation. A vertex is considered stable if it has more neighbors inside the segment than outside. The stability of a segment is the fraction of its vertices that are stable.

A high stability value (close to 1) indicates a segment where vertices are more connected internally than externally. A stability lower than 0.5 means that the majority of neighbors are external for more than half of the vertices.

Uses the NetworKit implementation.

Creates edges between vertices that are equal in a chosen attribute. If the source attribute of A equals the destination attribute of B, an A → B edge will be generated.

The two attributes must be of the same data type.

For example, if you connect nodes based on the "name" attribute, then everyone called "John Smith" will be connected to all the other "John Smiths".

Converts the selected String typed edge attributes to the number type.

The attributes will be converted in-place. If you want to keep the original String attribute as well, make a copy first!

Converts the selected edge attributes to String type.

The attributes will be converted in-place. If you want to keep the original String attribute as well, make a copy first!

Converts the selected String typed vertex attributes to the number type.

The attributes will be converted in-place. If you want to keep the original String attribute as well, make a copy first!

Converts the selected vertex attributes to String type.

The attributes will be converted in-place. If you want to keep the original attributes as well, make a copy first!

Creates a copy of an edge attribute.

Copies the edges from a segmentation to the base graph. The copy is performed along the links between the segmentation and the base graph. If two segments are connected with some edges, then each edge will be copied to each pairs of members of the segments.

This operation has a potential to create trillions of edges or more. The number of edges created is the sum of the source and destination segment sizes multiplied together for each edge in the segmentation. It is recommended to drop very large segments before running this computation.

Copies the edges from the base graph to the segmentation. The copy is performed along the links between the base graph and the segmentation. If a base vertex belongs to no segments, its edges will not be found in the result. If a base vertex belongs to multiple segments, its edges will have multiple copies in the result.

This operation can take a graph attribute from another graph and copy it to the current graph.

It can be useful if we trained a machine learning model in one graph, and would like to apply this model in another graph for predicting undefined attribute values.

Creates a copy of a graph attribute.

Creates a copy of a segmentation.

Creates a copy of a vertex attribute.

Copies all vertex attributes from the segmentation to the parent.

This operation is only available when each vertex belongs to just one segment. (As in the case of connected components, for example.)

Copies all vertex attributes from the parent to the segmentation.

This operation available only when each segment contains just one vertex.

Calculates the Pearson correlation coefficient of two attributes. Only vertices where both attributes are defined are considered.

Note that correlation is undefined if at least one of the attributes is a constant.

Creates a graph in which the distribution of vertex degrees is as specified.

Uses the NetworKit implementation.

Creates a random graph using the Barabási–Albert model. The vertices are created one by one and connected to a set number of randomly chosen previously created vertices. This ensures a skewed degree distribution with "older" vertices tending to have a higher degree.

Uses the NetworKit implementation.

Creates a random graph with a given number of clusters. It randomly places each vertex into one of the clusters then adds an edge for each vertex pair with the given probabilities.

Uses the NetworKit implementation.

Creates a planar random graph with a power-law distribution. Starts with a triangle and in each step adds a new node that is connected to the two endpoints of a randomly selected edge.

See Modern architecture of random graphs: Constructions and correlations by Dorogovtsev et al.

Uses the NetworKit implementation.

Connects vertices in the base graph if they co-occur in any segments. Multiple co-occurrences will result in multiple parallel edges. Loop edges are generated for each segment that a vertex belongs to. The attributes of the segment are copied to the edges created from it.

This operation has a potential to create trillions of edges or more. The number of edges created is the sum of squares of the segment sizes. It is recommended to drop very large segments before running this computation.

Connects segments with large enough overlaps.

Creates a random graph using the Erdős–Rényi model. In this model each pair of vertices is connected independently with the same probability. It creates a very uniform graph with no tendency to skewed degree distributions or clustering.

Uses the NetworKit implementation.

Creates small test graph with 4 people and 4 edges between them.

Executes custom Python code to define a graph. Ideal for creating complex graphs programmatically and for loading datasets in non-standard formats.

The following example creates a small graph with some attributes.

graph_attributes is a SimpleNamespace which makes it easy to set graph attributes.

vs (for "vertices") and es (for "edges") are both Pandas DataFrames. You can write natural Python code and use the usual APIs and packages to construct them. Pandas and Numpy are already imported as pd and np. es must have src and dst columns which are the indexes of the source and destination vertex for each edge. These can be used to index into vs as in the example.

Like all operations in LynxKite, this code is executed only when the results are needed. But we need to know the type of the attributes even before that, so that we can continue building the workspace. To make this possible, you have to specify the type of the outputs through the Inputs and Outputs parameters.

The currently supported types for outputs are:

In the previous example we would set:

Alternatively, you can let LynxKite infer the outputs from the code. In this case you still need to specify the type of the outputs, but you can do so in the code using a type annotation.

This code is equivalent to the first one, but declares the attribute types in the code, and uses the empty DataFrames that vs and es are initialized with:

(This inference is based on simple regular expression parsing of the code and does not cover all possibilities. Please list the outputs explicitly if the inference fails.)

Working with vectors

Vector-typed attributes are still stored as single columns in the vs and es DataFrames.

To output a vector-typed attribute use v.tolist():

Executes custom R code to define a graph. Ideal for creating complex graphs programmatically and for loading datasets in non-standard formats.

The following example creates a small graph with some attributes.

graph_attributes is an object you can use to get and set graph attributes.

vs (for "vertices") and es (for "edges") are both tibbles. You can write natural R code and use the usual APIs and packages to compute new attributes. Dplyr is already imported. es must have src and dst columns which are the indexes of the source and destination vertex for each edge.

Like all operations in LynxKite, this code is executed only when the results are needed. But we need to know the type of the attributes even before that, so that we can continue building the workspace. To make this possible, you have to specify the type of the outputs through the Inputs and Outputs parameters.

The currently supported types for outputs are:

In the previous example we would set:

Working with vectors

To output a vector-typed attribute, you can create a list column or a matrix:

Creates a random graph based on randomly placed points on the hyperbolic plane. The points corresponding to vertices are placed on a disk. If two points are closer than a threshold (by the hyperbolic distance metric), an edge will be created between those two vertices.

The motivation for this is to reflect popularity (how close the point is to the center) and interest (in which direction the point lies). This leads to realistic clustering properties in the generated random graph.

The radius of the disk and the neighborhood radius can be chosen to ensure a desired average and power-law exponent for the degree distribution.

Instead of a strict neighborhood radius, within which edges are always created and outside of which they never are, we can also consider probabilistic edge generation. In this case the shorter the distance between two points, the more likely that an edge should be generated.

The temperature parameter is defined in a way that makes the strict neighborhood radius case an edge case (T=0) and we can smoothly increase the influence of randomness by increasing the temperature.

See Hyperbolic Geometry of Complex Networks by Krioukov et al for more details.

Uses the NetworKit implementation.

LFR stands for Lancichinetti, Fortunato, and Radicchi, the authors of Benchmark graphs for testing community detection algorithms and Benchmarks for testing community detection algorithms on directed and weighted graphs with overlapping communities upon which this generator is based.

The LFR random graph features overlapping communities. Each vertex is randomized into multiple communities while ensuring a desired power-law community size distribution. Then edges within communities are generated to match the desired power-law vertex degree distribution. Finally edges are swapped around to create cross-community connections.

Uses the NetworKit implementation.

Creates a random graph as described in Modelling Spatial Structures by Mocnik et al. The model is based on randomly placing the vertices in Euclidean space and generating edges with a higher probability for pairs of vertices that are closer together.

Uses the NetworKit implementation.

Creates a random graph using the model described in A distributed diffusive heuristic for clustering a virtual P2P supercomputer by Gehweiler et al.

The vertices are randomly placed in a 2-dimensional unit square with a torus topology. Vertices within a set radius are connected when permitted by the maximum degree constraint.

Some dense circular areas within the unit suqare are picked at the beginning and these are populated first. Any remaining vertices are then placed uniformly. This leads to a clustering effect that models the internal networks of companies and institutions as observed in real peer-to-peer network topologies.

Uses the NetworKit implementation.

Creates edges randomly, so that each vertex will have a degree uniformly chosen between 0 and 2 × the provided parameter.

For example, you can create a random graph by first applying operation Create vertices and then creating the random edges.

Creates edges randomly so that the resulting graph is scale-free.

This is an iterative algorithm. We start with one edge per vertex and in each iteration the number of edges gets approximately multiplied by Per iteration edge number multiplier.

Creates a new vertex set with no edges. Two attributes are generated: id and ordinal. id is the internal vertex ID, while ordinal is an index for the vertex: it goes from zero to the vertex set size.

Creates a plot from the input table. The plot can be defined using the Vega-Lite JSON API.

The data field in the Vega-Lite specification will be automatically filled in by LynxKite, giving you access to the input table. (Limited to 10,000 rows.)

LynxKite comes with several Built-ins, many of them based on the Custom plot box. You can dive into these custom boxes to see the code used to build them.

Connect vertices in the base graph with segments based on matching attributes.

This operation can be used (among other things) to create connections between two graphs once one has been imported as a segmentation of the other. (See Use other graph as segmentation.)

Derives a new column on a table input via an SQL expression. Outputs a table.

Generates a new attribute based on existing attributes. The value expression can be an arbitrary Scala expression, and it can refer to existing attributes on the edge as if they were local variables. It can also refer to attributes of the source and destination vertex of the edge using the format src$attribute and dst$attribute.

For example you can write weight * scala.math.abs(src$age - dst$age) to generate a new attribute that is the weighted age difference of the two endpoints of the edge.

You can also refer to graph attributes in the Scala expression. For example, assuming that you have a graph attribute age_average, you can use the expression if (src$age < age_average / 2 && dst$age > age_average * 2) 1.0 else 0.0 to identify connections between relatively young and relatively old people.

Back quotes can be used to refer to attribute names that are not valid Scala identifiers.

The Scala expression can return any of the following types: - String, - Double, which will be presented as number - Int, which will be automatically converted to Double - Long, which will be automatically converted to Double - `Vector`s or `Set`s combined from the above.

In case you do not want to define the output for every input, you can return an Option created from the above types. E.g. if (income > 1000) Some(age) else None.

Generates a new graph attribute based on existing graph attributes. The value expression can be an arbitrary Scala expression, and it can refer to existing graph attributes as if they were local variables.

For example you could derive a new graph attribute as something_sum / something_count to get the average of something.

Generates a new attribute based on existing vertex attributes. The value expression can be an arbitrary Scala expression, and it can refer to existing attributes as if they were local variables.

For example you can write age * 2 to generate a new attribute that is the double of the age attribute. Or you can write if (gender == "Male") "Mr " + name else "Ms " + name for a more complex example.

You can also refer to graph attributes in the Scala expression. For example, assuming that you have a graph attribute income_average, you can use the expression if (income > income_average) 1.0 else 0.0 to identify people whose income is above average.

Back quotes can be used to refer to attribute names that are not valid Scala identifiers.

The Scala expression can return any of the following types: - String, - Double, which will be presented as number - Int, which will be automatically converted to Double - Long, which will be automatically converted to Double - `Vector`s or `Set`s combined from the above.

In case you do not want to define the expression for every input, you can return an Option created from the above types. E.g. if (income > 1000) Some(age) else None.

Throws away edge attributes.

Throws away all edges. This implies discarding all edge attributes too.

Throws away graph attributes.

Discards edges that connect a vertex to itself.

Throws away a segmentation value.

Throws away vertex attributes.

Creates a vertex embedding using the PyTorch Geometric implementation of the node2vec algorithm.

Exports edge attributes from a graph in LynxKite to a corresponding graph in Neo4j.

The relationships in Neo4j are identified by a key property (or properties). You must have a corresponding edge attribute in LynxKite by the same name. This will be used to find the right relationship to update in Neo4j.

The properties of the Neo4j relationships will be updated with the exported edge attributes using a Cypher query like this:

In the event of duplicate keys on either end this will update the properties of all the matching Neo4j relationships with the values from the last matching LynxKite edge.

Exports a graph from LynxKite to Neo4j. The whole graph will be copied to Neo4j with all attributes. No existing data is modified in Neo4j.

A !LynxKite export timestamp property is added to each new node and relationship in Neo4j. This helps clean up the export if needed.

The Cypher query to export nodes is, depending on whether an attribute specifies the node labels:

The Cypher query to export relationships is, depending on whether an attribute specifies the relationship types:

Apache AVRO is a row-oriented remote procedure call and data serialization framework.

CSV stands for comma-separated values. It is a common human-readable file format where each record is on a separate line and fields of the record are simply separated with a comma or other delimiter. CSV does not store data types, so all fields become strings when importing from this format.

Export data to a Delta table.

Export a table directly to Apache Hive.

JDBC is used to connect to relational databases such as MySQL. See Database connections for setup steps required for connecting to a database.

JSON is a rich human-readable data format. It produces larger files than CSV but can represent data types. Each line of the file stores one record encoded as a JSON object.

Apache ORC is a columnar data storage format.

Apache Parquet is a columnar data storage format.

Exports vertex attributes from a graph in LynxKite to a corresponding graph in Neo4j.

The nodes in Neo4j are identified by a key property (or properties). You must have a corresponding vertex attribute in LynxKite by the same name. This will be used to find the right nodes to update in Neo4j.

The properties of the Neo4j nodes will be updated with the exported vertex attributes using a Cypher query like this:

In the event of duplicate keys on either end this will update the properties of all the matching Neo4j nodes with the values from the last matching LynxKite vertex.

Exposes the internal edge ID as an attribute. Useful if you want to identify edges, for example in an exported dataset.

Exposes the internal vertex ID as an attribute. This attribute is automatically generated by operations that generate new vertex sets. (In most cases this is already available as attribute ‘id’.) But you can regenerate it with this operation if necessary.

This box represents computation outside of LynxKite. See the @external decorator in the Python API.

This box represents computation outside of LynxKite. See the @external decorator in the Python API.

This box represents computation outside of LynxKite. See the @external decorator in the Python API.

This box represents computation outside of LynxKite. See the @external decorator in the Python API.

This box represents computation outside of LynxKite. See the @external decorator in the Python API.

This box represents computation outside of LynxKite. See the @external decorator in the Python API.

This box represents computation outside of LynxKite. See the @external decorator in the Python API.

This box represents computation outside of LynxKite. See the @external decorator in the Python API.

This box represents computation outside of LynxKite. See the @external decorator in the Python API.

This box represents computation outside of LynxKite. See the @external decorator in the Python API.

An attribute may not be defined on every edge. This operation sets a default value for the edges where it was not defined.

An attribute may not be defined on every vertex. This operation sets a default value for the vertices where it was not defined.

Keeps only vertices and edges that match the specified filters.

You can specify filters for multiple attributes at the same time, in which case you will be left with vertices/edges that match all of your filters.

Regardless of the exact the filter, whenever you specify a filter for an attribute you always restrict to those edges/vertices where the attribute is defined. E.g. if say you have a filter requiring age > 10, then you will only keep vertices where age attribute is defined and the value of age is more than ten.

The filtering syntax depends on the type of the attribute in most cases.

Filters a graph or table with SQL expressions.

This has the same effect as using a SQL box with select * from vertices where <FILTER> and select * from edge_attributes where <FILTER> and then recombining the tables into a graph. But it is more efficient.

When used with a table input it is identical to a SQL box with select * from input where <FILTER>. But it saves a bit of typing.

Uses the label propagation algorithm to identify communities in the graph. The communities are represented as a segmentation on the graph.

Label propagation starts with assigning a unique label to each vertex. Then each vertex takes on the most common label in their neighborhood. This step is repeated until the labels stabilize.

Uses the NetworKit implementations of PLP and LPDegreeOrdered.

Uses the Louvain method to identify communities in the graph. The communities are represented as a segmentation on the graph.

The Louvain method is a greedy optimization toward maximal modularity. High modularity means many edges within communities and few edges between communities. Specifically we compare the edge counts to what we would expect if the clusters were chosen at random.

Runs on a GPU using RAPIDS cuGraph if KITE_ENABLE_CUDA=yes is set. Uses the NetworKit implementation otherwise.

Creates a segment for every connected component of the graph.

Connected components are maximal vertex sets where a path exists between each pair of vertices.

Runs on a GPU using RAPIDS cuGraph if KITE_ENABLE_CUDA=yes is set.

Creates a segmentation of overlapping communities.

The algorithm finds maximal cliques then merges them to communities. Two cliques are merged if they sufficiently overlap. More details can be found in Information Communities: The Network Structure of Communication.

It often makes sense to filter out high degree vertices before detecting communities. In a social graph real people are unlikely to maintain thousands of connections. Filtering high degree vertices out is also known to speed up the algorithm significantly.

If we deleted all parts of a graph outside of the k-core, all vertices would still have a degree of at least k. More visually, the 0-core is the whole graph. If we discard the isolated vertices we get the 1-core. If we repeatedly discard all degree-1 vertices, we get the 2-core. And so on.

Read more on Wikipedia.

This operation outputs the number of the highest core that each vertex belongs to as a vertex attribute.

Runs on a GPU using RAPIDS cuGraph if KITE_ENABLE_CUDA=yes is set.

Creates a segmentation of vertices based on the maximal cliques they are the member of. A maximal clique is a maximal set of vertices where there is an edge between every two vertex. Since one vertex can be part of multiple maximal cliques this segmentation might be overlapping.

Tries to find a partitioning of the vertices with high modularity.

Edges that go between vertices in the same segment increase modularity, while edges that go from one segment to the other decrease modularity. The algorithm iteratively merges and splits segments and moves vertices between segments until it cannot find changes that would significantly improve the modularity score.

Finds the minimum (or maximum) spanning tree in a graph. The edges marked by the emitted edge attribute (in_tree by default) form a tree for each component in the graph. This tree will have the lowest (or highest) possible total edge weight.

Uses the NetworKit implementation.

Given a directed graph in which each vertex has two associated quantities, the "gain", and the "root cost", and each edge has an associated quantity, the "cost", this operation will yield a forest (a set of trees) that is a subgraph of the given graph. Furthermore, in this subgraph, the sum of the gains minus the sum of the (edge and root) costs approximate the maximal possible value.

Finding this optimal subgraph is called the Prize-collecting Steiner Tree Problem.

The operation will result in four outputs: (1) A new edge attribute, which will specify which edges are part of the optimal solution. Its value will be 1.0 for edges that are part of the optimal forest and not defined otherwise; (2) A new vertex attribute, which will specify which vertices are part of the optimal solution. Its value will be 1.0 for vertices that are part of the optimal forest and not defined otherwise. (3) A new graph attribute that contains the net gain, that is, the total sum of the gains minus the total sum of the (edge and root) costs; and (4) A new vertex attribute that will specify the root vertices in the optimal solution: it will be 1.0 for the root vertices and not defined otherwise.

Creates a segment for every triangle in the graph. A triangle is defined as 3 pairwise connected vertices, regardless of the direction and number of edges between them. This means that triangles with one or more multiple edges are still only counted once, and the operation does not differentiate between directed and undirected triangles. Since one vertex can be part of multiple triangles this segmentation might be overlapping.

In a graph that has two different String identifier attributes (e.g. Facebook ID and MSISDN) this operation will match the vertices that only have the first attribute defined with the vertices that only have the second attribute defined. For the well-matched vertices the new attributes will be added. (For example if a vertex only had an MSISDN and we found a matching Facebook ID, this will be saved as the Facebook ID of the vertex.)

The matched vertices will not be automatically merged, but this can easily be performed with the Merge vertices by attribute operation on either of the two identifier attributes.

This operation allows the user to join (i.e., carry over) attributes from one graph to another one. This is only allowed when the target of the join (where the attributes are taken to) and the source (where the attributes are taken from) are compatible. Compatibility in this context means that the source and the target have a "common ancestor", which makes it possible to perform the join. Suppose, for example, that operation Take edges as vertices have been applied, and then some new vertex attributes have been computed on the resulting graph. These new vertex attributes can now be joined back to the original graph (that was the input for Take edges as vertices), because there is a correspondence between the edges of the original graph and the vertices that contain the newly computed vertex attributes.

Conversely, the edges and the vertices of a graph will not be compatible (even if the number of edges is the same as the number of vertices), because no such correspondence can be established between the edges and the vertices in this case.

Additionally, it is possible to join segmentations from another graph. This operation has an additional requirement (besides compatibility), namely, that both the target of the join (the left side) and the source be vertices (and not edges).

Please, bear it in mind that both attributes and segmentations will overwrite the original attributes and segmentations on the right side in case there is a name collision.

When vertex attributes are joined, it is also possible to copy over the edges from the source graph (provided that the source graph has edges). In this case, the original edges in the target graph are dropped, and the source edges (along with their attributes) will take their place.

The resulting graph is just a disconnected graph containing the vertices and edges of the two originating graphs. All vertex and edge attributes are preserved. If an attribute exists in both graphs, it must have the same data type in both.

The resulting graph will have as many vertices as the sum of the vertex counts in the two source graphs. The same with the edges.

Segmentations are discarded.

Creates a visualization from the input graph. You can use the box parameter popup to define the parameters and layout of the visualization. See Graph visualizations for more details.

Grows the segmentation along edges of the parent graph.

This operation modifies this segmentation by growing each segment with the neighbors of its elements. For example if vertex A is a member of segment X and edge A → B exists in the original graph then B also becomes the member of X (depending on the value of the direction parameter).

This operation can be used together with Use base graph as segmentation to create a segmentation of neighborhoods.

Uses the SHA-256 algorithm to hash an attribute: all values of the attribute get replaced by a seemingly random value. The same original values get replaced by the same new value and different original values get (almost certainly) replaced by different new values.

Treat the salt like a password for the data. Choose a long string that the recipient of the data has no chance of guessing. (Do not use the name of a person or project.)

The salt must begin with the prefix SECRET( and end with ), for example SECRET(qCXoC7l0VYiN8Qp). This is important, because LynxKite will replace such strings with three asterisks when writing log files. Thus, the salt cannot appear in log files. Caveat: Please note that the salt must still be saved to disk as part of the workspace; only the log files are filtered this way.

To illustrate the mechanics of irreversible hashing and the importance of a good salt string, consider the following example. We have a data set of phone calls and we have hashed the phone numbers. Arthur gets access to the hashed data and learns or guesses the salt. Arthur can now apply the same hashing to the phone number of Guinevere as was used on the original data set and look her up in the graph. He can also apply hashing to the phone numbers of all the knights of the round table and see which knight has Guinevere been making calls to.

Apache AVRO is a row-oriented remote procedure call and data serialization framework.

CSV stands for comma-separated values. It is a common human-readable file format where each record is on a separate line and fields of the record are simply separated with a comma or other delimiter. CSV does not store data types, so all fields become strings when importing from this format.

Import a Delta Table.

Import a table from BigQuery without a SQL query. (See also Import from BigQuery (Standard SQL result).)

Some BigQuery-specific datatypes like ARRAY<STRUCT> are not fully supported within LynxKite.

BigQuery access is provided through the Apache Spark SQL connector for Google BigQuery.

Execute a BigQuery Standard SQL query and get the result as a table. (See also Import from BigQuery (raw table).)

Some BigQuery-specific datatypes like ARRAY<STRUCT> are not fully supported within LynxKite.

BigQuery access is provided through the Apache Spark SQL connector for Google BigQuery.

Import an Apache Hive table directly to LynxKite.

Imports raw data from Neo4j data files. The nodes or relationships are output as table. To reconstruct the graph, you can use one Import from Neo4j files box to get the nodes, another to get the relationships, then the Use table as vertices and Use table as edges boxes to construct a graph from the two tables.

How is "Import from Neo4j files" different from "Import from Neo4j"?

Import from Neo4j talks to a running Neo4j instance and outputs a graph. Import from Neo4j files (this box) requires no running Neo4j instance — it reads the Neo4j storage files directly. And it outputs a table for nodes or relationships.

Import from Neo4j is more convenient to use. But it requires both Neo4j and LynxKite to be provisioned sufficiently to allow building a graph from the full dataset. Import from Neo4j files on the other hand avoids constructing a graph either on the Neo4j side or on the LynxKite side. It can run successfully with minimal resources. It’s great for huge datasets.

Import a graph from the Neo4j graph database.

Neo4j does not have a strict schema. Different nodes may have different attributes. In LynxKite the list of vertex attributes is defined for the whole graph. But each vertex may leave any attribute undefined.

If you import Neo4j nodes that have different attributes, such as movies that have a title and actors that have a name, the resulting graph will have both title and name attributes. title will only be defined on movies, name will only be defined on actors.

The same happens with edges.

If multiple node types have attributes of the same name, those attributes need to have the same type. If this is not the case, you can narrow down the query by node label.

JDBC is used to connect to relational databases such as MySQL. See Database connections for setup steps required for connecting to a database.

JSON is a rich human-readable data format. JSON files are larger than CSV files but can represent data types. Each line of the file in this format stores one record encoded as a JSON object.

Apache ORC is a columnar data storage format.

Apache Parquet is a columnar data storage format.

Makes a previously saved snapshot accessible from the workspace.

Makes the union of a list of previously saved table snapshots accessible from the workspace as a single table.

The union works as the UNION ALL command in SQL and does not remove duplicates.

Gives easy access to graph datasets commonly used for benchmarks.

See the PyTorch Geometric documentation for details about the specific datasets.

This special box represents an input that comes from outside of this workspace. This box will not have a valid output on its own. When this workspace is used as a custom box in another workspace, the custom box will have one input for each input box. When the inputs are connected, those input states will appear on the outputs of the input boxes.

Input boxes without a name are ignored. Each input box must have a different name.

See the section on Custom boxes on how to use this box.

Finds the best matching between a base graph and a segmentation. It considers a base vertex A and a segment B a good "match" if the neighborhood of A (including A) is very connected to the neighborhood of B (including B) according to the current connections between the graph and the segmentation.

The result of this operation is a new edge set between the base graph and the segmentation, that is a one-to-one matching.

For every position vertex attribute looks up features in a Shapefile and returns a specified attribute.

Shapefiles can be obtained from various sources, like OpenStreetMap.

Throws away all segmentation links.

Warning! Experimental operation.

Map an undirected graph to a hyperbolic surface. Vertices get two attributes called "radial" and "angular" that can be used for edge strength evaluation or link prediction. The algorithm is based on Network Mapping by Replaying Hyperbolic Growth.

The coordinates are generated by simulating hyperbolic growth. The algorithm’s results are most useful when the graph to be mapped follows a power-law degree distribution and has high clustering.

Multiple edges going from A to B that share the same value of the given edge attribute will be merged into a single edge. The edges going from A to B are not merged with edges going from B to A.

Multiple edges going from A to B will be merged into a single edge. The edges going from A to B are not merged with edges going from B to A.

Edge attributes can be aggregated across the merged edges.

Multiple segmentation links going from A base vertex to B segmentation vertex will be merged into a single link.

After performing a Merge vertices by attribute operation, there might be multiple parallel links going between some of the base graph and segmentation vertices. This can cause unexpected behavior when aggregating to or from the segmentation. This operation addresses this behavior by merging parallel segmentation links.

An attribute may not be defined on every edge. This operation uses the secondary attribute to fill in the values where the primary attribute is undefined. If both are undefined on an edge then the result is undefined too.

An attribute may not be defined on every vertex. This operation uses the secondary attribute to fill in the values where the primary attribute is undefined. If both are undefined on a vertex then the result is undefined too.

Merges each set of vertices that are equal by the chosen attribute. Vertices where the chosen attribute is not defined are discarded. Aggregations can be specified for how to handle the rest of the attributes, which may be different among the merged vertices. Any edge that connected two vertices that are merged will become a loop.