# rudof

[![Latest Version](https://img.shields.io/crates/v/rudof-cli.svg)](https://crates.io/crates/rudof-cli)
[![PyPI](https://img.shields.io/pypi/v/pyrudof)](https://pypi.org/project/pyrudof/)
[![rudof](https://github.com/rudof-project/rudof/actions/workflows/ci.yml/badge.svg)](https://github.com/rudof-project/rudof/actions/workflows/ci.yml)
[![dependency status](https://deps.rs/repo/github/rudof-project/rudof/status.svg)](https://deps.rs/repo/github/rudof-project/rudof)
[![CodeScene general](https://codescene.io/images/analyzed-by-codescene-badge.svg)](https://codescene.io/projects/72637)
[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit)](https://github.com/pre-commit/pre-commit)


This repo contains an RDF and Knowledge Graphs processing tool implemented in Rust.

The implementation supports RDF and property graphs data, as well as 
[ShEx](http://shex.io/),
[SHACL](https://www.w3.org/TR/shacl/),
[DCTap](https://www.dublincore.org/specifications/dctap/), 
[PGSchema](https://dl.acm.org/doi/10.1145/3589778),
and conversions between different RDF data modeling formalisms.

The code can be used as a Rust library
but it also contains a binary called `rudof`
which can be used as an RDF playground.

We provide binaries for Linux, Windows, Mac and Docker
(see [releases](https://github.com/rudof-project/rudof/releases)),
as well as Python bindings.

- [Documentation](https://rudof-project.github.io/rudof/)
- [Using rudof as Jupyter notebooks](https://rudof-project.github.io/tutorials)
- [Installation](https://github.com/rudof-project/rudof?tab=readme-ov-file#installation)
- [List of issues](https://github.com/rudof-project/rudof/issues)
- [Discussion](https://github.com/rudof-project/rudof/discussions)
- [FAQ](https://github.com/rudof-project/rudof/wiki/FAQ)
- [How to guides](https://github.com/rudof-project/rudof/wiki/How%E2%80%90to-guides)
- [Roadmap](https://github.com/rudof-project/rudof/issues/1)

<!-- 
>[!NOTE]
>Starting from version 0.2.17, all `rudof` dependencies are version-aligned. Therefore, if you need to use a specific dependency, it must match the same version used by rudof.

>[!WARNING]
>The use of versions between 0.2.19 and 0.3.0 (inclusive) is discouraged due to some problems with the CI/CD pipeline.
>Its possible that some of the published versions in that range may not work as expected, or even does not exist. 
-->

## Features

`rudof` currently supports the following:

- RDF and RDF 1.2 parsing, conversion and visualization.
- SPARQL querying to RDF data and endpoints
- Parsing SPARQL service description
- ShEx
- SHACL
- DCTAP
- Property graphs and Schemas for Property Graphs ([PGSchema]()https://dl.acm.org/doi/10.1145/3589778)

Future features we are planning to add:

- rdf-config
- LinkML

## Installation

### Official releases

You can download a binary from the [latest release](https://github.com/rudof-project/rudof/releases/latest) page.
There you will also find the compiled packages
for the installation on your system using a package manager.

#### Published Debian packages

rudof is available as a [Debian package](https://packages.debian.org/source/sid/rudof) for Debian-based distributions (like Ubuntu).

It is also possible to download the binary from <https://github.com/rudof-project/rudof/releases>
and install the `.deb` package running the following commands after replacing X.X.X by the latest version:

```sh
wget https://github.com/rudof-project/rudof/releases/download/vX.X.X/rudof_vX.X.X_amd64.deb
sudo dpkg -i rudof_vX.X.X_amd64.deb
```

#### Windows

The binary can be downloaded from <https://github.com/rudof-project/rudof/releases>.

#### Mac

The binary is available at:
<https://github.com/rudof-project/rudof/releases>
so you can download the corresponding binary to your machine.

The usual way to run/install a binary in Mac is to download it in a folder,
add that folder to your PATH and activating the binary using:

```sh
chmod +x <binary_file>
```

After that, the processor may complain the first time about security
and you have to agree to use it.
Once you agree, it should work.

<details markdown="block">
<summary>Compiling from source</summary>

### Compiling from source

`rudof` has been implemented in Rust
and is compiled using [cargo](https://doc.rust-lang.org/cargo/).
The command `cargo run` can be used to compile and run locally the code.

For example:

```sh
cargo run -- validate examples/user.ttl --schema examples/user.shex --shapemap examples/user.sm
```
#### Troubleshooting
If the example doesn’t work as expected, here are a few things you can try:
- **Use the --release flag** to compile in release mode, which can resolve some build issues and improve performance:

```sh
cargo run --release -- validate examples/user.ttl --schema examples/user.shex --shapemap examples/user.sm
```

- **Run the command inside WSL** (Windows Subsystem for Linux). If you're using Windows, compiling the project in WSL can help resolve environment-related issues, as Rust tends to compile more reliably and efficiently in Linux-based systems.

### Compiling from source and installing the binary (Debian)

Install `cargo deb` (only the first time)

```sh
cargo install cargo-deb
```

Create the `.deb` package by:

```sh
cargo deb
```

And run:

```sh
sudo dpkg -i target/debian/rudof_<VERSION>_amd64.deb
```

### Alternative for Linux

To create a binary for Linux with debug information:

```sh
cargo build --target x86_64-unknown-linux-gnu
```

The binary will be created in: `target/x86_64-unknown-linux-gnu/debug/rudof`

If you want a release binary which is more optimized, you can run:

```sh
cargo build --target x86_64-unknown-linux-gnu
```

In this case, the binary will be `target/x86_64-unknown-linux-gnu/release/rudof`

## Docker

The library is also published as a Docker image
(`angelip2303/rudof:latest`).

</details>

## Usage

### Some examples

The folder `examples` contains several example files with ShEx schemas and RDF data.

### Validate a simple RDF file with a ShEx schema using a ShapeMap

```sh
rudof validate examples/user.ttl --schema examples/user.shex --shapemap examples/user.sm
```

We maintain a Wiki page with some common [Usage scenarios and How-to guides](https://github.com/rudof-project/rudof/wiki/Howto-guides).

### Debugging information

It is possible to change the debug level information with:

```sh
export RUST_LOG=info
```

where `value` can be `error`, `warn`, `info`, `debug` or `trace` to show more information. A
bare level like this is automatically scoped to rudof's own crates, with dependencies (HTTP/TLS
libraries, the shell's line editor, etc.) capped at `warn` — so `RUST_LOG=debug` won't flood
your terminal with unrelated internals. Write a filter containing `=` or `,` yourself if you
want more control (e.g. to also see a specific dependency's own logs), and it's used exactly
as written:

```
export RUST_LOG=info,shacl_validation=trace,hyper=trace
```

In case you use nu_shell, you can use:

```
$env.RUST_LOG = 'info,shacl_validation=trace,hyper=trace'
```

This can also be set persistently in `rudof.toml` (`[logging] level = "debug"`, see the
[Config reference](https://rudof-project.github.io/rudof/references/config.html#logging--diagnostic-output)),
and, inside the interactive shell, changed live for the rest of the session with
`config set logging.level debug` — unlike `$RUST_LOG`, which can only be read once, at
process start.

## Command line usage

```sh
RDF and Knowledge Graphs processing tool

Usage: rudof [OPTIONS] [COMMAND]

Commands:
  mcp                Export rudof as an MCP server
  shapemap           Show information about ShEx ShapeMaps
  shex               Show information about ShEx schemas
  pgschema           Show information about Property Graph Schemas
  validate           Validate RDF data using ShEx or SHACL
  shex-validate      Validate RDF using ShEx schemas
  shacl-validate     Validate RDF data using SHACL shapes
  data               Show information about RDF data
  node               Show information about a node in an RDF Graph
  shacl              Show information about SHACL shapes
  dctap              Arguments for the `dctap` command
  convert            Arguments for the `convert` command
  compare            Compare two shapes (which can be in different formats)
  rdf-config         Show information about rdf config
  service            Show information about SPARQL service
  query              Run SPARQL queries
  generate           Generate synthetic RDF data from ShEx or SHACL schemas
  materialize        Materialize an RDF graph from a ShEx schema and Map semantic-action state
  pgschema-validate  Validate Property Graph data using PGSchema
  completion         Generates a shell completion script for the specified shell
  config             Dump the effective configuration rudof is using as TOML
  help               Print this message or the help of the given subcommand(s)

Options:
  -d, --debug...  Increase logging verbosity
  -h, --help      Print help
  -V, --version   Print version
```

You can see the [manual](https://rudof-project.github.io/rudof/)

## Main modules

The repo is divided in the following modules:

- [rudof_iri](https://github.com/rudof-project/rudof/tree/master/rudof_iri) defines simple IRIs.
- [srdf](https://github.com/rudof-project/rudof/tree/master/srdf) simple RDF model which will be used for validation.
- [prefixmap](https://github.com/rudof-project/rudof/tree/master/prefixmap) Prefix maps implementation.
- [shapemap](https://github.com/rudof-project/rudof/tree/master/shapemap) ShapeMap implementation.
- [shex_ast](https://github.com/rudof-project/rudof/tree/master/shex_ast) defines the ShEx Abstract syntax
- [shex_compact](https://github.com/rudof-project/rudof/tree/master/shex_compact) contains the code required to handle ShEx compact syntax.
- [shex_validation](https://github.com/rudof-project/rudof/tree/master/shex_validation) contains the code required to validate RDF using ShEx.
- [shex_testsuite](https://github.com/rudof-project/rudof/tree/master/shex_testsuite) contains the code required to run the ShEx testsuite.
- [shacl_ast](https://github.com/rudof-project/rudof/tree/master/shacl_ast) defines the SHACL core Abstract syntax.
- [shacl_validation](https://github.com/rudof-project/rudof/tree/master/shacl_validation) contains the code required to validate RDF using SHACL.
- [dctap](https://github.com/rudof-project/rudof/tree/master/dctap) contains the code required to do handle DCTAP files.
- [shapes_converter](https://github.com/rudof-project/rudof/tree/master/shapes_converter) contains the code required to do conversion between different shapes formalisms.

## Publishing the crates

```sh
cargo workspaces publish
```

## Worskpaces

The project is using cargo workspaces wihch can be installed with:

```sh
cargo install cargo-workspaces
```

## Unit-testing

In order to test all the sub-projects

```sh
cargo test --all
```

Testing one specific subproject:

```sh
cargo test -p shex_validation
```

## Using the ShEx test-suite

The ShEx testsuite is included in a git submodule. In order to obtain it, it is necessary to do:

```sh
git submodule update --init --recursive
cargo run -p shex_testsuite
```

```sh
Usage: shex_testsuite [OPTIONS]

Options:
  -m, --manifest <Manifest FILE (.jsonld)>
          Name of Manifest file [default: shex_testsuite/shexTest/validation/manifest.jsonld]
  -c, --config <Config file>
          [default: shex_testsuite/config.toml]
  -x, --run_mode <MANIFEST_RUN_MODE>
          [default: collect-errors] [possible values: collect-errors, fail-first-error]
  -f, --manifest_mode <MANIFEST_MODE>
          [possible values: schemas, validation, negative-syntax, negative-structure]
  -p, --print_result_mode <PRINT_RESULT_MODE>
          [default: basic] [possible values: basic, failed, passed, not-implemented, all]
  -e, --entry <Entry names>

  -t, --trait <Trait names>

  -h, --help
          Print help
  -V, --version
          Print version
```

### Validation conformance tests for ShEx

```sh
cargo run -p shex_testsuite -- -m shex_testsuite/shexTest/validation/manifest.jsonld 
```
If you want to see the failed tests, you can run:

```sh
cargo run -p shex_testsuite -- -m shex_testsuite/shexTest/validation/manifest.jsonld -p failed
```



### Schemas tests

```sh
cargo run -p shex_testsuite -- -m shex_testsuite/shexTest/schemas/manifest.jsonld -f schemas -p failed
```

## Benchmarks

Performance benchmarks live in [`benchmarks/`](./benchmarks/). They are not published to crates.io and are not executed in CI. Run them locally with Criterion.

## License

Licensed under either of

- Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE) or <http://www.apache.org/licenses/LICENSE-2.0>)
- MIT license ([LICENSE-MIT](LICENSE-MIT) or <http://opensource.org/licenses/MIT>)

at your option.

## Contributors

<!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section -->
<!-- prettier-ignore-start -->
<!-- markdownlint-disable -->

<!-- markdownlint-restore -->
<!-- prettier-ignore-end -->

<!-- ALL-CONTRIBUTORS-LIST:END -->

### Contribution

Unless you explicitly state otherwise,
any contribution intentionally submitted for inclusion in the work by you,
as defined in the Apache-2.0 license,
shall be dual licensed as above,
without any additional terms or conditions.

### Local development

[Precommit](https://pre-commit.com/) has been configured to run formatters and linters before each commit.
It is recommended to use it to ensure that the code is formatted and linted correctly before pushing it to the repository.

In order to install it, you can run:

```sh
# Installs pre-commit it self
pip install pre-commit

# Installs the pre-commit hooks defined in .pre-commit-config.yaml
pre-commit install
```

#### Nix

If you have [Nix](https://nixos.org/download) with flakes enabled, the repository provides a ready-to-use development shell that includes all required dependencies.

Simply run the following command from the root of the repository:

```shell
nix develop
```

This drops you into a shell with everything pre-configured.  No manual installation of Rust, `cargo`, or any other tool is needed.

You can also run a one-off command without entering the shell interactively:

```shell
nix develop --command cargo test
```

> If you use [direnv](https://direnv.net/), add a `.envrc` file with `use flake` to activate the development environment automatically whenever you `cd` into the repository:
> ```shell
> echo "use flake" > .envrc && direnv allow
> ```
