Struct BulkLoader 

pub struct BulkLoader<'a> { /* private fields */ }

A bulk loader allowing to load a lot of data quickly into the store.

Memory usage is configurable using with_max_memory_size_in_megabytes and the number of used threads with with_num_threads. By default, the memory consumption target (excluding the system and RocksDB internal consumption) is around 2GB per thread and 2 threads. These targets are considered per loaded file.

Usage example a dataset:

use oxigraph::io::RdfFormat;
use oxigraph::model::*;
use oxigraph::store::Store;

let store = Store::new()?;

// quads file insertion
let file =
    "<http://example.com> <http://example.com> <http://example.com> <http://example.com> .";
let mut loader = store.bulk_loader();
loader.load_from_slice(RdfFormat::NQuads, file)?;
loader.commit()?;

// we inspect the store contents
let ex = NamedNodeRef::new("http://example.com")?;
assert!(store.contains(QuadRef::new(ex, ex, ex, ex))?);

Implementations

impl BulkLoader<'_>

pub fn with_num_threads(self, num_threads: usize) -> Self

Sets the maximal number of background threads to be used by the bulk loader.

The default value is the number of threads on the machine.

pub fn with_max_memory_size_in_megabytes(self, max_memory_size: usize) -> Self

Sets a rough idea about the maximal amount of memory to be used by this operation.

This number must be at least a few megabytes per thread.

Memory used by RocksDB and the system is not taken into account in this limit. Note that depending on the system behavior, this amount might never be reached or be blown up (for example, if the data contains very long IRIs or literals).

By default, a target 2GB per used thread is used.

pub fn without_atomicity(self) -> Self

Allow the bulk loader to save also data to the database during the bulk loading instead of only when commit is called.

When used with the RocksDB storage, it allows the storage to compact the data while the loading continues.

pub fn on_progress(self, callback: impl Fn(u64) + Send + Sync + 'static) -> Self

Adds a callback evaluated from time to time with the number of loaded triples.

pub fn on_parse_error( self, callback: impl Fn(RdfParseError) -> Result<(), RdfParseError> + Send + Sync + 'static, ) -> Self

Adds a callback catching all parse errors and choosing if the parsing should continue by returning Ok or fail by returning Err.

By default, the parsing fails.

pub fn load_from_reader( &mut self, parser: impl Into<RdfParser>, reader: impl Read, ) -> Result<(), LoaderError>

Loads a file using the bulk loader.

This function is optimized for large dataset loading speed. For small files, Store::load_from_reader might be more convenient.

See the struct documentation for more details.

To get better speed on valid datasets, consider enabling RdfParser::lenient option to skip some validations.

Usage example:

use oxigraph::store::Store;
use oxigraph::io::{RdfParser, RdfFormat};
use oxigraph::model::*;

let store = Store::new()?;

// insert a dataset file
let file = "<http://example.com> <http://example.com> <http://example.com> <http://example.com/g> .";
let mut loader = store.bulk_loader();
loader.load_from_reader(
    RdfParser::from_format(RdfFormat::NQuads).lenient(), // we inject a custom parser with options
    file.as_bytes()
)?;
loader.commit()?;

// insert a graph file
let file = "<> <> <> .";
let mut loader = store.bulk_loader();
loader.load_from_reader(
    RdfParser::from_format(RdfFormat::Turtle)
        .with_base_iri("http://example.com")?
        .without_named_graphs() // No named graphs allowed in the input
        .with_default_graph(NamedNodeRef::new("http://example.com/g2")?), // we put the file default graph inside of a named graph
    file.as_bytes()
)?;
loader.commit()?;

// we inspect the store contents
let ex = NamedNodeRef::new("http://example.com")?;
assert!(store.contains(QuadRef::new(ex, ex, ex, NamedNodeRef::new("http://example.com/g")?))?);
assert!(store.contains(QuadRef::new(ex, ex, ex, NamedNodeRef::new("http://example.com/g2")?))?);

pub fn load_from_slice( &mut self, parser: impl Into<RdfParser>, slice: &(impl AsRef<[u8]> + ?Sized), ) -> Result<(), LoaderError>

Loads serialized RDF in a slice using the bulk loader.

This function is optimized for large dataset loading speed. For small files, Store::load_from_reader might be more convenient.

See the struct documentation for more details.

To get better speed on valid datasets, consider enabling RdfParser::lenient option to skip some validations.

Usage example:

use oxigraph::store::Store;
use oxigraph::io::{RdfParser, RdfFormat};
use oxigraph::model::*;

let store = Store::new()?;

// insert a dataset file
let file = "<http://example.com> <http://example.com> <http://example.com> <http://example.com/g> .";
let mut loader = store.bulk_loader();
loader.load_from_slice(
    RdfParser::from_format(RdfFormat::NQuads).lenient(), // we inject a custom parser with options
    file
)?;
loader.commit()?;

// insert a graph file
let file = "<> <> <> .";
let mut loader = store.bulk_loader();
loader.load_from_slice(
    RdfParser::from_format(RdfFormat::Turtle)
        .with_base_iri("http://example.com")?
        .without_named_graphs() // No named graphs allowed in the input
        .with_default_graph(NamedNodeRef::new("http://example.com/g2")?), // we put the file default graph inside of a named graph
    file
)?;
loader.commit()?;

// we inspect the store contents
let ex = NamedNodeRef::new("http://example.com")?;
assert!(store.contains(QuadRef::new(ex, ex, ex, NamedNodeRef::new("http://example.com/g")?))?);
assert!(store.contains(QuadRef::new(ex, ex, ex, NamedNodeRef::new("http://example.com/g2")?))?);

pub fn parallel_load_from_file( &mut self, parser: impl Into<RdfParser>, path: impl AsRef<Path>, ) -> Result<(), LoaderError>

Loads RDF file using the bulk loader.

If the input format is N-Triples or N-Quads, it will spawn multiple parallel threads to parse the file.

This function is optimized for large dataset loading speed. For small files, Store::load_from_reader might be more convenient.

See the struct documentation for more details.

To get better speed on valid datasets, consider enabling RdfParser::lenient option to skip some validations.

Usage example:

use oxigraph::store::Store;
use oxigraph::io::{RdfParser, RdfFormat};
use oxigraph::model::*;

let store = Store::new()?;

// insert a dataset file
let file = "<http://example.com> <http://example.com> <http://example.com> <http://example.com/g> .";
let mut loader = store.bulk_loader();
loader.parallel_load_from_slice(
    RdfParser::from_format(RdfFormat::NQuads).lenient(), // we inject a custom parser with options
    file,
)?;
loader.commit()?;

// insert a graph file
let file = "<http://example.com> <http://example.com> <http://example.com> .";
let mut loader = store.bulk_loader();
loader.parallel_load_from_slice(
    RdfParser::from_format(RdfFormat::NTriples)
        .with_base_iri("http://example.com")?
        .without_named_graphs() // No named graphs allowed in the input
        .with_default_graph(NamedNodeRef::new("http://example.com/g2")?), // we put the file default graph inside of a named graph
    file,
)?;
loader.commit()?;

// we inspect the store contents
let ex = NamedNodeRef::new("http://example.com")?;
assert!(store.contains(QuadRef::new(ex, ex, ex, NamedNodeRef::new("http://example.com/g")?))?);
assert!(store.contains(QuadRef::new(ex, ex, ex, NamedNodeRef::new("http://example.com/g2")?))?);

pub fn parallel_load_from_slice( &mut self, parser: impl Into<RdfParser>, slice: &(impl AsRef<[u8]> + ?Sized), ) -> Result<(), LoaderError>

Loads serialized RDF in a slice using the bulk loader.

If the input format is N-Triples or N-Quads, it will spawn multiple parallel threads to parse the file.

This function is optimized for large dataset loading speed. For small files, Store::load_from_reader might be more convenient.

See the struct documentation for more details.

To get better speed on valid datasets, consider enabling RdfParser::lenient option to skip some validations.

Usage example:

use oxigraph::store::Store;
use oxigraph::io::{RdfParser, RdfFormat};
use oxigraph::model::*;

let store = Store::new()?;

// insert a dataset file
let file = "<http://example.com> <http://example.com> <http://example.com> <http://example.com/g> .";
let mut loader = store.bulk_loader();
loader.parallel_load_from_slice(
    RdfParser::from_format(RdfFormat::NQuads).lenient(), // we inject a custom parser with options
    file,
)?;
loader.commit()?;

// insert a graph file
let file = "<http://example.com> <http://example.com> <http://example.com> .";
let mut loader = store.bulk_loader();
loader.parallel_load_from_slice(
    RdfParser::from_format(RdfFormat::NTriples)
        .with_base_iri("http://example.com")?
        .without_named_graphs() // No named graphs allowed in the input
        .with_default_graph(NamedNodeRef::new("http://example.com/g2")?), // we put the file default graph inside of a named graph
    file
)?;
loader.commit()?;

// we inspect the store contents
let ex = NamedNodeRef::new("http://example.com")?;
assert!(store.contains(QuadRef::new(ex, ex, ex, NamedNodeRef::new("http://example.com/g")?))?);
assert!(store.contains(QuadRef::new(ex, ex, ex, NamedNodeRef::new("http://example.com/g2")?))?);

pub fn load_quads( &mut self, quads: impl IntoIterator<Item = impl Into<Quad>>, ) -> Result<(), StorageError>

Adds a set of quads using the bulk loader.

See the struct documentation for more details.

pub fn load_ok_quads<EI, EO: From<StorageError> + From<EI>>( &mut self, quads: impl IntoIterator<Item = Result<impl Into<Quad>, EI>>, ) -> Result<(), EO>

Adds a set of quads using the bulk loader while breaking in the middle of the process in case of error.

See the struct documentation for more details.

pub fn commit(self) -> Result<(), StorageError>

Saves all the quads loaded using the bulk loader into the store.