Start Logprep programaticly
It is possible to make use of the Logprep pipeline in plain python, without any input or output connectors or further configurations. If on the other hand you want to make use of the input connector preprocessors you have to at least use an input connector like the DummyInput. The integration in python
An example with input connector and preprocessors could look like this:
from logprep.framework.pipeline import Pipeline
event = {
"some": "data",
"test_pre_detector": "bad_information"
}
config = {
"pipeline": [
{
"predetector": {
"type": "pre_detector",
"specific_rules": [
"quickstart/exampledata/rules/pre_detector/specific"
],
"generic_rules": [
"quickstart/exampledata/rules/pre_detector/generic"
],
"pre_detector_topic": "output_topic"
}
}
],
"input": {
"my_input":{
"type": "dummy_input",
"documents": [event],
"preprocessing": {
"log_arrival_time_target_field": "arrival_time"
}
}
}
}
pipeline = Pipeline(config=config)
extra_outputs = pipeline.process_pipeline()
An example without input connector and preprocessors could look like this:
from logprep.framework.pipeline import Pipeline
event = {
"some": "data",
"test_pre_detector": "bad_information"
}
config = {
"pipeline": [
{
"predetector": {
"type": "pre_detector",
"specific_rules": [
"quickstart/exampledata/rules/pre_detector/specific"
],
"generic_rules": [
"quickstart/exampledata/rules/pre_detector/generic"
],
"pre_detector_topic": "output_topic"
}
}
],
}
pipeline = Pipeline(config=config)
extra_outputs = pipeline.process_event(event)
Hint
To make use of preprocessors call pipeline.process_pipeline()
.
Calling the respective method multiple times will result in iterating through the list of input
events.
To call the pipeline without input connector call pipeline.process_event(event)
.
Warning
When using the pipeline like this Logprep does not store any events or errors in an designated output. All relevant information are returned to the user and have to be taken care of the user themself.