Core part of managing we scraping project is having dashboard
where you can track progress and quality issues of each pipeline
Below is chart depicting the whole lifecycle of each data pipeline - from its creation to
quality control measures.
After task is created, it will go to project up-for-grabs pool.
It can be picked by Your or any other project member. After reservation is completed,
the owner will have 72 hours to create a valid solution.
Solution is submitted via web interface, by providing python code. Solution
goes to approval process, where moderator will review code quality,
spot check data and approve pipeline. This will schedule autorun. You
can approve your own pipelines as moderator.
It is likely that at some point some point your pipeline will start failing due to internal
or external reasons. Project owner wiill get email alert and be notified about issue.
