Maxun lets you train a robot in 2 minutes and scrape the web on auto-pilot. Web data extraction doesn't get easier than this!
# Installation
# Envirnoment Variables
| Variable | Mandatory | Description | If Not Set |
|--------------|-----------|----------------------------|----------------------------- |
| `NODE_ENV` | Yes | Sets whether you are running the app locally or in production. | |
| `JWT_SECRET` | Yes | JWT secret is utilized to generate authentication tokens. | |
| `DB_NAME` | Yes | Brief description here. | Describe what happens here. |
| `DB_USER` | Yes | Brief description here. | Describe what happens here. |
| `DB_PASSWORD` | Yes | Brief description here. | Describe what happens here. |
| `DB_NAME` | Yes | Brief description here. | Describe what happens here. |
| `DB_USER` | Yes | Brief description here. | Describe what happens here. |
| `DB_HOST` | Yes | Sets whether you are running the app locally or in production. | |
| `DB_PORT` | Yes | JWT secret is utilized to generate authentication tokens. | |
| `ENCRYPTION_KEY` | Yes | Brief description here. | Describe what happens here. |
| `MINIO_ENDPOINT` | Yes | Brief description here. | Describe what happens here. |
| `MINIO_PORT` | Yes | Brief description here. | Describe what happens here. |
| `MINIO_ACCESS_KEY` | Yes | Brief description here. | Describe what happens here. |
| `GOOGLE_CLIENT_ID` | Yes | Brief description here. | Describe what happens here. |
| `GOOGLE_CLIENT_SECRET` | Yes | Brief description here. | Describe what happens here. |
| `GOOGLE_REDIRECT_URI` | Yes | Brief description here. | Describe what happens here. |
| `REDIS_HOST` | Yes | Brief description here. | Describe what happens here. |
| `REDIS_PORT` | Yes | Brief description here. | Describe what happens here. |
| `MAXUN_TELEMETRY` | No | Brief description here. | Describe what happens here. |
# How Does It Work?
Maxun lets you create custom robots which emulate user actions and extract data. A robot can perform any of the actions: Capture List, Capture Text or Capture Screenshot. Once a robot is created, it will keep extracting data for you without manual intervention
### 1. Robot Actions
1. Capture List: Useful to extract structured and bulk items from the website. Example: Scrape products from Amazon etc.
2. Capture Text: Useful to extract individual text content from the website.
3. Capture Screenshot: Get fullpage or visible section screenshots of the website.
### 2. BYOP
BYOP (Bring Your Own Proxy) lets you connect external proxies to bypass anti-bot protection. Currently, the proxies are per user. Soon you'll be able to configure proxy per robot.
# Features
- ✨ Extract Data With No-Code
- ✨ Handle Pagination & Scrolling
- ✨ Run Robots On A Specific Schedule
- ✨ Turn Websites to APIs
- ✨ Turn Websites to Spreadsheets
- ✨ Adapt To Website Layout Changes (coming soon)
- ✨ Extract Behind Login, With Two-Factor Authentication Support (coming soon)
- ✨ Integrations (currently Google Sheet)
- +++ A lot of amazing things soon!
# Cloud
We offer a managed cloud version to run Maxun without having to manage the infrastructure and extract data at scale. Maxun cloud also deals with anti-bot detection, huge proxy network with automatic proxy rotation, and CAPTCHA solving. If this interests you, [join the cloud waitlist](https://docs.google.com/forms/d/e/1FAIpQLSdbD2uhqC4sbg4eLZ9qrFbyrfkXZ2XsI6dQ0USRCQNZNn5pzg/viewform) as we launch soon.
# Note
This project is in early stages of development. We're actively working to improve the product.
# Contributing
Please refer to [Contribution Guide](https://github.com/amhsirak/maxun/blob/master/.github/CONTRIBUTING.md).
# License
This project is licensed under AGPLv3.
# Contributors Thank you to the combined efforts of everyone who contributes!