Create Dynamic Reverse Proxy

Problem:

We are using different proxy services for scraping/crawling where we can only add 1 whitelisted access IP. But we are having different crawlers running with different IPs.

Solution:

All crawlers make their request via our "dynamic reverse proxy" to the service proxies. The service proxy get all requests from the same access ip (dynmaic reverse proxy).

Requirements:

- It should be fast (HAProxy, Ngnix, etc). It should handle thousands of requests concurrently.

- All headers should be passed through the dynamic request proxy.

- We should be able to easily add new upstream proxies. Optimum without downtime of the reverse proxy.

Please write a describe a brief architecture of you solution.

Looking forward to you propasal!

Linux Server Arkitekturë softuerësh Hosting Uebi

ID Projekti: #18305998

Rreth projektit