Skip to main content

Webhooks

Webhooks in PullNode offer a powerful way to automate your workflows by pushing scraping results from your projects directly to external services, applications, or servers in real-time. This feature is especially useful for integrating your scraped data with other tools or databases without the need for manual downloading and uploading.

Setting Up a Webhook​

To set up a webhook for your project in PullNode, follow these simple steps:

  1. Navigate to Your Project Dashboard: Go to the project for which you want to set up the webhook.
  2. Access Webhook Settings: Find the "Integrate" section in your project settings.
  3. Enter Webhook URL: Input the URL where you want the scraping results to be pushed into the Webhook URL field. This URL is provided by the external service you're integrating with (e.g., Zapier, Integromat, your custom server endpoint).

Example: https://hooks.zapier.com/hooks/catch/123456/abcde

  1. Configure Trigger Settings: Decide when you want the webhook to be triggered.
  • Send Hooks on API: Enable this setting if you want the webhook to be sent on all API runs, including manual and scheduled ones. If disabled, webhooks will only be sent on scheduled runs.
  1. Save Your Project: Once you've configured your webhook settings, click "Save project" to apply the changes.

Use Cases​

Webhooks can be used in a variety of scenarios, including but not limited to:

  • Automated Data Processing: Push data to a server for immediate processing and analysis.
  • Integration with Other Services: Automatically update tools like Google Sheets, Salesforce, or Airtable with the latest data without manual intervention.
  • Real-time Notifications: Send alerts to Slack, Discord, or other communication platforms when new data is available.

Testing Your Webhook​

After setting up your webhook, it's important to test it to ensure that data is being pushed correctly:

  1. Trigger a Run: Initiate a run of your scraper either through the API or a scheduled run, depending on your webhook settings.
  2. Check the Receiving End: Verify that the data has been received at the URL you specified. If you're using a service like Zapier, you can check the task history to see the incoming data.

Troubleshooting​

If you're not seeing data at your webhook URL, consider the following:

  • URL Correctness: Ensure that the webhook URL is correctly entered without any typos.
  • Service Downtime: Check if the receiving service is operational and not experiencing downtime.
  • Firewall or Security Settings: Make sure that your receiving endpoint is not blocking incoming requests from PullNode.

Conclusion​

Webhooks are a versatile tool that can significantly streamline your data workflow. By automating the push of scraped data to other services, you can enhance the efficiency and immediacy of your data processing tasks.