Webhooks

Brief history

The concept of “webhooks” was first introduced in 2007 by Jeff Lindsay. At that time, as APIs became more common, applications mostly got updates from each other through “polling” which meant constantly sending requests to check if new information was available. 

Lindsay’s idea was inspired by the existing programming concept of “hooks” or “callbacks,” where a piece of code could automatically react when a specific event happened. The breakthrough of webhooks was applying this efficient, event-driven “push” method to communication over the web using HTTP. 

Why webhooks matter

Webhooks are important because they create real-time communication between different computer systems, avoiding the need for constant “polling” (where one system repeatedly checks another for new information). Instead, systems only send data when something new actually happens, which leads to immediate responses, better efficiency, and less use of computer resources.

– Real-time updates: Information is sent instantly to the receiving system, with no delay from waiting for the next scheduled check.
– Efficiency & scalability: Webhooks reduce unnecessary requests and bandwidth, which improves how well systems perform and allows them to handle more connections as they grow.
– Automation enablement: They are key to making automated workflows work. Webhooks can trigger actions like running code tests, sending notifications, or syncing databases without anyone having to do it manually.

How Webhooks work

Event registration (Setting up the webhook)

– Subscription setup: You start by telling the source application (like GitHub or Stripe) where to send notifications. This means giving them a “callback URL” (your unique web address) and choosing which specific events you want to be notified about (e.g., when new code is pushed, or a payment succeeds).
– Verification (optional): Some platforms will perform an initial check. They might send a special token or a challenge to your URL to confirm that you actually control that address.

Event triggering (When something happens)

– Event occurs: When the source application detects one of the events you subscribed to (e.g., a new pull request on GitHub, a customer buys something, or someone submits a form), it marks that event for delivery.
– Webhook dispatch: The source application then takes the details of that event, packages it into an HTTP request (usually a POST request with a JSON body), and sends it to the webhook URL you provided.

Payload & delivery (What’s sent and how)

HTTP request structure

– Headers: These contain extra information about the request, such as the type of content being sent (application/json), the type of event (X-Event-Type), and sometimes authentication details (X-Signature).
– JSON payload: This is the main body of the request. It contains all the detailed information about the event, including when it happened, its type, the actual data related to the event (like order information), and unique IDs.

Delivery protocols

Webhooks use a “push-based” delivery. This means, unlike polling (where you keep asking for updates), webhooks instantly send you data as soon as an event occurs. This makes integrations real-time and very efficient.

Endpoint handling (What your application does)

Authentication & validation

Your application (the “listener” at the webhook URL) must first check iif the webhook request is real and from the expected source. This is usually done by checking a shared secret or a special signature to prevent fake notifications.

Processing the event

– You should clean and validate the data in the payload.
– Crucially, your application should send back a quick “200 OK” response immediately. This tells the sender you received the webhook.
– Then, hand off the actual processing of the event to a background task or a separate system. This prevents your webhook endpoint from timing out if the processing takes too long.
– Logging & idempotency: It’s important to keep logs of event IDs and timestamps. This helps you deal with situations where the sender might re-send a webhook (if they didn’t get your “200 OK”) and prevents your system from processing the same event twice.

Real-world flow example

Imagine you’ve set up a GitHub webhook:

1. Register: You tell GitHub to send a webhook to your server’s URL whenever a pull_request event happens.
2. Trigger: A developer on your team opens a new pull request on GitHub.
3. Send: GitHub immediately sends an HTTP POST request to your specified URL. This request includes a JSON payload containing all the details about the new pull request.
4. Process: Your server receives the request and verifies its signature (to make sure it’s from GitHub). It quickly sends back a “200 OK” response to GitHub. Then, it sends the pull request details to a background system, which might start a continuous integration (CI) pipeline or send a notification to your Slack channel.

Webhooks vs. Polling (Traditional APIs)

This section explains the main differences between webhooks and polling, helping you decide which one to use for your application.

Polling

– Initiator: Your program (client) regularly asks the server for data.
– Latency: Delayed (your program waits for its next scheduled check).
– Efficiency: Inefficient; many requests might not get new data.
– Implementation: Simpler to set up and control how often you check.
– Reliability: Naturally retries on the next check; less chance of missing updates.
– Scalability: Can overload servers if you check too often.
– Control: Your program fully controls when it asks for data.

Webhooks

– Initiator: The server automatically sends data to your program (client endpoint).
– Latency: Near real-time; data arrives as soon as an event happens.
– Efficiency: Efficient; data is only sent when there’s something new.
– Implementation: More complex; requires setting up a special web address, security, and checking for valid requests.
– Reliability: Risk of missing events if you don’t build in ways to retry sending data.
– Scalability: Handles more requests better because data is pushed only when an event occurs.
– Control: The server decides when data is sent, and your program reacts to it.

Pros & cons

Polling

Pros

– Very easy to set up and fix problems.
– Good for APIs that don’t offer webhooks.
– Automatically retries sending requests by just trying again later.

Cons

– Information can be old; there’s a delay.
– Many requests are wasted because they don’t get new data.
– Doesn’t work well when there are many users or high demand.

Webhooks

Pros

– Get notifications in real-time with very little delay.
– Uses computer resources efficiently; only sends data when an event happens.
– Scales well for systems that react to events.

Cons

– Requires a secure web address that can be reached from the internet, and you need to validate incoming data.
– There’s a chance of missing events if your system isn’t set up to handle retries.
– More complicated to set up than polling.

When to use each

Choose Webhooks if:

– You need updates in real-time (like for payments, continuous delivery, or live notifications).
– Events happen rarely or at unpredictable times, and you need very low delays.
– You want efficiency and your system works by reacting to events.

Choose Polling if:

– You are using APIs that don’t support webhooks, or their webhook reliability is poor.
– The data changes frequently but doesn’t need to be updated instantly (like general weather updates).
– You prefer simplicity and complete control over how often you check for updates.

Real-world uses for Webhooks

Webhooks are used across many different industries, showing how versatile and powerful they are for connecting software.

E-commerce & payment processing (Example: Stripe)

– Order and shipping: When a customer places an order on an online store, a webhook can instantly tell a shipping company, automatically starting the delivery process.
– Payment updates: Services like Stripe use webhooks to send real-time alerts for successful payments, failed charges, refunds, or disputes. This allows your system to immediately update inventory, start fulfilling the order, or notify the customer.
– Subscription management: Webhooks are crucial for handling subscriptions. They alert systems when a subscription starts, an invoice is paid, or a subscription is canceled, ensuring smooth billing and service access.

CI/CD & code repository events (Example: GitHub)

– Automated development: When a developer pushes new code or creates a pull request on GitHub, webhooks can trigger automated processes like Jenkins or GitHub Actions. This means tests run, software is built, and infrastructure is deployed automatically.
– Notifications: Webhooks send alerts to other systems or chat platforms (like Slack or Microsoft Teams) when things happen in the code repository, such as new issues being opened or deployments completing.

Messaging & notifications

– Sales lead alerts: If a new sales lead comes into a marketing system, a webhook can instantly send that information to Slack or Microsoft Teams, notifying sales representatives right away.
– Support tickets: When a new support ticket is created in a helpdesk system, a webhook can alert team members via email or messaging apps.

Data synchronisation & workflow automation

– Keeping systems in sync: If a customer’s information is updated in your CRM (Customer Relationship Management) system, webhooks can ensure that your marketing tools or email automation platforms are immediately updated as well.
– IoT alerts: In factories or other industrial settings, webhooks can notify central systems about changes in equipment status, low battery levels, or when assets enter or leave specific areas.

Monitoring & system events

– Real-time alerts: Monitoring systems use webhooks to send immediate alerts if there’s a service outage, performance drops, or a security breach. This allows for quick responses or automatic fixes.

Multi-channel publishing

– CMS to hosting: Content management systems (CMS) can use webhooks to tell hosting platforms (like Netlify or Vercel) when content has changed. This automatically triggers a website rebuild and deployment, ensuring the site is always up-to-date.
– Social media automation: A new blog post or tweet can automatically be shared to other social media platforms using webhooks.

Challenges with Webhooks

While webhooks are powerful, they also come with their own set of difficulties and potential points of failure. Here’s a look at the main challenges and how to deal with them:

Reliability & delivery failures

– “At-least-once” delivery: Most webhook systems try their best to deliver a message, often by trying again if it fails. This means you’ll likely receive duplicate messages or messages that arrive out of order. They don’t guarantee you’ll get it exactly once or in the right order.
– Network problems & downtime: If either the sender or receiver’s internet connection goes down, messages can be missed. You need to build in ways to retry sending/receiving messages (with pauses between attempts) to handle this.
– Different retry rules: Every webhook provider has its own rules for retrying failed messages – some might retry for just a few minutes, others for hours. Your system needs to be flexible enough to handle these different behaviors.

Idempotency & handling duplicates

Process each event once
Because messages might be sent multiple times (due to retries), your system must be designed to process each unique event only one time.

Idempotency patterns
Store unique IDs: Save unique IDs that come with the webhook (like a delivery ID from GitHub or an ID in the message itself). If you receive a message with an ID you’ve already processed, just ignore it.
Idempotent coding: Design your actions so that doing them multiple times doesn’t cause harm. For example, ensuring that adding a charge or sending a notification twice doesn’t create extra charges or duplicate notifications.

Security risks

Fake messages & replay attacks
People might try to send you fake webhook messages or re-send old ones.
– Verify payloads: Always check the authenticity of the message using special signatures (HMAC signatures) or tokens provided by the sender.
– Rotate secrets: Regularly change the secret keys used for verification.
– Only use HTTPS: Always make sure your webhook endpoint uses HTTPS to protect the data while it’s traveling.

Exposed endpoints: If your webhook receiving address isn’t properly secured (e.g., no authentication or only allowing certain IP addresses), bad actors could flood it with requests or even compromise your system.

Ordering & consistency

– Out-of-order events: Without guarantees that messages will arrive in order, your systems could end up in a confused state (e.g., processing a message to delete an item before the message to create it).
– How to handle: Use sequence numbers or timestamps included in the webhook message to reorder critical events if they arrive out of sequence.

Scalability & rate control

Sudden bursts of traffic
If many events happen at once (like during a big sale), your webhook endpoint can get overwhelmed.
– Solutions: Use message queues (to hold messages until your system can process them), set limits on how many requests you accept (rate limiting), and use auto-scaling (where your system automatically adds more resources when needed) to protect your system.

Retry storms
If many messages fail and all try to retry at the same time, it can create a “thundering herd” effect, overwhelming your system.
– Solution: Implement “exponential backoff with randomness.” This means retries happen with increasing delays and a random wait time, spreading out the load.

Observability & monitoring

– Silent failures: Without proper logging and alerts, you might not even know if webhooks are being lost, which can lead to bad data or missed opportunities.
– Insufficient tracking: Make sure you’re tracking detailed information about your webhooks: how long they take to deliver, how often they succeed, and if you’re getting duplicates. This data helps you see how well your system is working and troubleshoot problems.

Sportmonks Football API – Pull-based integration

Sportmonks delivers extensive football data, including live scores, match events (goals, cards, subs), player/team stats, standings, odds, and historical info, through a RESTful pull model across over 2,500 leagues.

Pull-based data flow

1. API requests
   You query specific endpoints like /livescores/inplay or /fixtures via GET requests, including your API token in headers or parameters 
2. Filtering & includes
   Use query filters (e.g., include=scores;events, filters=eventTypes:14,19) to limit payload size and fetch only necessary data.
   
3. Polling frequency
   For real-time updates, clients typically poll every few seconds, reflecting a latency of ~15 seconds, depending on request intervals.
4. Rate limiting
   Calls are rate-limited per entity (e.g., max ~3,000 calls/hour), so requests must be optimised to stay within limits.

Unlock the power of real-time football data

Sportmonks currently delivers football data through a reliable pull-based API model. Whether you’re showing live scores, match events, or player stats, you can fetch exactly what you need with flexible filters and includes, giving you control over every request.

Need rapid updates? You can poll every few seconds while staying within generous rate limits, ensuring your app stays in sync with the action. Try the Sportmonks Football API today and build smarter, faster football experiences.