Real-Time Events in Journeys

In this guide, we’ll walk through how to set up a real-time journey. Real-time journeys enable dynamic, event-driven engagement based on live data.

Prerequisites

Before you begin, make sure you have:

Apache Kafka, AWS SQS, or Google Pub/Sub for event streaming.
SendGrid or Salesforce Marketing Cloud (SFMC) for email marketing (currently, these are the only real-time destinations supported).
Real-time events enabled in your org. If you don’t see this, contact your GrowthLoop rep to enable it.

Step 1: Configure a Real-Time Source Connection

Log into GrowthLoop and go to the Organization page.
Open the Source Connections tab and click New Source Connection.
In the modal, select Kafka.
Fill out the required details for the event queue connection:

Field Name	Required	Description	Example	Notes
Name	✅ Yes	A unique name to identify this Kafka connection.	`ProductionKafkaCluster`, `OrdersEventBroker`	Must be unique across connections. Used for display and reference.
Description	❌ No	Add context about the connection.	`Kafka cluster for production events`	Helpful for documentation/debugging.
Bootstrap Servers	✅ Yes	Comma-separated list of broker addresses.	`kafka-1.company.com:9092,kafka-2.company.com:9092`	At least one must be reachable.
Topic Name	✅ Yes	Kafka topic the consumer will subscribe to.	`user-signups`, `order.created`	Must already exist in the cluster.
Group ID	❌ No	Consumer group ID.	`signup-event-workers`	Auto-generated if left blank.
Security Protocol	✅ Yes	Defines communication/security protocol.	—	Common: `PLAINTEXT`, `SSL`, `SASL_SSL`.
Mechanism	⚠️ Conditional	Required if using SASL.	`PLAIN`, `SCRAM-SHA-256`	Must match Kafka server config.
Username	⚠️ Conditional	SASL username.	—	Provided by Kafka admin.
Password	⚠️ Conditional	SASL password.	—	Keep secure.

Click Create. A test connection will run, and if successful, your new source connection will be saved.

👍
Success
You’ve successfully configured a real-time source connection!

Step 1a: Configure an AWS SQS Real-Time Source Connection

If your events are published to an AWS SQS queue, you can configure SQS as a real-time source.

Go to the Organization page > Source Connections > New Source Connection.
Select AWS SQS.

Field Name	Required	Description
Name	✅ Yes	Descriptive name for the SQS connection.
Description	❌ No	Optional context about the queue.
Queue URL	✅ Yes	Full URL of the SQS queue.
AWS Region	✅ Yes	Region where the queue is hosted (e.g., us-east-1).
AWS Access Key ID	✅ Yes	IAM access key with permission to read from the queue.
AWS Secret Access Key	✅ Yes	Secret key associated with the access key.
AWS Queue Type	✅ Yes	standard or fifo (FIFO queues must end with `.fifo`).

AWS Permissions (Required)

The AWS credentials must include at least the following IAM permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes"
      ],
      "Resource": "arn:aws:sqs:<REGION>:<ACCOUNT_ID>:<QUEUE_NAME>"
    }
  ]
}

You may optionally grant sqs:ChangeMessageVisibility if long processing times are expected.

Operational Notes

Ensure the queue’s visibility timeout is long enough for message processing.
FIFO queues require exactly-once semantics and proper deduplication settings.
Use least-privilege IAM users or roles and rotate credentials regularly.

Click Create to validate credentials and test the connection.

Step 1b: Configure a Google Pub/Sub Real-Time Source Connection

If your events are published to Google Pub/Sub, you can configure Pub/Sub as a real-time source.

Go to the Organization page > Source Connections > New Source Connection.
Select Google Pub/Sub.

Field Name	Required	Description
Name	✅ Yes	Descriptive name for the source connection.
Description	❌ No	Optional context or notes.
Subscription Name	✅ Yes	Name of an existing Pub/Sub subscription.
Service Account Key	✅ Yes	Upload a JSON key for a service account with subscriber access.

Service Account Requirements

Create a Google Service Account.
Grant roles/pubsub.subscriber.
Generate and upload a JSON key file.
The subscription must already exist before configuration.

Operational Notes

Messages are pulled continuously using Pub/Sub’s pull model.
Messages are acknowledged after successful processing.
The system can scale consumption through parallel pulls.

Click Create to authenticate, validate access, and test message ingestion.

Step 2: Create a Real-Time Event

In the side menu, go to Datasets and select the Realtime Events tab.
Click New Realtime Event.
Fill out the event registration form:

Field Name	Required	Description
Event Source	✅ Yes	Select the real-time source connection (e.g., `ProductionKafkaCluster`).
Event Name	✅ Yes	A human-readable label for this event (e.g., `UserSignup`, `OrderPlaced`).
Path to Event	✅ Yes	Field path in the Kafka message to inspect (supports dot notation, e.g., `user.status.type`).
Event Value	✅ Yes	Value in the field above that triggers the event (e.g., `signup`). Messages without this value are ignored.
Register Event Schema	✅ Yes	JSON Schema describing the event payload for validation and field extraction.

Click Save.

👍
Success
You’ve successfully created a real-time event!

Step 3: Add the Real-Time Event to a Dataset Group

Switch to the Dataset Group tab.
Select your Dataset Group, then open the Realtime Events tab.
Click Add Realtime Event.
In the modal, select the event you created.
Map the real-time field to your dataset’s primary key.

Step 4: Create a Real-Time Journey

In the side menu, go to Journeys and click New Journey.
In setup, choose Realtime Journey as the type.
Select the Dataset Group with your registered event.
From Select Realtime Event, choose your event.
Configure the rest of the journey and click Continue. You’ll see a real-time entry node appear on the canvas with a lightning icon.
Next, drag a Criteria node onto the canvas and filter users by event fields (e.g., event type, attributes).
Then, drag a Destination node and choose a real-time destination (SendGrid or SFMC).

📘
Note
The fields available in real-time destination node are sourced from the real-time event message schema and can be used for email recipient selection and template personalization.

Click Publish to activate the journey.

👍
Success
You’ve successfully created a real-time journey!

📘
Need help?
If you run into issues with Real-Time Journeys, reach out to us at [email protected].