When building applications, we often interact with various systems, such as in-house microservices or third-party systems with HTTP endpoints as applications would need data and depend on third-party systems for business-critical tasks or data.
HTTP API invocation from Lambda functions
This process may require using AWS Lambda functions that utilize HTTP libraries (such as request or axios for NodeJS) to construct the request with the appropriate headers and body, ensuring that the request is sent and the response is processed successfully.
Increased complexity: Introducing the Lambda function to invoke HTTP endpoints adds components that must be managed and maintained.
Increased latency: Since the request response has to hop off another architectural component, it adds to the latency of the response.
Increased cost: Running a separate Lambda function to handle the HTTP invocation will incur additional costs, as you'll be charged for the Lambda function execution in addition to the Step Functions execution.
Also, this opens up the flexibility to invoke the HTTP endpoints with whitelisted IP addresses and more security aspects of the Lambda function with Secrets Manager to maintain HTTP credentials.
HTTP API invocation with EventBridge API destinations
In an Event-Driven world, Amazon EventBridge's API destination feature allows you to configure the HTTP endpoint and credentials from Secrets Manager, collectively known as EventBridge Connection.
In a blog about How Freshworks Developer Platform integrates with AWS via EventBridge, I've explained in detail the integration but in this section of the blog, we will focus on the API destination part of the architecture.
MyConnection:
Type: AWS::Events::Connection
Properties:
AuthorizationType: BASIC
Description: 'My connection with username/password'
AuthParameters:
BasicAuthParameters :
Username : !Ref APIKeyValue
Password : !Ref Password
MyApiDestination:
Type: AWS::Events::ApiDestination
Properties:
Name: 'FreshdeskAPI'
ConnectionArn: !GetAtt MyConnection.Arn
InvocationEndpoint: !Ref FreshdeskURL
HttpMethod: PUT
InvocationRateLimitPerSecond: 10
EventRule:
Type: AWS::Events::Rule
Properties:
Description: "EventRule"
State: "ENABLED"
EventBusName: !Ref MyEventBus
EventPattern:
source:
- "fw-sentiment"
Targets:
- Arn: !GetAtt MyApiDestination.Arn
RoleArn: !GetAtt EventBridgeTargetRole.Arn
Id: "MyAPIdestination"
InputTransformer:
InputPathsMap:
id: $.detail.id
priority: $.detail.priority
InputTemplate: >
{
"priority": <priority>
}
HttpParameters:
PathParameterValues:
- $.detail.id
DeadLetterConfig:
Arn: !GetAtt MyDLQueue.Arn
API Destination brings to the table
Improved Security: API connections use AWS Secrets Manager to securely store and manage authentication credentials, reducing the risk of exposing sensitive information, which is more secure than hardcoding credentials in Lambda functions.
Reduced complexity: Using API connections with HTTP Tasks in Step Functions eliminates the need to write and maintain Lambda functions solely for making HTTP requests, simplifying the architecture and reducing the amount of code to manage.
Dependency of EventBridge Bus: Introduces dependency on Event Buses to be able to invoke API destinations. Although it's managed, adding in additional components would mean that you have to define the Event Rules.
HTTP invoke on Step Functions
The above image from Application Composer indicates that when creating a State Machine with HTTPInvoke State, it would also require EventBridge Connection. You can also refer to the SAM template of the State Machine.
StateMachine:
Type: AWS::Serverless::StateMachine
Properties:
Definition:
StartAt: Call third-party API
States:
Call third-party API:
Type: Task
Resource: arn:aws:states:::http:invoke
Parameters:
Method: GET
ApiEndpoint: ${ApiEndpoint}
Authentication:
ConnectionArn: ${EBConnectionARN}
Retry:
- ErrorEquals:
- States.ALL
BackoffRate: 2
IntervalSeconds: 1
MaxAttempts: 3
JitterStrategy: FULL
ResultPath: $.ApiResponse
End: true
Logging:
Level: ALL
IncludeExecutionData: true
Destinations:
- CloudWatchLogsLogGroup:
LogGroupArn: !GetAtt StateMachineLogGroup.Arn
Policies:
- AWSXrayWriteOnlyAccess
- Statement:
- Effect: Allow
Action:
- logs:CreateLogDelivery
- logs:GetLogDelivery
- logs:UpdateLogDelivery
- logs:DeleteLogDelivery
- logs:ListLogDeliveries
- logs:PutResourcePolicy
- logs:DescribeResourcePolicies
- logs:DescribeLogGroups
Resource: '*'
Tracing:
Enabled: true
Type: STANDARD
DefinitionSubstitutions:
ApiEndpoint: !Ref APIEndpoint
EBConnectionARN: !GetAtt EventBridgeConnection.Arn
Now that the State Machine is defined, let's also define EventBridge Connection along with the authorization credentials.
EventBridgeConnection:
Type: AWS::Events::Connection
Properties:
AuthorizationType: BASIC
AuthParameters:
BasicAuthParameters:
Password: !Ref EventBridgeConnectionPassword
Username: !Ref EventBridgeConnectionUsername
The resources - StateMachine and EventBridgeConnection are defined but for the StateMachine to successfully invoke the HTTP point, you would need to authorize the State Machine IAM execution policy as below -
- Statement:
- Effect: Allow
Action:
- events:RetrieveConnectionCredentials
Resource:
- !GetAtt EventBridgeConnection.Arn
- Effect: Allow
Action:
- secretsmanager:GetSecretValue
- secretsmanager:DescribeSecret
Resource:
- !Sub arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:events!connection/*
- Effect: Allow
Action:
- states:InvokeHTTPEndpoint
Resource:
- !Sub arn:aws:states:${AWS::Region}:${AWS::AccountId}:stateMachine:*
Note that, EventBridge Connection under the hood uses AWS Secrets Manager so you would also need to allow GetSecretValue and DescribeSecret.
HTTP Invoke state with it's advantages
Since HTTP invocation on Step Functions uses EventBridge Connection, it brings in the added advantage over invoking HTTP endpoints via Lambda Functions.
Enhanced execution insights: Step Functions provides you with built-in monitoring and logging capability of the State Machine execution which makes it easier to trace and debug the HTTP requests within the context of the workflow.
Streamlined workflow management: Directly integrating HTTP requests within Step Functions allows for more streamlined and cohesive workflow definitions. This makes it easier to manage and update workflows without needing to modify and redeploy Lambda functions
Error retry and handling: Step Functions' error handling techniques give you the flexibility to define the errors and their retry by handling the errors elegantly. Additionally, EventBridge Connections also retries failed requests by default.
HTTP Invoke state with it's disadvantages
Dependency on EventBridge Connections: The HTTP invoke state depends on EventBridge connections for managing authentication credentials. This adds extra setup steps and dependencies, which can make the process more complicated. Users need to create and manage EventBridge connections and related secrets in AWS Secrets Manager, which can be confusing for some.
No Support for Private Endpoints: The HTTP invoke state does not support private endpoints within a VPC. This limitation means it can't be used for secure, internal communication, forcing developers to use Lambda functions or other methods to interact with private APIs.
IAM Permissions Complexity: To use the HTTP invoke state, you need to set up specific IAM permissions for the state machine's role. This includes permissions to make HTTP requests, use the EventBridge connection, and access the connection's secret. Managing these permissions can add complexity and potential security risks if not configured correctly.
Error debugging: While error retry and handling are supported, error debugging becomes a challenge as HTTP requests could fail for various reasons - timeouts, authentication fail, missing header or parameter that can be hard to debug the error as it's an EventBridge Connection used under the hood.
Wrap up
Yes, orchestrating the HTTP invocations is made possible with Step Functions but it does have its advantages and disadvantages. While building Serverless applications or workflows, the trade-offs are important to consider.
At times, you would need the flexibility to make dynamic HTTP invocations which is possible on the Lambda function rather than via EventBridge Connection. Additionally, it would require you to be careful with the IAM execution policy to allow the needed Secrets.
