Customer is seeing “too many streams” error (real-time)

Customer is seeing “too many streams” error (Streaming)

Introduction

When using Streaming, customers may sometimes encounter a 4102 Error: “This account has exceeded the number of allowed streams”. This is due to the customer exceeding the throttle limit assigned to their account. For example, attempting 110 concurrent streams at a throttle limit of 100.

A common cause of this issue is that the customer is not terminating their Streaming sessions.

If the customer requests very high limits (i.e., >500), the current communication channel is to ask in #ask-streaming. They will determine whether the appropriate infrastructure is in place to handle this level of limits, and will likely ask for more information on traffic patterns, peak traffic, etc. to help shape traffic accordingly.

Specifically for Streaming, there is a total capacity limit with considerations to cost and server capacity. When resolving this, the Support Engineer must surface the request to the Engineering team with the anticipated concurrency limit based on the customer’s historical traffic.

This is also related to throttling issues faced in async transcriptions:

Customer is being throttled (Async)

Procedures

1. Acquire details of the session from the customer.

   • Error message

   • Time and date of error occurrence

   • Session ID

2. View the error log to verify the issue. Then identify how many times they have been throttled in the timeframe.

   • This can be done on Datadog by viewing this graph and checking to see if the customer’s account ID shows up.

3. Check if they are continuing to have throttled jobs. This is important to see if it’s a transient issue or just a spike in traffic.

   • Notify the customer whether we can still see the issue or if it has subsided.

4. It’s also necessary to check if the customer is terminating their Streaming sessions. This is a common cause of smaller customers exceeding the concurrency limit as active sessions are not terminated.

5. After reviewing Datadog for quantitative analysis, if the issue persists and the customer requests a concurrency increase, the Support Engineer must contact Engineering (if the requested concurrency is over 500) in #ask-streaming to see if this can be done immediately or only after appropriate infrastructure changes are made.

Resources

• Where to ask Engineering: #ask-streaming

• API Support Manager: Lee Vaughn

• Tool for gathering customer information: Retool

• Tool for analyzing logs: Datadog

Appendices

1. Customer ID: an internal ID used to identify the customer in our database.

2. Customer email: the email that is associated with the account and its API usage.

3. Throttle Limit/Concurrency Limit: Maximum number of audio files the user can process in parallel