GRPC max size?

[arbesfeld]

We noticed an error today using the BigTable emulator. We were making large insert requests against a BigTable table and it was hanging. When we killed the emulator this message appeared:

grpc: received message length 18874667 exceeding the max size 4194304 Error: grpc: received message length 18874667 exceeding the max size 4194304
    at ClientReadableStream._emitStatusIfDone (/Users/mata/AppHub/logrocket/node_modules/google-cloud/node_modules/grpc/src/node/src/client.js:189:19)
    at ClientReadableStream._receiveStatus (/Users/mata/AppHub/logrocket/node_modules/google-cloud/node_modules/grpc/src/node/src/client.js:169:8)
    at /Users/mata/AppHub/logrocket/node_modules/google-cloud/node_modules/grpc/src/node/src/client.js:577:14

I read through the documentation and didn't see a mention of a max message size. Is this the case? Any workarounds that you know of for handling large (~10MB) data?

[callmehiphop]

@garye does this sound right? I was looking around in the v2 files just now and the closest thing I found was mention of a max insertion of 100000 mutations.

[garye]

I think they're hitting a gRPC limit, not a bigtable limit.
https://github.com/grpc/grpc-go/blob/71d2ea4f75286a63b606aca2422cd17ff37fd5b8/rpc_util.go#L242

This is meant to be consistent across grpc implementations:
grpc/grpc-go#780

The workaround is to just send smaller individual requests - any individual bulk mutation, for example, should be < 4MB.

[stephenplusplus]

@callmehiphop can we add a mention in the docs to slice up large requests to avoid this error?

[arbesfeld]

@garye is there a way to increase the GRPC limit?

[garye]

@arbesfeld It looks like no for node, at least based on searching around and this thread: https://groups.google.com/d/msg/grpc-io/UiCvZRS1B9I/WvUmc-RfBwAJ.

[bensimian]

Has this been updated to provide an API org option to increase the size?