Skip to content

feat: Add support for increasing partitions in python - #74

Merged
palmere-google merged 3 commits into
masterfrom
palmere-increase
Dec 14, 2020
Merged

feat: Add support for increasing partitions in python#74
palmere-google merged 3 commits into
masterfrom
palmere-increase

Conversation

@palmere-google

Copy link
Copy Markdown
Contributor

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> 🦕

@palmere-google
palmere-google requested a review from a team December 10, 2020 04:32
@product-auto-label product-auto-label Bot added the api: pubsublite Issues related to the googleapis/python-pubsublite API. label Dec 10, 2020
@google-cla google-cla Bot added the cla: yes This human has signed the Contributor License Agreement. label Dec 10, 2020
Comment thread google/cloud/pubsublite/internal/wire/make_publisher.py Outdated
Comment thread google/cloud/pubsublite/internal/wire/partition_count_watching_publisher.py Outdated
Comment thread google/cloud/pubsublite/internal/wire/partition_count_watching_publisher.py Outdated
Comment thread google/cloud/pubsublite/internal/wire/partition_count_watching_publisher.py Outdated
def set_box():
box.val = PartitionCountWatcherImpl(mock_admin, topic, 0.001)

# Initialize publisher on another thread with a different event loop.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any asyncio class cannot be accessed from a different event loop, or it is incorrect.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, before I didn't understand this was necessarily constructed in the same event loop as the publisher.

I still kind of think the property that everything works even if you call aenter from a different thread than you constructed the watcher is nice. I'm happy to remove it though if you think that would be better

@dpcollins-google dpcollins-google Dec 11, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a utility to the testing helpers that does exactly this? I think this is a good idea, and we should do this more places. I.e. a "runOnThread" function that takes a Callable[[], T] and returns a T

@palmere-google
palmere-google force-pushed the palmere-increase branch 3 times, most recently from 4528a93 to 43ea68d Compare December 10, 2020 19:46
Partition(index): self._publisher_factory(Partition(index))
for index in range(current_count, partition_count)
}
await asyncio.gather(*[p.__aenter__() for p in new_publishers.values()])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

run nox from the root directory before merging this.

Comment thread .gitignore Outdated
.installed.cfg
lib
lib64
venv

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

put this down with 'env/' and add the slash.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@dpcollins-google dpcollins-google changed the title Add support for increasing partitions in python feat: Add support for increasing partitions in python Dec 14, 2020
@palmere-google
palmere-google merged commit e117a1a into master Dec 14, 2020
@anguillanneuf
anguillanneuf deleted the palmere-increase branch March 25, 2022 22:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: pubsublite Issues related to the googleapis/python-pubsublite API. cla: yes This human has signed the Contributor License Agreement.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants