Add FLAN-T5 #1398

yifanmai · 2023-03-01T23:24:23Z

percyliang · 2023-03-01T23:43:07Z

src/helm/proxy/clients/together_client.py

@@ -12,6 +13,12 @@
 from .client import Client, wrap_request_time, truncate_sequence


+MODEL_ALIASES = {
+    "bloomz": "bloomz-176b-alpa",


This is fine, but it's worth noting that the exact Together model will have an impact on efficiency metrics.

Actually I do wonder if we want to use the actual Together names just to be completely transparent...

I don't like the idea of using Together names because they haven't been particularly stable. Most of the current Together model names are already stale. Also I don't think it makes sense to have implementation details in the name that we show users.

The Together names should be (more) stable now. I think it's easier to merge than to separate later. If we ever need to distinguish between different implementations, we need to separate it out (certainly will be different for efficiency, but I think the models might not provide exactly the same predictions).

I agree it'd be nice to have simpler names for users - we can perhaps use aliases on our side which resolve immediately to an implementation.

How about:

User sees "together/bloomz"

Cache key and raw request both use "bloomz-176b-alpa"

Then if the Together implementation changes name, it will automatically invalidate the cache.

I'm thinking about the H3 model which is named "h3-2.7b-h3" (see #1404), which just seems strange to expose to users.

That works - the caching of the underlying model makes me feel better about this. And if there is a change, we can always migrate.. Ideally when the user makes a request, it will map together/bloomz to together/bloomz-176b-alpa (or whatever is the version de jour), and the user could also request together/bloomz-176b-alpa to get particular implementations if they want.

teetone · 2023-03-01T23:43:16Z

src/helm/benchmark/window_services/window_service_factory.py

@@ -62,7 +62,7 @@ def get_window_service(model_name: str, service: TokenizerService) -> WindowServ
            window_service = SantaCoderWindowService(service)
        elif model_name == "huggingface/gpt2":
            window_service = GPT2WindowService(service)
-        elif model_name == "together/bloom":
+        elif model_name == "together/bloom" or model_name == "together/bloomz":
            window_service = BloomWindowService(service)


I don't think it's the same tokenizer: https://huggingface.co/bigscience/bloomz#cpu

Added BLOOMZ tokenizer.

As far as I can tell, it's the exact same tokenizer, just under a different name.

teetone · 2023-03-01T23:44:29Z

src/helm/proxy/clients/together_client.py

@@ -12,6 +13,12 @@
 from .client import Client, wrap_request_time, truncate_sequence


+MODEL_ALIASES = {
+    "bloomz": "bloomz-176b-alpa",


-alpha: Will there be future versions?

According to Together: "the convention is <model-name>-<size>-<framework>"

There might be alternate implementations.

If there will be alternate implementations, is it okay to cache results as plain bloomz?

As discussed in the other thread, I changed this to cache results as the full name including the framework e.g. bloomz-176b-alpa.

teetone · 2023-03-01T23:45:38Z

src/helm/benchmark/window_services/window_service_factory.py

@@ -78,7 +78,7 @@ def get_window_service(model_name: str, service: TokenizerService) -> WindowServ
            window_service = OPTWindowService(service)
        elif model_name == "together/t0pp":
            window_service = T0ppWindowService(service)
-        elif model_name == "together/t5-11b":
+        elif model_name == "together/t5-11b" or model_name == "together/flan-t5-xxl":


I think this model also has its own tokenizer too.

Added tokenizer.

teetone · 2023-03-01T23:47:31Z

src/helm/proxy/models.py

@@ -323,6 +333,15 @@ def engine(self) -> str:
        # Does not support echo=True
        tags=[TEXT_MODEL_TAG, LIMITED_FUNCTIONALITY_TEXT_MODEL_TAG, ABLATION_MODEL_TAG, NO_NEWLINES_TAG],
    ),
+    Model(
+        group="together",


Do we have to update schema.yaml - remove todo?

teetone

Just a few more comments. Could we also add unit tests for the new window services?

teetone · 2023-03-02T21:30:20Z

src/helm/benchmark/window_services/bloomz_window_service.py

+    def max_sequence_length(self) -> int:
+        """
+        The model was trained with a sequence length of 2,048.
+        Source: https://huggingface.co/bigscience/bloom


Update the link. Also, it's probably correct, but just want to make sure the sequence length is 2048.

teetone · 2023-03-02T21:34:44Z

src/helm/benchmark/window_services/flan_t5_window_service.py

+    @property
+    def max_sequence_length(self) -> int:
+        """Return the max sequence length."""
+        # From https://arxiv.org/pdf/1910.10683.pdf, "we use a maximum sequence length of 512".


This comment is for T5. Can we update this?

Removed the stale links. I checked on Hugging Face AutoTokenizer that both of these are correct.

teetone · 2023-03-02T21:37:32Z

src/helm/proxy/clients/together_client.py

@@ -12,6 +13,12 @@
 from .client import Client, wrap_request_time, truncate_sequence


+MODEL_ALIASES = {
+    "bloomz": "bloomz-176b-alpa",


If there will be alternate implementations, is it okay to cache results as plain bloomz?

teetone · 2023-03-02T21:37:51Z

src/helm/proxy/clients/together_client.py

@@ -12,6 +13,12 @@
 from .client import Client, wrap_request_time, truncate_sequence


+MODEL_ALIASES = {
+    "bloomz": "bloomz-176b-alpa",


Is this a typo: alpa?

No, it's https://opt.alpa.ai/

yifanmai · 2023-03-08T21:21:26Z

This is ready for review again. PTAL

percyliang

Great!

yifanmai · 2023-03-09T05:00:17Z

@teetone could you take another look? This PR is blocked on your review.

yifanmai · 2023-03-09T16:54:52Z

Thanks!

yifanmai requested review from percyliang and teetone March 1, 2023 23:24

percyliang reviewed Mar 1, 2023

View reviewed changes

teetone requested changes Mar 1, 2023

View reviewed changes

yifanmai requested review from teetone and percyliang March 2, 2023 18:57

teetone requested changes Mar 2, 2023

View reviewed changes

yifanmai force-pushed the yifanmai/flan-t5 branch from b055277 to cc6b7c5 Compare March 6, 2023 21:30

yifanmai requested a review from teetone March 8, 2023 19:23

yifanmai added 3 commits March 8, 2023 13:09

Add Flan-T5

c1e78e6

Remove links

23cb56c

Remove BLOOMZ

184de70

yifanmai force-pushed the yifanmai/flan-t5 branch from 1770731 to 184de70 Compare March 8, 2023 21:09

percyliang approved these changes Mar 8, 2023

View reviewed changes

teetone approved these changes Mar 9, 2023

View reviewed changes

yifanmai merged commit 3e55882 into main Mar 9, 2023

yifanmai deleted the yifanmai/flan-t5 branch March 9, 2023 16:54

JoelNiklaus pushed a commit to JoelNiklaus/helm that referenced this pull request Mar 9, 2023

Add FLAN-T5 (stanford-crfm#1398)

9b64768

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add FLAN-T5 #1398

Add FLAN-T5 #1398

yifanmai commented Mar 1, 2023 •

edited

Loading

percyliang Mar 1, 2023

percyliang Mar 1, 2023

yifanmai Mar 2, 2023

percyliang Mar 3, 2023

yifanmai Mar 4, 2023

percyliang Mar 4, 2023 •

edited

Loading

teetone Mar 1, 2023

yifanmai Mar 2, 2023

teetone Mar 1, 2023

yifanmai Mar 2, 2023

teetone Mar 2, 2023

yifanmai Mar 6, 2023

teetone Mar 1, 2023

yifanmai Mar 2, 2023

teetone Mar 1, 2023

yifanmai Mar 2, 2023

teetone left a comment

teetone Mar 2, 2023

teetone Mar 2, 2023

yifanmai Mar 6, 2023

teetone Mar 2, 2023

teetone Mar 2, 2023

percyliang Mar 3, 2023

yifanmai commented Mar 8, 2023

percyliang left a comment

yifanmai commented Mar 9, 2023

yifanmai commented Mar 9, 2023

Add FLAN-T5 #1398

Add FLAN-T5 #1398

Conversation

yifanmai commented Mar 1, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

percyliang Mar 4, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

teetone left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yifanmai commented Mar 8, 2023

percyliang left a comment

Choose a reason for hiding this comment

yifanmai commented Mar 9, 2023

yifanmai commented Mar 9, 2023

yifanmai commented Mar 1, 2023 •

edited

Loading

percyliang Mar 4, 2023 •

edited

Loading