support Bedrock by kevintruong · Pull Request #81 · Tanuki/tanuki.py

kevintruong · 2023-11-21T22:58:12Z

No description provided.

kevintruong · 2023-11-21T23:00:54Z

hi @JackHopkins
WIP for bedrock support
I'm testing at my side, working. But need align with you to make a good progress for the MR.

How can I can contact you ...etc (discord invite is invalid)

thanks

dnlkwak · 2023-11-21T23:18:45Z

Hey Kevin - You can join here: https://discord.gg/ymmV6FKg LMK if you can't and I can add you via email! Daniel Kwak, CEO Paperplane ( https://www.paperplane.ai/ ) 404-704-2999

…

On Tue, Nov 21, 2023 at 6:01 PM, Kevin Truong < ***@***.*** > wrote: hi @ JackHopkins ( https://github.com/JackHopkins ) WIP for bedrock support I'm testing at my side, working. But need align with you to make a good progress for the MR. How can I can contact you ...etc (discord invite is invalid) thanks — Reply to this email directly, view it on GitHub ( #81 (comment) ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AHVPE5SGIJAPRSL24LSID7LYFUXDDAVCNFSM6AAAAAA7VKKPTOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRRHAZTEOBQGQ ). You are receiving this because you are subscribed to this thread. Message ID: <monkeypatch/monkeypatch. py/pull/81/c1821832804 @ github. com>

JackHopkins

Can you have a look at these comments and let me know what you think?

Additionally, it'd be really helpful to have some automated tests for Bedrock (possibly mocking out the Bedrock client) so we can ensure that we don't break anything as we continue development.

JackHopkins · 2023-11-22T07:31:36Z

src/monkey_patch/language_models/bedrock_api.py

+                if ("error" in response and
+                        "code" in response["error"] and
+                        response["error"]["code"] == 'invalid_api_key'):
+                    raise Exception(f"The supplied OpenAI API key {self.api_key} is invalid")


Please change to "Anthropic API key"

JackHopkins · 2023-11-22T07:31:45Z

src/monkey_patch/language_models/bedrock_api.py

+                continue
+
+        if not choice:
+            raise Exception("OpenAI API failed to generate a response")


Same again here.

JackHopkins · 2023-11-22T07:34:21Z

src/monkey_patch/language_models/bedrock_api.py

+            modelId=model,
+            contentType="application/json",
+            accept="application/json")
+        return json.loads(response.get('body').read().decode())['completion']


We shouldn't be returning this here - we should loop the counter below in the case where the bedrock_runtime throws an exception.

JackHopkins · 2023-11-22T07:40:34Z

src/monkey_patch/language_models/language_modeler.py

+            return 'anthropic.claude-instant-v1'
+
+        elif api_model_name == 'openai':
+            return ""


Let me double check to ensure whether this works.

JackHopkins · 2023-11-22T07:39:52Z

src/monkey_patch/language_models/bedrock_api.py

+                    "Content-Type": "application/json",
+                }
+                response = requests.post(
+                    OPENAI_URL, headers=openai_headers, json=params, timeout=50


We should replace this with the invoke_model you have above.

JackHopkins · 2023-11-22T07:44:07Z

src/monkey_patch/function_modeler.py

        training_file_id = response["id"]
        # submit the finetuning job
        try:
            finetuning_response = openai.FineTuningJob.create(training_file=training_file_id, model="gpt-3.5-turbo",


We need to figure out a way to call the logic to trigger a finetune in the relevant LLM provider class (i.e OpenAI, Bedrock, etc) - this is not the right place for this logic to live now we are adding support for more providers.

JackHopkins · 2023-11-22T07:44:24Z

src/monkey_patch/language_models/bedrock_api.py

+from monkey_patch.language_models.llm_api_abc import LLM_Api
+import os
+
+OPENAI_URL = "https://api.openai.com/v1/chat/completions"


This can be removed.

JackHopkins · 2023-11-22T07:45:21Z

src/monkey_patch/language_models/bedrock_api.py

+OPENAI_URL = "https://api.openai.com/v1/chat/completions"
+import requests
+
+AWS_REGION = os.environ.get("AWS_DEFAULT_REGION", "us-east-1")


We should instantiate all envvars / bedrock specific logic inside the init of the Bedrock class - catching and reraising any thrown exceptions.

JackHopkins · 2023-11-22T07:46:45Z

src/monkey_patch/language_models/language_modeler.py

 from monkey_patch.utils import approximate_token_count


+class ApiModelFactory:


Can we please have this factory inside its own file.

JackHopkins · 2023-11-22T07:48:05Z

src/monkey_patch/monkey.py

    logging.basicConfig(level=ALIGN_LEVEL_NUM)
    logger = logger_factory(__name__)
-    language_modeler = LanguageModel()
+    language_modeler = LanguageModel(api_model=os.getenv('API_MODEL'))


We should add the API_MODEL envvar to .example.env to make it obvious that it is required.

JackHopkins reviewed Nov 22, 2023

View reviewed changes

kevintruong added 2 commits November 23, 2023 06:08

support Bedrock

977ee69

fixed some comments

b411af8

kevintruong force-pushed the fea/bedrock branch from 4422bae to b411af8 Compare November 22, 2023 23:09

		from monkey_patch.utils import approximate_token_count


		class ApiModelFactory:

Conversation

kevintruong commented Nov 21, 2023

Uh oh!

kevintruong commented Nov 21, 2023

Uh oh!

dnlkwak commented Nov 21, 2023 via email

Uh oh!

JackHopkins left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants