-
Notifications
You must be signed in to change notification settings - Fork 7.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HIGH PRIORITY - Attach images to prompts #332
Conversation
atrokhym
commented
Nov 19, 2024
Could you please post a video of how the interface works? This is a major feature involving vision, so it may have to be prioritized against the roadmap in the coming days. Thanks for the submission! |
We also have many models and providers, probably we need to disable such features when untested provider/model are used |
app/components/chat/BaseChat.tsx
Outdated
messages, | ||
children, // Add this | ||
}, ref) => { | ||
console.log(provider); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this should be removed
app/components/chat/UserMessage.tsx
Outdated
// function sanitizeUserMessage(content: string) { | ||
// return content.replace(modificationsRegex, '').replace(MODEL_REGEX, 'Using: $1').replace(PROVIDER_REGEX, ' ($1)\n\n').trim(); | ||
// } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this commented block should be removed
app/lib/.server/llm/stream-text.ts
Outdated
// Extract model | ||
const modelMatch = message.content.match(MODEL_REGEX); | ||
// const modelMatch = message.content.match(MODEL_REGEX); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this comment should be removed
app/lib/.server/llm/stream-text.ts
Outdated
const model = modelMatch ? modelMatch[1] : DEFAULT_MODEL; | ||
|
||
// Extract provider | ||
const providerMatch = message.content.match(PROVIDER_REGEX); | ||
// const providerMatch = message.content.match(PROVIDER_REGEX); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this comment should be removed
app/lib/.server/llm/stream-text.ts
Outdated
if (item.type === 'text') { | ||
return { | ||
type: 'text', | ||
text: item.text?.replace(/\[Model:.*?\]\n\n/, '').replace(/\[Provider:.*?\]\n\n/, '') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
these regexes are being repeated multiple times, can you use MODEL_REGEX and PROVIDER_REGEX or add them to constants.ts?
app/routes/api.chat.ts
Outdated
// console.log('=== API CHAT LOGGING START ==='); | ||
// console.log('StreamText:', JSON.stringify({ | ||
// messages, | ||
// result, | ||
// }, null, 2)); | ||
// console.log('=== API CHAT LOGGING END ==='); | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
these comments should be removed
package.json
Outdated
@@ -73,6 +73,7 @@ | |||
"jose": "^5.6.3", | |||
"js-cookie": "^3.0.5", | |||
"jszip": "^3.10.1", | |||
"lucide-react": "^0.460.0", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this package is not needed and should be removed
pnpm-lock.yaml
Outdated
[email protected]([email protected]): | ||
dependencies: | ||
react: 18.3.1 | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it should be removed
@@ -204,7 +203,7 @@ export const EditorPanel = memo( | |||
const isActive = activeTerminal === index; | |||
|
|||
return ( | |||
<React.Fragment key={index}> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what is the reason of removing the index key?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removing that key will throw an error, which is why it's a React.Fragment instead of <></> - please revert
app/components/chat/UserMessage.tsx
Outdated
const textContent = Array.isArray(sanitizedContent) | ||
? sanitizedContent.find(item => item.type === 'text')?.text || '' | ||
: sanitizedContent; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can this be merged/inserted inside sanitizeUserMessage
function? otherwise you are unnecessarily looping twice over the contents
The upload button could be only available for those models with vision capabilities and frozen/disabled for the others. The dynamic LLMs can be fetched from the OpenRouter API and filter those which have |
@atrokhym Thank you so much for this, I hope we can get it merged in soon! I appreciate your reviews a ton too @pjmartorell, fantastic! |
@pjmartorell |
@atrokhym Thank you! Let me know if you can get around to reviewing again @pjmartorell! |
|
This function is really great! May I ask when the merger will take place? |
@atrokhym Would you say this is all ready to go? I can review this within the next couple of days too! |
it was ready for merge couple of days back...but now more conflicts from main branch are coming...so i need to resolve them first...will do ASAP |
Sounds good, thank you so much @atrokhym! I would love to showcase this for my update video on Sunday! |
@atrokhym This is looking phenomenal! There are a lot of merge conflicts with main right now - could you resolve those/rebase your branch on main? Also I have a couple of suggestions to really make this PR shine:
Thank you for all your work on this! |
@coleam00 - could you do review, please? |
Looks good @atrokhym, thank you for fixing the conflicts! Did you get a chance to see my couple of suggestions as well? |
yes and i think item-1 looks much better now for item-2 - as of now i don't know to do it....i would need to do some research... |
Is it possible to add the ability to paste an image from the clipboard? |
For this option i have changed the code of the BaseChat.tsx : /*
* @ts-nocheck
* Preventing TS checks with files presented in the video for a better presentation.
*/
import type { Message } from 'ai';
import React, { type RefCallback, useEffect, useState } from 'react';
import { ClientOnly } from 'remix-utils/client-only';
import { Menu } from '~/components/sidebar/Menu.client';
import { IconButton } from '~/components/ui/IconButton';
import { Workbench } from '~/components/workbench/Workbench.client';
import { classNames } from '~/utils/classNames';
import { MODEL_LIST, PROVIDER_LIST, initializeModelList } from '~/utils/constants';
import { Messages } from './Messages.client';
import { SendButton } from './SendButton.client';
import { APIKeyManager } from './APIKeyManager';
import Cookies from 'js-cookie';
import * as Tooltip from '@radix-ui/react-tooltip';
import styles from './BaseChat.module.scss';
import type { ProviderInfo } from '~/utils/types';
import { ExportChatButton } from '~/components/chat/chatExportAndImport/ExportChatButton';
import { ImportButtons } from '~/components/chat/chatExportAndImport/ImportButtons';
import { ExamplePrompts } from '~/components/chat/ExamplePrompts';
import FilePreview from './FilePreview';
// @ts-ignore TODO: Introduce proper types
// eslint-disable-next-line @typescript-eslint/no-unused-vars
const ModelSelector = ({ model, setModel, provider, setProvider, modelList, providerList, apiKeys }) => {
return (
<div className="mb-2 flex gap-2 flex-col sm:flex-row">
<select
value={provider?.name}
onChange={(e) => {
setProvider(providerList.find((p: ProviderInfo) => p.name === e.target.value));
const firstModel = [...modelList].find((m) => m.provider == e.target.value);
setModel(firstModel ? firstModel.name : '');
}}
className="flex-1 p-2 rounded-lg border border-bolt-elements-borderColor bg-bolt-elements-prompt-background text-bolt-elements-textPrimary focus:outline-none focus:ring-2 focus:ring-bolt-elements-focus transition-all"
>
{providerList.map((provider: ProviderInfo) => (
<option key={provider.name} value={provider.name}>
{provider.name}
</option>
))}
</select>
<select
key={provider?.name}
value={model}
onChange={(e) => setModel(e.target.value)}
className="flex-1 p-2 rounded-lg border border-bolt-elements-borderColor bg-bolt-elements-prompt-background text-bolt-elements-textPrimary focus:outline-none focus:ring-2 focus:ring-bolt-elements-focus transition-all lg:max-w-[70%] "
>
{[...modelList]
.filter((e) => e.provider == provider?.name && e.name)
.map((modelOption) => (
<option key={modelOption.name} value={modelOption.name}>
{modelOption.label}
</option>
))}
</select>
</div>
);
};
const TEXTAREA_MIN_HEIGHT = 76;
interface BaseChatProps {
textareaRef?: React.RefObject<HTMLTextAreaElement> | undefined;
messageRef?: RefCallback<HTMLDivElement> | undefined;
scrollRef?: RefCallback<HTMLDivElement> | undefined;
showChat?: boolean;
chatStarted?: boolean;
isStreaming?: boolean;
messages?: Message[];
description?: string;
enhancingPrompt?: boolean;
promptEnhanced?: boolean;
input?: string;
model?: string;
setModel?: (model: string) => void;
provider?: ProviderInfo;
setProvider?: (provider: ProviderInfo) => void;
handleStop?: () => void;
sendMessage?: (event: React.UIEvent, messageInput?: string) => void;
handleInputChange?: (event: React.ChangeEvent<HTMLTextAreaElement>) => void;
enhancePrompt?: () => void;
importChat?: (description: string, messages: Message[]) => Promise<void>;
exportChat?: () => void;
uploadedFiles?: File[];
setUploadedFiles?: (files: File[]) => void;
imageDataList?: string[];
setImageDataList?: (dataList: string[]) => void;
}
export const BaseChat = React.forwardRef<HTMLDivElement, BaseChatProps>(
(
{
textareaRef,
messageRef,
scrollRef,
showChat = true,
chatStarted = false,
isStreaming = false,
model,
setModel,
provider,
setProvider,
input = '',
enhancingPrompt,
handleInputChange,
promptEnhanced,
enhancePrompt,
sendMessage,
handleStop,
importChat,
exportChat,
uploadedFiles = [],
setUploadedFiles,
imageDataList = [],
setImageDataList,
messages,
},
ref,
) => {
const TEXTAREA_MAX_HEIGHT = chatStarted ? 400 : 200;
const [apiKeys, setApiKeys] = useState<Record<string, string>>({});
const [modelList, setModelList] = useState(MODEL_LIST);
useEffect(() => {
// Load API keys from cookies on component mount
try {
const storedApiKeys = Cookies.get('apiKeys');
if (storedApiKeys) {
const parsedKeys = JSON.parse(storedApiKeys);
if (typeof parsedKeys === 'object' && parsedKeys !== null) {
setApiKeys(parsedKeys);
}
}
} catch (error) {
console.error('Error loading API keys from cookies:', error);
// Clear invalid cookie data
Cookies.remove('apiKeys');
}
initializeModelList().then((modelList) => {
setModelList(modelList);
});
}, []);
const updateApiKey = (provider: string, key: string) => {
try {
const updatedApiKeys = { ...apiKeys, [provider]: key };
setApiKeys(updatedApiKeys);
// Save updated API keys to cookies with 30 day expiry and secure settings
Cookies.set('apiKeys', JSON.stringify(updatedApiKeys), {
expires: 30, // 30 days
secure: true, // Only send over HTTPS
sameSite: 'strict', // Protect against CSRF
path: '/', // Accessible across the site
});
} catch (error) {
console.error('Error saving API keys to cookies:', error);
}
};
const handleFileUpload = () => {
const input = document.createElement('input');
input.type = 'file';
input.accept = 'image/*';
input.onchange = async (e) => {
const file = (e.target as HTMLInputElement).files?.[0];
if (file) {
const reader = new FileReader();
reader.onload = (e) => {
const base64Image = e.target?.result as string;
setUploadedFiles?.([...uploadedFiles, file]);
setImageDataList?.([...imageDataList, base64Image]);
};
reader.readAsDataURL(file);
}
};
input.click();
};
const handlePaste = async (e: React.ClipboardEvent) => {
const items = e.clipboardData?.items;
if (!items) {
return;
}
for (const item of items) {
if (item.type.startsWith('image/')) {
e.preventDefault();
const file = item.getAsFile();
if (file) {
const reader = new FileReader();
reader.onload = (e) => {
const base64Image = e.target?.result as string;
setUploadedFiles?.([...uploadedFiles, file]);
setImageDataList?.([...imageDataList, base64Image]);
};
reader.readAsDataURL(file);
}
break;
}
}
};
const baseChat = (
<div
ref={ref}
className={classNames(
styles.BaseChat,
'relative flex flex-col lg:flex-row h-full w-full overflow-hidden bg-bolt-elements-background-depth-1',
)}
data-chat-visible={showChat}
>
<ClientOnly>{() => <Menu />}</ClientOnly>
<div ref={scrollRef} className="flex flex-col lg:flex-row overflow-y-auto w-full h-full">
<div className={classNames(styles.Chat, 'flex flex-col flex-grow lg:min-w-[var(--chat-min-width)] h-full')}>
{!chatStarted && (
<div id="intro" className="mt-[26vh] max-w-chat mx-auto text-center px-4 lg:px-0">
<h1 className="text-3xl lg:text-6xl font-bold text-bolt-elements-textPrimary mb-4 animate-fade-in">
Where ideas begin
</h1>
<p className="text-md lg:text-xl mb-8 text-bolt-elements-textSecondary animate-fade-in animation-delay-200">
Bring ideas to life in seconds or get help on existing projects.
</p>
</div>
)}
<div
className={classNames('pt-6 px-2 sm:px-6', {
'h-full flex flex-col': chatStarted,
})}
>
<ClientOnly>
{() => {
return chatStarted ? (
<Messages
ref={messageRef}
className="flex flex-col w-full flex-1 max-w-chat pb-6 mx-auto z-1"
messages={messages}
isStreaming={isStreaming}
/>
) : null;
}}
</ClientOnly>
<div
className={classNames(
' bg-bolt-elements-background-depth-2 p-3 rounded-lg border border-bolt-elements-borderColor relative w-full max-w-chat mx-auto z-prompt mb-6',
{
'sticky bottom-2': chatStarted,
},
)}
>
<ModelSelector
key={provider?.name + ':' + modelList.length}
model={model}
setModel={setModel}
modelList={modelList}
provider={provider}
setProvider={setProvider}
providerList={PROVIDER_LIST}
apiKeys={apiKeys}
/>
{provider && (
<APIKeyManager
provider={provider}
apiKey={apiKeys[provider.name] || ''}
setApiKey={(key) => updateApiKey(provider.name, key)}
/>
)}
<FilePreview
files={uploadedFiles}
imageDataList={imageDataList}
onRemove={(index) => {
setUploadedFiles?.(uploadedFiles.filter((_, i) => i !== index));
setImageDataList?.(imageDataList.filter((_, i) => i !== index));
}}
/>
<div
className={classNames(
'shadow-lg border border-bolt-elements-borderColor bg-bolt-elements-prompt-background backdrop-filter backdrop-blur-[8px] rounded-lg overflow-hidden transition-all',
)}
>
<textarea
ref={textareaRef}
className={`w-full pl-4 pt-4 pr-16 focus:outline-none focus:ring-0 focus:border-none focus:shadow-none resize-none text-md text-bolt-elements-textPrimary placeholder-bolt-elements-textTertiary bg-transparent transition-all`}
onKeyDown={(event) => {
if (event.key === 'Enter') {
if (event.shiftKey) {
return;
}
event.preventDefault();
sendMessage?.(event);
}
}}
value={input}
onChange={(event) => {
handleInputChange?.(event);
}}
onPaste={handlePaste}
style={{
minHeight: TEXTAREA_MIN_HEIGHT,
maxHeight: TEXTAREA_MAX_HEIGHT,
}}
placeholder="How can Bolt help you today?"
translate="no"
/>
<ClientOnly>
{() => (
<SendButton
show={input.length > 0 || isStreaming || uploadedFiles.length > 0}
isStreaming={isStreaming}
onClick={(event) => {
if (isStreaming) {
handleStop?.();
return;
}
if (input.length > 0 || uploadedFiles.length > 0) {
sendMessage?.(event);
}
}}
/>
)}
</ClientOnly>
<div className="flex justify-between items-center text-sm p-4 pt-2">
<div className="flex gap-1 items-center">
<IconButton title="Upload file" className="transition-all" onClick={() => handleFileUpload()}>
<div className="i-ph:paperclip text-xl"></div>
</IconButton>
<IconButton
title="Enhance prompt"
disabled={input.length === 0 || enhancingPrompt}
className={classNames(
'transition-all',
enhancingPrompt ? 'opacity-100' : '',
promptEnhanced ? 'text-bolt-elements-item-contentAccent' : '',
promptEnhanced ? 'pr-1.5' : '',
promptEnhanced ? 'enabled:hover:bg-bolt-elements-item-backgroundAccent' : '',
)}
onClick={() => enhancePrompt?.()}
>
{enhancingPrompt ? (
<>
<div className="i-svg-spinners:90-ring-with-bg text-bolt-elements-loader-progress text-xl animate-spin"></div>
<div className="ml-1.5">Enhancing prompt...</div>
</>
) : (
<>
<div className="i-bolt:stars text-xl"></div>
{promptEnhanced && <div className="ml-1.5">Prompt enhanced</div>}
</>
)}
</IconButton>
{chatStarted && <ClientOnly>{() => <ExportChatButton exportChat={exportChat} />}</ClientOnly>}
</div>
{input.length > 3 ? (
<div className="text-xs text-bolt-elements-textTertiary">
Use <kbd className="kdb px-1.5 py-0.5 rounded bg-bolt-elements-background-depth-2">Shift</kbd> +{' '}
<kbd className="kdb px-1.5 py-0.5 rounded bg-bolt-elements-background-depth-2">Return</kbd> for
a new line
</div>
) : null}
</div>
</div>
</div>
</div>
{!chatStarted && ImportButtons(importChat)}
{!chatStarted && ExamplePrompts(sendMessage)}
</div>
<ClientOnly>{() => <Workbench chatStarted={chatStarted} isStreaming={isStreaming} />}</ClientOnly>
</div>
</div>
);
return <Tooltip.Provider delayDuration={200}>{baseChat}</Tooltip.Provider>;
},
);
Br,
Stijn |
@Stijnus - thank you alot for those 2 improvement! I added them into branch |
i run through those comments a while ago (resolved them as suggested) |
cool! thx everybody! |
Hallelujah! Thank you guys for the awesome work. |
HIGH PRIORITY - Attach images to prompts