WIP / DEVELOP / PROTOTYPE

_______________________ WIP DOCUMENT

OVERVIEW

Conversational Assistant powered by ChatGPT

WHAT IS IT?

It’s a conversational assistant designed for exploring various creators and related artists. It offers guided and categorized modes, allowing for an easy and interactive experience through a video game-style interface.

WHERE?

This takes place in a large installation with a big screen. Users interact using a simple device, which could be a mechanical or touch keyboard, a gamepad, a custom four-button panel, or even a microphone.

SCREENSHOTS

FEATURES

• A Guided and categorized Creators Explorer:
• A Conversation Assistant.
• Powered by OpenAI ChatGPT API and/or other LLM local engines.
• Designed to dive into different types of creators or artists,
  as well as Explore other similar or related ones.
• Ask and get similar creators.
  - Summarizations, Sentences or Keywords.
• Response Text-To-Voice using ElevenLabs.

WORKFLOW

• Prompt a Text in the input box.
  Ask by keyboard/touch screen or even microphone.

• Play around the clickable floating menu.

• Explore for different Creators Areas or Categories:
  - Music Band
  - Book Writer
  - Film Director
  - Film Actor
  - Film Title
  - Illustrator
  - Painter
  - Philosopher

• Get Similar/Related
  - Creators
  - Music albums
  - Books
  - Paintings
  - Films.
  By the author or related to his style, his artistic movement, friendship, countryman or contemporary, …etc.

• Enjoy a guided conversation. Ask for a prompt and Get:
  - Summarizations.
  - Sentences or descriptive key phrases.
  - List of keywords.
  - Similar to.

• Customize the response size:
  - Short, Medium, Long, or Unlimited.
• Text-to-speech from the response.
  - Using ElevenLabs API.
• Videogame styled:
  - State-of-the-Art interface rendering and animations.
  - Responsive layout for every screen size: landscape or portrait.
  - Sony Playstation / Microsoft XBOX game consoles inspired UI workflow.
  - UI sounds for a full nice experience.

WORKFLOW / VIDEO CAPTURES

• chatSURF 3/4
• chatSURF 1/4

UPGRADES / IDEAS / TODO

• Input text using the Voice-To-Text feature using OpenAI Whisper API.
• Query search after the response to get and auto-play music songs or even videos from YouTube, using the Google API or SerpApi.

RELATED MODULES / UPGRADES / OTHER PROJECTS

• SUBTITLE TEXT PLAYER
  - Subtitles .srt files reader/explorer player.
  - Layouted and styled rendering.
  https://github.com/moebiussurfing/ofxSurfingTextSubtitle

• BIG SPACE TEXT PLAYER
  - Animated text paragraphs on a layout canvas divided by columns.
  - Custom colors and styles.
  - Reads contents from plain .txt files.
  https://medium.com/@moebiussurfing/wip-text-player-c0b2452f2028

• SerpAPI FOR A YOUTUBE PLAYER
  - Get the assistant answer, and query it as a search to YouTube.
  - Get the URL to the top result: a video or an audio.
  - Preload that content and play a sound in the background or even display a video with audio.
  https://github.com/moebiussurfing/ofxSerpAPI

SOURCE CODE

GitHub - moebiussurfing/chatSURF

AUTHOR

A prototype developement by @moebiusSurfing
( ManuMolina ) 2024.

OVERVIEW

This document serves as a revision of a Prototype App previously coded by myself for LA INVISIBLE Lighting Design Studio in 2021.

I would like to mention the crucial work of Daniel Bravo, the creative director. He started by devising the original idea and elaborated and defined the features that the application should have. In addition to performing all the supervision in the physical place, as well as coordinating with the rest of the human team, as the writer-creator of the contents, the company of the screens, or properly coordinating with the final client.

On my side, all the code programming was developed remotely.

** (Look below at the bottom for all the authoring credits)

This text aims to present the developed tool as a proposal for an upgradeable/customizable module that can be applied to similar circumstances, including:

• Installations featuring a live rendering of styled columns of animated text messages/lines.
• These columns can draw text inputs from a variety of sources, ranging from online queries through APIs like Twitter or scraping to local data like text files or databases.

REFERENCE. THE ORIGINAL PROJECT.

“Bastir de nou el món”

PROJECT GOAL

• We had over a thousand textual poetry sentences that we sought to arrange in a columnar format, with timed slides, carefully calibrated pauses and animated speed, gradual fading and diverse hues, among other features.
• Once everything was in place, we could export the videos as MP4 files or present them live.
• In this instance, five final videos had already been rendered and set to loop.
• This could have been accomplished using Adobe After Effects and rendering the output to video files, but we opted for a more flexible solution.
• Our preference was to have the ability to make real-time modifications and additions to the text, as well as rapidly adjust the columnar layout, speed, slide duration, and so on, to conform to the overall timing requirements. This would eliminate the need for laborious After Effects scripting and multiple renditions.
• To that end, we developed a custom prototype application. This app was a desktop application for Windows and macOS, developed using C++ with openFrameworks, and provided a real-time workflow and high performance for fast and convenient modification of the layout to suit physical spatial dimensions or lighting conditions.”

Key Features of the Prototype Application:

• Columnar arrangement of text poetry sentences.
• Timed slides with customizable pauses and animated speed.
• Gradual fading with options for different hues.
• Live presentation or export to MP4 file format.
• Real-time modifications to text and layout.
• High performance with a real-time workflow.
• Fast and flexible adjustment of settings to conform to overall timing requirements.
• Desktop application for Windows and macOS.
• Developed using C++ with openFrameworks.
• Supports modifications to accommodate physical spatial dimensions and lighting conditions.

WORKFLOW

1. Start loading the 5 normal .txt text files with the poet sentences. (They were short, medium, or longer.) Here you can see how the files look:

Acompanyar
Afectes
Aire
Aerosols
Aforament
Alè
Anticossos
Assistència
Autoconfinament
Balcó
..

Què és? La caça i el Rajoï.
La casa era alhora escola i oficina.
Les meves prioritats han canviat.
On no hi arriba una hi arriba l’altra.
Estic descobrint el meu propi barri.
A la feina, ara som menys de la meitat. 
Les cases plenes i la ciutat sense cotxes.
Continuem oberts per la gent.
Ara ja puc fer les reunions amb pijama.
Cosíem mascaretes sense parar.
Vam aprendre a viure en la incertesa.
..

Ara torno a cuinar els plats que tant us agraden.
Ha estat una oportunitat per conèixer millor els veïns.
Mai havies conegut tant bé els racons de la casa.
Sembla que no acabi, però ens en sortirem.
Aquest Nadal, l’avi podrà venir a dinar?
És estrany, però al final ens adaptem a tot.
Espero que això ens porti una ciutat menys sorollosa.
La vaig haver d’abraçar a través d’una cortina de plàstic.
De cop i volta, les paraules d’ús habitual han canviat, també els hàbits.
Ens vam posar totes a fer mascaretes i pantalles de protecció.
..

2. Do the layout job. The Layout Editor is responsible for creating a visually appealing presentation by customizing the appearance of the columns, including their width, color, font type, and other relevant factors.

3. The Global Timeline is established to manage the overall flow of the presentation, with defined initial and final pauses, slide durations, and animation curves for each projector.

4. Individual settings are configured for up to five separate projectors or screens, providing versatility in presentation options.

5. Finally, the final output is exported as five .mp4 videos, ready for seamless looping projection at the event location.

DESIGNS FROM ARTIST

These are blueprints previously created by the artist, Daniel Bravo, to inspire the creation and requirements of the Prototype App. (Adobe Illustrator and 3D)

• PICTURES IN THE PHYSICAL PLACE

THE PROTOTYPE APP

VIDEO

SCREENSHOTS

• Design Layout: Each of the 5 projectors has his how design. (Amount of columns, widths, and paragraph styles…etc.)

• Timeline: Global duration for each of the 5 projectors. Initial and final pauses, easing animations or fade out to ending black screen.

• Easing / Tween animators: Custom elastic transition when pushing new text sentences or removing older lines can be fine-tuned with the easing animations. (These features allowed imitating modern touch devices feel.)

• Export process: Timeline previews each dialog position, elapsed time in minutes / seconds, debug pauses etc.

• Final frames: Exported frames (without GUI) will look like this. They will be merged to MP4 using internal FFmpeg scripts.

REUSE OR UPGRADE

• Live feature allows “printing” live messages like typically tweets from Twitter from users or hashtags, SMS, Telegram or WhatsApp channels, newspapers feeds, scrapped websites, etc.
  Or maybe as replies to answered questions from touch devices in the place from the local visitors.
• Other alternatives to design presentations can be implemented, of course.
  i.e., we originally thought about adding chat bubbles with text inside like some app chats, imitating that messages were coming from real people during the pandemic:

• The point is that the final result has a “film After Effects rendered quality”, but it can be rendered live, not only playing previously rendered videos. Then text content can be fed from everywhere live / “on the fly”.

DERIVED IDEAS / FUTURE NEW INSTALLATIONS

• Lately, I am thinking about installations for text visualization from big collections of author’s bibliographies.
• Kind of museum showing text content about a literature writer author.
  Something like the visitor making a query search for a sentence, a topic or a keyword.
• Then the installation will search on that big data content and will print the related sentences.
• I made some testing using IA / Deep Learning using Word2Vec.
  That allows search synonyms or identify topics and non-literal searches/responses.
• i.e., searching for “food”, and being replied with all the sentences of the novels of the author where he talks about everything about “eating scenes”.
• Then we can “print” these responses styled and animated.
• Currently, with the GPT (and IA) hype, maybe there’s a lot of stuff that can be done with displaying text content.

PROPOSAL EXAMPLES / IDEAS (TODO):

• 1. Twitter:
  Real-time feed from subscribed hashtags or users.
• 2. News scrapping:
  Get real-time news on a topic using APIs from newspapers.
• 3. Controlled by visitors:
  A menu of options/ABCD/topics that one user can pick/curate for all the installation projections.
• 4. Random text:
  From big databases depending on a mood/event/schedule time in a precise moment in an event: topics/texts are changing.
• 5. Authors exhibitions:
  Showcase with quotes of exposed author/artist/writer/painter/film director/… Visitors can query a keyword or pick from a list of available topics. Then some related text sentences will be played.
  Powered with Deep Learning, Word2Vec or GPT searches.
• 6. Twitter + Web scrapping/API:
  1. Check for the current trending topic every hour.
  2. Search main/top/most read article of some big newspapers about that topic.
  3. Display some picked text sentences/featured paragraphs.

VIDEOS PLAYLIST

** Credits:

An original installation by María Güell, and LA INVISIBLE Lighting Design Studio.

Art Direction:
Daniel Bravo

Text:
Edurard Escofet

Photo:
Rafael Varga

Code:
Manu Molina
(myself, aka moebiusSurfing)

Client:
Ajuntament de Barcelona.

Location:
Barcelona, 2021.

CONTACT

Manu Molina
moebiussurfing@gmail.com

GitHub: github.com/moebiussurfing
Linktree: my profiles and social links.