synwrks

"Painting is a long effort of imitation of what we love" - Renato Guttuso

This quote seems to capture the most instinctive aspect of artistic inspiration, and while not contending in painting, Synesthetic Works does arise from a kind of love for the ideas described in The Imagination Machine Today. Those observations, while very generic, lead to practical or stylistic choices when it comes to producing. I'd like to capture those choices in this article, to be a record, perhaps eventually a tombstone, for the thoughts behind this work. This article will focus on the more direct outcomes of those preferences as manifested in the "method". I will also call out other fundamental components determining the final product: the "subject" and the "sharing modality", without describing those in as much depth, as they tend to be more fluid.

Ideally, you'd read this after going through The Imagination Machine 2121, and then The Imagination Machine Today. Although you don't have to, and might not want to do this to yourself. In fact, ideally you wouldn't read this, as what follows is excruciatingly boring, unless you curiously have genuine interest in the method behind these works. The context from those previous writings can be distilled into a "guiding principle" that goes as follows: explore the individual's power to manifest their imagination with fidelity through technological advancement. The rest of the article will break this into smaller components and describe the deriving decisional razors.

Subject

This is the contents of your imagination, what the art is trying to manifest. Because this project is focused on a method, and the method has fidelity at its core, it would not make sense to maintain a fixed subject. This is because if the artist's fantasies start shifting focus, and the art's subject can't follow that shift, there would be no point in "exploring the individual's power to manifest their imagination with fidelity", as their imagination would be distant from what's being manifested. The art project would have fallen victim of external constraints (be that branding, or audience retention, or reuse etc...).

For that reason, the subject is intended to remain fluid. At the moment of writing, it gravitates around science fiction-inspired themes.

Sharing Modality

This is a combination of the medium of choice and the social setting through which the art is shared, the two additional conduits (after creation) in the transmission of one's thought to the public. The medium can be a screen, speakers, a canvas, a virtual reality headset, text printed on paper etc... A social setting could be a family in a living room, a crowd in a club, a crowd in a gallery, a person alone on their couch etc...

The two components of this dimension can hardly be considered independently from each other: the crowd in a club and a person alone on their couch could both be relying on the speakers as a medium, but the two social settings will impose very different expectations and constraints on the manifestation, perhaps even on the creation.

This aspect of the creative process will evolve together with life, social norms, technology etc... At the moment of writing, this project's work aims to be shared in the following ways:

• original audio and videos on social media platforms,
• original videos animated to music from other inspiring artists, in live mixing performances,
• original, mute looping videos distributed as NFTs,
• showings of original audio/video pieces.

Method

This is the aspect this artistic project is defined by. It is defined by more granular guidelines derived from the guiding principle, which are unlikely to change over time. One distinction to be called out is the following: the guidelines will help consistently answer questions as they are posed by the work, over time however the same guideline can lead to different answers as the context changes. This for example happens when new technology emerges.

Explore the individual's power...

As powerful and enriching as collaboration surely is, this project is focused on "testing" what a single person can do. This is a necessary constraint imposed by the aim for "fidelity" explained further down: If an individual is forced to rely on a large team for the representation of his fantasy, its details will be diluted and distorted by each involved individual's interpretation/opinion/own fantasy.

This specifically applies to the phases of the creative process in which the original depiction is conceived (for example, when the song is composed and recorded, or the video is first rendered). That is because the "sharing modality" will inevitably introduce distortion arising from the very act of transmitting your creation to other people. For example, if people will listen to your song through the radio, or will read your story in a book, the specific speakers' or book's characteristics will introduce distortion that can be mitigated via technical treatment (e.g. mastering, or formatting). This phase inevitably benefits from the collaboration with intermediaries who can represent these recipients' experience (of which the artist might know little about) in the final product. The progressive elimination of this distortion is part of the technological advancement described in The Imagination Machine Today

As the "sharing modality" not only includes the medium/technology, but also the kind of interaction you'd like recipients to have with the art, the individual creator's responsibility can shift in even deeper ways. For example an interactive installation will cede some control to the audience, while a live performance will have to work around constraints introduced by the venue or event. This art project will be somewhat flexible in its "sharing modality", which is seen as a lever that can be pushed in either direction to different social ends, but with a strong bias towards maintaining "fidelity". This is why even as this project relies on digital media, it will produce non-interactive art for the most part: letting the public influence a story would distance it from what the artist originally imagined.

Time Efficiency

One major aspect of empowering an individual to create, is to lower the time budget needed. In The Imagination Machine 2121 this effort is reduced to zero, as creation happens at the same time as fruition, through a seamless and unconstrained flow of information. With our present technology, tools will have been chosen to alleviate time cost as much as possible. An example of this could be the current choice for offline path-tracing rendering techniques over real-time ones: as an autobiographical example, if the artist only focuses on creating during downtime from a full-time job, the extremely long rendering times of path-tracing are an acceptable tradeoff, if they help making the most of the small and sparse moments of creative work.

In this example, it is fine to have to wait for a day to render a short clip, since that day has to be spent on other endeavors anyways. It would not be helpful however to spend the only "creation" hour of that day figuring out how to work around concerns typically introduced by a focus on rendering performance. Real time rendering techniques tend to rely on lower level abstractions: for example the artist might have to know how to mimic reflections because the physics-based calculations are too expensive to be completed in a fraction of a second. They might have to deal with stricter constraints on geometry complexity, or learn different workflows for static versus moving lights etc... What we look for is a high enough abstraction level for the artist to be able to focus on their inner eye more, and the technical details less: "I want this material to be reflective... done.", "I want this surface to be fluorescent, and this lamp to move around... done".

The solution that best balances this tradeoff of course is under constant re-evaluation, as technology evolves at a neck-breaking pace, and options are so abundant it's difficult to be sure whether the "best" tool for the job is ever being used. At the moment of writing for example, doubts run deep regarding what should be used between Unreal Engine 5 and Blender 3.x, or whether TouchDesigner offers enough functionality to replace some of the software prototypes written for the project.

Automation

This is perhaps the most obvious takeaway from The Imagination Machine Today, and the most powerful lever at our disposal for making the creative process time-efficient. After all, the "tools" themselves result from automation.

This project will involve scripting and building new tools whenever the opportunity for higher efficiency is individuated. This can affect the process at any stage. In Synesthesia #1 or #2 for example, most of the animation was derived directly from the music score through Python scripting: this generated most of the geometry and scenes. At the same time the loops used in the visual DJ sets are less procedural in nature (their scripting doesn't depend on variable input except for timing and hard-coded configuration), however layers are blended audio-reactively thanks to software prototypes built to make this process particularly fast in the scenario where you have a lot of music (tens of music tracks) accompanied by very little data (only the waveform).

There is however another main reason why automation plays a big role. For many, the goal of making art is to reach a state of "flow": maximum engagement with the activity is reached by hitting a sweet spot in terms of the challenge posed by the task and amount of skill required. This is perhaps one of the most beautiful states of mind an individual can aspire to stay in, and perhaps one of the reasons we are drawn towards creative crafts instinctively.

The investment in automation will always aim towards making the various steps of the process simple and engaging enough for them to be addressed as bite-sized units of work, while the tedious or repetitive tasks are delegated to the machine as much as possible. At the same time, in cases where the machine could completely replace a task the artist finds to be a source of "flow", automating that step will not be prioritized.

Interface

The concept of flow is a good segway into this aspect of the method: the interface used to access automation power. One can easily see how the easiest way to manifest a thought, would be for that thought to be automatically read from your head and projected onto the medium. Short of that, we'd like to be able to describe it in words, which offers a low level of challenge, but an enormous level of distortion (an image is, after all, worth a thousand words). We could then think of drawing it, which further bases the effort/distortion ratio onto our level of skillfulness (i.e. previous investment) and so forth... It is obvious this aspect must be core to the search of flow, as it aims at balancing the levels of challenge and skill involved in the craft.

There is a tradeoff for all of us, in which just the right amount of effort yields just the right amount of satisfaction with the result, which will vary depending on the kind of result we are trying to achieve and our own skill set. This is a less abstract, more practical representation of the "levels of abstraction" described in The Imagination Machine Today: a very complex implementation could be hidden behind a very simple "interface". Such interface will usually be catered to humans, but maybe more specifically a computer programmer, or a painter, or a sculptor, a guitar player versus a piano player, a writer versus a radio host.

Part of the time spent on Synesthetic Works projects will then be invested in experimenting with various such interfaces, in the search for the best or newest conduits between creator and machine. Some examples of this follow:

• Digital hardware interfaces can be assembled into instruments of any shape and kind capable of producing MIDI, which in turn can be used to produce all sorts of sounds. This means the final result and the input modality can be completely disjoint: the former can be closer to what we imagined, without constraints from the instrument of choice introducing significant distortion. For example, most cornerstone melodies from Synesthesia X videos are first found on digital replicas of older instruments (a MIDI guitar, a small keyboard), while the rest of the composition is usually finalized on launchpad, mouse and keyboard.

• 3D modeling can take an enormous amount of time and know-how. While many assets are best built via polygonal or NURBS modeling, the most elaborated and recognizable models found in this project's videos and loops will very likely be "sculpted". Within this project, sculpting is always done in VR, where software like Adobe Medium allows for a very intuitive experience akin to clay modeling. This is also incredibly fast, which drives the point regarding the interplay between abstractions, automation and efficiency made in The Imagination Machine Today.

  • There are other VR tools that offer a very similar interface to Medium, but through a "painting" metaphor instead. While extremely aligned with most core values of this project, they are unlikely to be employed due to the specific compromise they seek to make, which pushes for more immediacy at too great a cost for fidelity.

Modularization and Reuse

One of the biggest struggles that comes with striving for fidelity, is that it is rare for pure and unconstrained imagination to be using samples and premade assets. While inspiration from the real world abounds, it is unlikely to match the 3D model you'll find on an assets store, or the drums sample you'll find in a samples library. For this reason sampling and citation in the context of this project, while not forbidden, is somewhat rare.

Efforts will however be made to modularize and reuse original work as much as possible. Over time, the reuse of common elements in different contexts and works will, hopefully, contribute to a sense of "soft world building" which is indeed a fruit of the imagination. This practice is clearly visible in works from many digital artists, and it acknowledges the feedback effect present between creation and fantasy: elements that have been imagined will be created, and their manifestation will itself influence future imagination, and so forth.

This is perhaps where automation plays a core role: the facilitation of reuse and remixing. This is for example seen in the process behind the making of the visual DJ sets, where a software prototype has been made to easily assign reusable loops to music tracks: the visuals are stretched and animated to a set of animation envelopes semi-automatically specified for each song. This tradeoff between reuse, automation and authorship is what allows a relatively large music library (what you'd need to populate a DJ set) to be visualized in a way that resembles the fantastic flights one's mind embarks on while listening to a playlist.

...to manifest their imagination with fidelity...

Realism and Simulation

The word "manifest" is chosen intentionally over "express", as expression can be done through language, metaphors, representations that simply refer to what you might have imagined or felt, rather than describing it fully. Redundantly with the term "fidelity", this marks the importance of representing the original vision as closely as possible.

Aside from influencing the language, this influences technical choices as well. One that often comes up is between lower-fidelity techniques for faster rendering, versus expensive algorithms like path-tracing, which yield more realistic results through a process that more closely simulates the physical phenomena of light. At the moment of writing, technology is blurring the lines between these approaches (making the former more realistic, and the latter faster), the choice so far has been falling on the latter (e.g. Blender's Cycles).

Similarly, in the context of music production, physics-based synthesizers will be often sought out to closely resemble the sound of real instruments, with the added flexibility provided by synthesis.

Synthesis

"Imagination" is a fairly broad term, which as described in The Imagination Machine Today can indeed collect a number of phenomena under its umbrella. What this term really does help define is what's not part of this project's method.

As the image or sound that is being captured only resides in one's mind, videography or sound recordings will almost never be employed. Digital synthesis will be front and center in this production. Although this term usually appears in the context of electronic-music, here it is intended as the creation from scratch of sound and images both, and includes 3D rendering.

Experimentation Technique

Given the objectives described above, every "white canvas" will be approached with a clear idea in mind of what the final result should resemble. This means that the role played by experimentation will be in service of achieving the specific result. Most importantly, this will involve testing various routes and techniques in the attempt to obtain a specific effect, or to recognize what is out of reach.

This does not prevent a positive feedback loop from happening (discovering a better color combination by accident, allowing an improved idea to ensue while producing). This however does exclude experimentation as the point of the artistic piece: no creation within this project will start with the intent to try out a new tool, to "see what comes out of it".

In more practical terms, this translates into avoiding certain genres of digital art which, for example, heavily rely on machine learning for the synthesis of outcomes that cannot be predicted. Another example could be the generation of random procedural results, as it could be a random melody generator. This does not mean avoiding ML or procedural techniques in general: if the results of such techniques can be predicted, and they are leveraged in the effort to automate a repetitive step of the process, then their use will be evaluated. In fact, randomness and procedural algorithms are forms of automation at the core of most visual works within the project, for aspects like generating grass, particles, slight variations of a motif, certain sound effects etc...

...through technological advancement.

Technological Nostalgia

Given the focus on leveraging technology, new advancements fitting the tradeoffs described will be welcomed in the process. Choices will not depend on arguments driven by technological nostalgia or the necessity for a practice to be already established in popular culture.

Let's picture the artist's work as being characterized by a certain "capacity" for "actions" that need to be taken, with the goal of producing an "output". You could roughly correlate those elements in the following way: output = capacity/actions. The more capacity an individual has (e.g. time at one's disposal, training) the more output could be produced. At parity of available capacity, the more actions one has to take in order to achieve anything, the less output will be produced. Assuming capacity to be the constrained resource, innovation is seen as the tool for reducing the actions needed.

Perhaps a fitting example could be the DJing technique adopted for the visual mixes. Use of automatic beat syncing, fully digital setups (no use of vinyls), automatic BPM detection and similar tools seems to be frowned upon in the mixing community. This is for good reason: as each of these chip away at the original challenges of the job, they leave little to one's own skill. When framed within the context described in The Imagination Machine Today however, these are nothing else than examples of automating actions away. Hopefully it isn't the case that the capacity of the artist has diminished, meaning output can grow instead: the same one person who before was mixing music, can now worry about mixing both music and video, controlling lighting, and perhaps more.

Dimensions of Perception

The Imagination Machine 2121 describes a scenario in which all senses are fully immersed in the consumption of another's imagination. Most of today's digital art only partially addresses one or two senses. Installations and theatrical experiences of course similarly fully immerse their audiences, but the constraints involved are so huge, their representation of one's original vision is highly distorted and usually conveyed through metaphors ("expressed" rather than "manifested"). As technology evolves however, this gap too can slowly be closed.

The most obvious latest development that comes to the rescue is virtual reality. While it can represent a large additional cost to produce content for it, it does bring within reach the fidelity of digital art and the immersion typical of art installations. This is seen as one of the ways technology can reduce the distortion incurred by thought in its journey towards manifestation: what you imagine is after all never conscripted to a floating rectangle in front of you, and it can often be "a place you go to with your mind". For this reason this project includes investments towards this new medium: "flat" digital art will be often transposed to VR versions of it and vice-versa. As for any novel field, this is an area where rapid software prototyping has been the most useful.

Conclusion

If you made it this far: thank you. This was a long and quite boring read. I hope it shed some light on the motivations behind the work you hopefully enjoyed, and maybe even inspired you for some of your own.

If you didn't make it this far, I hope you are on some other page enjoying the work itself :)

- C