Why not 'merge' all these planners?

solitud · Post by **solitud** » Thu Oct 25, 2012 5:02 am

O.k. that IT joke never gets old, but honestly we should not take the whole topic too serious.

Nevertheless my statements with the experience of two weeks in the rack planning application business:

- BananaPlug is building a new API. Until this is not finished I will not use squiggletronics data at all. I donÃ‚Â´t want to stress me with double effort in building importers for old and new formats, reimporting proprietary identifiers, etc.
- I will not accept squiggletronics as the only master authority for module data. We can only work together on an equal footing.
- I will not stress users to fill in mandatory fields, waiting for their uploaded modules to be approved, redirect them somewhere else.
- Diversity is good. Also is imperfection. ThatÃ‚Â´s the whole idea of a modular synth!
- We have a slightly biased perspective. Most of the users have no idea what we are talking about here in this thread and mistakenly think everything works pretty well

dougcl · Post by **dougcl** » Thu Oct 25, 2012 6:21 am

solitud wrote: - I will not accept squiggletronics as the only master authority for module data. We can only work together on an equal footing.

Okay this is just about identifiers. That's it. Is it clear? You are asking questions now on 5U that have already been solved.

You are unaware of Module Module although it has already been done.

Why reinvent everything?

When you do, you (probably unwittingly) create a competitive system in which winner takes all. Does that make sense?

Is it clear that not sharing identifiers means that the same problems of identification happen in every database?

Is it clear that not sharing identifiers means that it will be very difficult to effectively support and use distributed data (many databases) in the future?

Is it clear that your arbitrary choice to use a new set of identifiers is more work for you in the short and long term than using the ones that already exist?

solitud wrote: - I will not stress users to fill in mandatory fields, waiting for their uploaded modules to be approved, redirect them somewhere else.

This is not a real use case.

solitud wrote: - Diversity is good. Also is imperfection. ThatÃ‚Â´s the whole idea of a modular synth!

Right, that's why there needs to be a minimal agreement so that the pieces can interoperate. Same as modular.

solitud wrote: - We have a slightly biased perspective. Most of the users have no idea what we are talking about here in this thread and mistakenly think everything works pretty well

Keep in mind that a user started this thread. Further, the users will become all too aware of the situation if you tire of moderating a shared library.

BananaPlug · Post by **BananaPlug** » Thu Oct 25, 2012 9:19 am

solitude:

BananaPlug is building a new API. Until this is not finished I will not use squiggletronics data at all. I donÃ‚Â´t want to stress me with double effort in building importers for old and new formats, reimporting proprietary identifiers, etc.

That could be misleading.
The update adds support for the Identifiers which were added to the XML standard. Anyone currently importing info from the library is parsing the XML from the zip files and the library is simply keeping up with the XML specification. Adding the identifiers is not something which should "stress" anybody. I am adding an API to make it easier for a program to search for specific modules and retrieve the information directly without having to open up zips and parse XML. An application will be able to use the API to selectively download recently updated modules, etc. As librarian of a collection of data donated by the community of users I'm trying to cooperate with everybody to the benefit of those users.

How Identifiers Fit In:
The library uses the fields below for classification.

Manuf: 4MS
Model: Noise Swash
Format: E
Modifier: <null>

Deriving an id from that initial text can be problematic. Sometimes its necessary to go back and revise the text to correct errors in the original data or conform with evolving preferences. FYI, the library does not allow more than one module to have the same exact set of data for those fields. The Modifier field is used to differentiate between alternative versions.

One of the new Identifiers will be assigned when the module is created (before moderation) so when you download your zip from ModuleMaker it will have that id in it already. Assuming it goes into the library that identifier will stay with it, surviving any edits. If a batch of modules from a third party is added to the library those modules may already have an Identifier from that system. The library will add it's own Identifier. With support for multiple Identifiers each module is cross referenced to the other participating systems.

I have some ideas about how everyone could benefit from multiple planners cooperating with an agnostic library but I'm getting ahead of myself. We'll leave that for later when the API is ready.

solitud · Post by **solitud** » Thu Oct 25, 2012 12:09 pm

dougcl wrote:Is it clear

...
I think this sounds rude in any language.

solitud · Post by **solitud** » Thu Oct 25, 2012 12:31 pm

@BananaPlug: thanks for clearing this up but I ... still donÃ‚Â´t get it exactly. I even putted the text into Google Translate with some funny results.

I already have build an Importer which recursivly unzipps and parses the collection zip files. If I import this data into my database am I able to update the data from Squiggltronics using the Manuf/Model/Format/Modifier scheme via the future API?
Currently there is no property "id", "modified" whatever in this files!
Meaning: will you rebuild all the collection zip files with this new id?
Or are the collection zips deprecated in future?

BananaPlug · Post by **BananaPlug** » Thu Oct 25, 2012 1:19 pm

If I import this data into my database am I able to update the data from Squiggltronics using the Manuf/Model/Format/Modifier scheme via the future API?

The short answer is yes. However, it's possible that a module's Manuf/Model/Format/Modifier changes. Usually that's because of spelling corrections or fixing up confusing similar versions. You should be able to get most of them without any difficulty.

will you rebuild all the collection zip files with this new id? Or are the collection zips deprecated in future?

The zip files on the site (The Collections page) are made from the database info, can be refreshed automatically and all will be remade when I install the updates. Currently the individual zip files are always up to date with any changes made in the database but the collections are rebuilt less frequently. When I add some new modules to the database the collections those modules belong in all get refreshed.

The next update will have the id and modtime (in UTC). This is an example from my test system.

edit: Refer to the schema for XML structure. This example is incorrect.

Code: Select all

<?xml version="1.0" encoding="UTF-8"?>
<module>
	<properties>
		<manuf>TestCo</manuf>
		<format>E</format>
		<model>Zoomatron</model>
		<modifier>Rev2</modifier>
		<modtime>2012-10-23 02&#58;36&#58;32</modtime>
		<depth_mm>35</depth_mm>
		<HP>20</HP>
		<mA>40</mA>
		<moduleImageFilename>zhizzy.jpg</moduleImageFilename>
		<priceUSD>775</priceUSD>
		<V>12</V>
		<identifiers>
			<authority>squiggletronics</authority>
			<ID>Z123456789</ID>
		</identifiers>
	</properties>
</module>
<link>
	<title>Muffs Wiki Page</title>
	<url>http&#58;//wiki.muffwiggler.com</url>
</link>

edit: The contents of ID (Z123456789 above) will be a little different.
edit: It's timestamp not modtime.

solitud · Post by **solitud** » Thu Oct 25, 2012 1:42 pm

An xml tells more than 1000 words, thank you!
I recognized that you changed the charset to utf-8, great, this was one point on my list I didnÃ‚Â´t dare to ask anymore!
In Dougs XML there is (was?) a property "timestamp". Is this "modtime" now?

BananaPlug · Post by **BananaPlug** » Thu Oct 25, 2012 4:36 pm

I'm interested in opinions about using UTF-8. What I want to do is allow full UTF-8. When making the zip filename from the various data fields they will be convert to "safe" characters. In the updates underway now my tools will be UTF-8 capable but ModuleMaker will be restricting the text for Manuf/Model/Format/Modifier to simple characters until we've had a chance to discuss.

You are right about the timestamp. My mistake. The ModuleLibrary uses more than one timestamp internally and when XML is made the most recent one will be put in the XML. Updating the image will affect the timestamp.

solitud · Post by **solitud** » Thu Oct 25, 2012 5:44 pm

UTF-8 is sometimes harder to achieve than it should be. In my experience the whole chain of data has to be utf-8, from database, tables to html, else things go bananas. Except filesystem of course. I donÃ‚Â´t know what solution you come up with the identifier thing. Module names and manufactures can be UTF-8, think about μstep. I also can imagine that Mr Doepfers real name is DÃƒÂ¶pfer.
But all those special chars are not usable as part of an identifier I think.

P.S. I have imported squiggletronics dotcom collection to the modular grid!

John Noble · Post by **John Noble** » Thu Oct 25, 2012 6:14 pm

dougcl wrote:You are asking questions now on 5U that have already been solved.

Doug, this is completely out of place coming from you since you've been mandating data standards unilaterally for things you don't understand, namely Euro power specifications.

Knut deserves nothing but applause and our assistance--if he asks for it. He's doing a fabulous job on his own. Keep in mind that my favorite planner is actually Dev Gupta's http://eurorackplanner.com project, so it's not some kind of tribalist "us vs. them" thing.

Can we stick to technical stuff, like the proposed XML update I posted to the schema thread?

dougcl · Post by **dougcl** » Thu Oct 25, 2012 6:46 pm

John, clearly some frustration here all around. Please note that while this thread is under the RackPlanner subforum, it's really about databases and database developers collaborating.

It has nothing to do with RackPlanner, or any planner for that matter. We could just as easily be talking about interactive searches on your web page.

The vision, and I think everyone is on the same page here, is not to have a single system dictating everything to everyone. It's not to merge data.

The vision is to allow many databases, each with strengths in unique areas, to provide meaningful use together.

Assigning one database the role of moderating module identities is going to help everyone, and should not be considered a threat or power grab in any way. Think of it merely as the unique strength of squiggletronics. It's what squiggletronics can bring to the table.

John Noble · Post by **John Noble** » Thu Oct 25, 2012 8:51 pm

dougcl wrote:Assigning one database the role of moderating module identities is going to help everyone, and should not be considered a threat or power grab in any way. Think of it merely as the unique strength of squiggletronics. It's what squiggletronics can bring to the table.

With the greatest respect for Bananaplug and everything he does, The Eurorack Database is more complete and much better scrubbed. I'm not guessing as I have reviewed both. I don't list custom/kit/DIY modules for reasons to lengthy to express here, but the nature of those modules is such that no one could reliably assign a universal ID anyway. I don't even list the short run modules *I* have sold.

If someone wants to use my data or have my site act as a conduit for their data, I can provide facilities for them to do so as I have for others. I can give write authority via an API to the Squiggletronics admin to populate and maintain the Squiggletronics ID if someone wants to do the work to match the data, is willing to maintain it regularly, and can point to a reasonably large group of end users that will get a concrete benefit from it. What I won't do is rely on anyone else for clearance to enter new modules and publish the data.

(I have another ~10 modules to enter tonight, headed to 765, I think? Where does it all end?)

BananaPlug · Post by **BananaPlug** » Thu Oct 25, 2012 9:44 pm

The library will be useful to different planners in different ways and I'm open to discussing how that happens. Right now I have work to do laying the ground work for that and when I'm a little further along I'll toss out some ideas.

synthcube · Post by **synthcube** » Thu Oct 25, 2012 9:49 pm

Meanwhile, there are good efforts underway to develop planning tools, but that are replicating very basic efforts like uploading motm module jpgs. Seriously? This landscape is so complex that we can't collaborate on a single standard against which all sorts of user front ends might be developed? Do I, as an motm user, need to submit multiple versions of my panel jpgs in order to have a choice of the best latest planner functionality? Why should solitud, who has developed an awesome euro format front end, have to start from scratch on learning what 5u means?

solitud · Post by **solitud** » Fri Oct 26, 2012 4:16 am

synthcube wrote:Meanwhile, there are good efforts underway to develop planning tools, but that are replicating very basic efforts like uploading motm module jpgs. Seriously? This landscape is so complex that we can't collaborate on a single standard against which all sorts of user front ends might be developed? Do I, as an motm user, need to submit multiple versions of my panel jpgs in order to have a choice of the best latest planner functionality? Why should solitud, who has developed an awesome euro format front end, have to start from scratch on learning what 5u means?

The missing MOTMs are not a problem of lacking cooperation. ItÃ‚Â´s simply a technical issue of the modulargrid. As I learned the hard way aspect ratios differ between MU an MOTM. My Image resize engine is not capable to mix this things up.

As it is now I have to introduce a new module universe for the MOTM stuff.
But you see, I am really lacking some knowledge in this area and I think many other people do, too.

Some say there is Dotcom format and there is 5U.
Some say itÃ‚Â´s all 5U regarding the height.
Some say Dotcom is a subcategory of MU.

I also must be careful to ask this simple questions, because people will seriously doubt in my competence in this sector and may react aggressiv.

But as you said, I started already to import existing data and the quantity of content will soon be there.

After that we have to improve quality.

solitud · Post by **solitud** » Fri Oct 26, 2012 5:57 am

P.S.: MOTM is now available in a new universe called "5U".
I also placed a link for reference to Squiggletronic.

BananaPlug · Post by **BananaPlug** » Fri Oct 26, 2012 9:31 am

FYI - The Module Library module zips and collections have all been rebuilt with a "timestamp" property in the XML. This was a fix, it was empty before.
It indicates the most recent modification to any part of the information the library has about the module. It's timezone is UTC.

Code: Select all

<?xml version="1.0" encoding="UTF-8"?>
<module>
	<properties>
		<manuf>Modcan</manuf>
		<format>A</format>
		<model>4075</model>
		<modifier></modifier>
		<HP>1</HP>
		<moduleImageFilename>4075.jpg</moduleImageFilename>
		<timestamp>2012-07-01 17&#58;13&#58;56</timestamp>
	</properties>
</module>

The "identifier" information will be added in a few days when other updates are ready.

studio.h · Post by **studio.h** » Tue Jul 11, 2017 4:46 pm

Reason is an affront to power.

MOD WIGGLER

MOD WIGGLER

Why not 'merge' all these planners?