16

I imagine a lot of us on this site are fans of Mathematica's design principles (for both the language and function library). When a new version is released it is always exciting for me to see what new features have been polished enough to become part of the standard library. At the same time, I don't feel that software sophistication as a whole is improving as fast as it should (when compared to say the increase in size of datacenters). I think we are the right community to start building an open source, consistent set of extended packages for the standard library functions. We could contribute in a similar manner that we contribute here on SE (short bursts of work on a wide variety of problem domains), but this would also incorporate integration and organization work (building meta-algorithms, adding tests, etc). I had some initial discussions last week on the Wikidata mailing list, but there wasn't a critical mass of programmers that are used to working in short bursts on a wide variety of problems. I think that in exchange for our extra work compared to just answering SE questions, we would help build a resource that all of us would use to accelerate and expand our coding projects in ways that we haven't even imagined.

The Wikidata discussion is called "Accelerating software innovation with Wikidata and improved Wikicode": http://lists.wikimedia.org/pipermail/wikidata-l/2013-July/thread.html

I have a lot more I could say about this, including specifics on how to start, but I wanted to gather some more feedback after people have glanced at the Wikidata discussion.

  • I have been working on a set of tools that are supposed to vastly simplify publishing one's code, keeping it under version control, and using code of others in a robust way. Some parts of these tools are ready on the prototype level, but overall this is not ready yet for even asking others to test. But this is currently my main side project, and I expect to get it to a decent shape in the next couple of months. – Leonid Shifrin Jul 18 '13 at 22:39
  • That sounds fantastic. I was thinking of just starting a GitHub project that we can upload package files and tests into for now and then we can deal with scale issues as they arise. – Michael Hale Jul 19 '13 at 1:43
  • What I currently have is a Mathematica GitHub client which allows to create GitHub gists from Mathematica, as well as load them and do basic versioning. Plus, a project format, basically a project is a number of files, plus a project file written in a certain way. I've made a number of gists this way, already using this system. You can have a look here (which, being a gist, incidentally is a part of the system itself). – Leonid Shifrin Jul 19 '13 at 2:01
  • ...wrote this in OO-style, since this stuff seems to be best done this way. So far, the system relies on curl for http requests, but I plan eventually to start using pure Mathematica functions, since now it is possible. In fact, most of these components I managed with the system itself (I mean, committed newer versions). Still, the system is a little raw, and does not yet have a UI. If you are interested in trying it, I will write an installer for it in the next couple of days, and some instructions. I will also write some rad map of what I consider currently missing. – Leonid Shifrin Jul 19 '13 at 2:13
  • This is probably different from what you had in mind though. The idea is that one can post projects from within Mathematica. The other part, namely the project loader that would resolve project dependencies and take into account versioning, is still missing. It would actually require another layer on top of the current gist-based VCS module, as well. Plus the UI for the project management is halfway done only. The system is mostly decentralized, while we may have a single cetral repo / wiki with links to these gists. I am off for today, can continue in chat tomorrow or later. – Leonid Shifrin Jul 19 '13 at 2:23
  • I've skimmed through most of your code. I haven't used Workbench and I was wondering how similar your project file format is to it. Your OO extensions could certainly be useful in various areas. I typically just inline something like Fold[#2 /. #1 &, ruleTree, {"property1", "property2", "property3"}] for parsing results I get from JSON imports, but if you have to do a lot of it I could see myself using that package. I think if lots of us start uploading to separate Gists that we should still try to associate things with Wikipedia articles so that particular functionality has a specific... – Michael Hale Jul 19 '13 at 3:53
  • ... place it should belong. Would I look for "nested rules" or "JSON import parsing" for that functionality? With this approach I would go to JSON.m and it would have functionality for parsing because JSON has a Wikipedia article but rule trees does not. – Michael Hale Jul 19 '13 at 3:54
  • This sounds like a nice plan, though you might elaborate a bit. What do you mean precisely by a "consistent set of extended packages for the standard library functions". This is a rather broad. Do you have anything specific in mind? – Sjoerd C. de Vries Jul 19 '13 at 10:37
  • I think we should make the highest level of encapsulation and modularity for a set of community curated packages correspond to Wikipedia articles. This way they are easy to find because they all correspond to things that are considered broadly important and match how people think about things. It would just determine a scaffold for refactoring and organizing our most popular and useful code. We can integrate code from our own projects, good answers here, or browse computer science articles by popularity until we find the most popular algorithms that aren't included in the standard library. – Michael Hale Jul 19 '13 at 10:46
  • Then over time we will be able to build up to higher level functionality than we could on our own. Simulate[{Ball[],Stairs[]},5] In your head you can imagine a default output that could generate, but the amount of logic and details to specify sensible defaults to such terse, high-level functions and object descriptions requires a substantial amount of backing code that needs to be well organized and consistently designed. – Michael Hale Jul 19 '13 at 10:49
  • Think about all of the assumptions we make when we read that simple line of potential code. We assume the ball is about the size of a soccer ball. We assume the initial positions of the ball and stairs don't intersect and that the ball is above the stairs. We assume we'd like to see the behavior of the system for 5 seconds. So it's sort of like we are trying to apply the design principles of the NDSolve meta-algorithm to lots of things. Or like we're making an open source Wolfram Alpha, but we aren't being slowed down by worrying about natural language processing. – Michael Hale Jul 19 '13 at 11:15
  • If I may suggest something, consider using the boost project as a template, instead of a wiki-model. – rcollyer Jul 19 '13 at 13:53
  • @rcollyer I was thinking of modeling the code repository under the node js community package portal. But I also think that we should first have technical means of making publishing, storing / hosting, using and forking the code convenient. This is what my tools are aimed at. I am also not sure that Wikipedia is the best format, but no one prevents us to use it as one of the formats. – Leonid Shifrin Jul 19 '13 at 15:11
  • 1
    I have created a chat room for further discussion of these issues. – Leonid Shifrin Jul 19 '13 at 15:15
2

How about we start with a wiki approach and then retreat by adding constraints as we run into problems? I'm hesitant to start by using Gists because I think collaboration by using forked edits will be harder than just looking at the edit history of a wiki page.

I think the first module I will work on will be for the solar constant. The article is the first result in the search engines and Wolfram Alpha just gives you the standard 1.361 kW/m/m. But what if you want to get the atmospheric solar irradiance adjusted for a specific time of year to account for the elliptical shape of the Earth's orbit? Or what if you want to get the irradiance from a specific radiation frequency interval? I will start to add that functionality later tonight.

All are welcome to start contributing! Every wiki page has an associated talk page to hold design discussions for specific modules. Place tests on a subpage of each module called "tests". I also have a section for putting utilities to import modules (put your code in a <code><nowiki>...</nowiki></code> block), run tests, etc. We can continue discussing on here ideas for moving the project to a more constrained hosting environment of course.

http://en.wikipedia.org/wiki/User:Wakebrdkid/Wikicode

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .