HN user

kozlovsky

783 karma

I'm the author of Pony ORM

http://ponyorm.com

My email starts with alexander.kozlovsky and ends with gmail.com

Posts35
Comments38
View on HN
writings.stephenwolfram.com 1y ago

What If We Had Bigger Brains? Imagining Minds Beyond Ours

kozlovsky
1pts1
www.medscape.com 6y ago

Do Covid-19 Vent Protocols Need a Second Look?

kozlovsky
2pts0
coronavirus.medium.com 6y ago

Untangling Covid-19, Altitude Sickness, and Conspiracy Theory

kozlovsky
1pts0
github.com 9y ago

PythonQL Intro and Tutorial

kozlovsky
4pts0
sw-or-gtfo.tumblr.com 9y ago

Social work or GTFO

kozlovsky
3pts0
www.nytimes.com 10y ago

Even with Confession of Cheating, World’s Doping Watchdog Did Nothing

kozlovsky
1pts0
www.theverge.com 10y ago

Facebook is building artificial intelligence to finally beat humans at Go

kozlovsky
2pts0
blog.jetbrains.com 10y ago

PyCharm 5 released

kozlovsky
8pts2
home.web.cern.ch 11y ago

The LHCb experiment at CERN discovers a new type of particle made up of 5 quarks

kozlovsky
7pts0
firstdoit.com 11y ago

How I learned to stop worrying and love React

kozlovsky
141pts106
wafflesatnoon.com 11y ago

Mysterious Blobs Rain Down Over Washington in 1994

kozlovsky
1pts0
shop.oreilly.com 11y ago

New Book of Kathy Sierra: Badass: Making Users Awesome

kozlovsky
1pts0
basecamp.com 11y ago

The story of Basecamp

kozlovsky
3pts0
medium.com 11y ago

Is Scurvy the New Diabetes

kozlovsky
2pts0
www.knockmeout.net 11y ago

Knockout.js 3.3 Released

kozlovsky
2pts0
lucumr.pocoo.org 11y ago

Why I Don't Hate Git: Hidden Consistency

kozlovsky
157pts48
codecapsule.com 11y ago

What every programmer should know about solid-state drives

kozlovsky
135pts31
www.quirksmode.org 11y ago

The difference between front-enders and back-enders

kozlovsky
13pts0
pythonfasterway.uni.me 11y ago

Python: Faster Way

kozlovsky
144pts64
medium.com 11y ago

Michael Levin wants to help people regrow lost limbs (2012)

kozlovsky
15pts0
www.youtube.com 12y ago

EuroPython 2014: How Pony ORM translates Python generators to SQL queries

kozlovsky
1pts0
jakeaustwick.me 12y ago

Give PonyORM a chance

kozlovsky
51pts63
www.nytimes.com 12y ago

Always Hungry? Here’s Why

kozlovsky
153pts131
mathwithbaddrawings.com 12y ago

Math Experts Split the Check

kozlovsky
4pts0
dreamindustriesunderattack.tumblr.com 13y ago

Russian startup Dream Industries is raided on by its investors

kozlovsky
2pts0
softwaremaniacs.org 13y ago

Programmer types: Scientists, Engineers and Teachers

kozlovsky
2pts0
www.sciencespacerobots.com 13y ago

Humanoid robot is being trained to work on the International Space Station

kozlovsky
1pts0
pragdave.blogs.pragprog.com 14y ago

A subtle potential bug in most Rails applications

kozlovsky
2pts0
www.youtube.com 14y ago

Elections in Russia denies the basic math

kozlovsky
3pts1
www.youtube.com 14y ago

Elections in Russia denies the basic laws of math

kozlovsky
4pts1

The exact meaning behind these arm signs remains unclear. The researchers observed them in various contexts – during mating, hunting, defensive situations, and sometimes spontaneously. This suggests the signs might serve multiple purposes depending on the situation.

Can these signs be an alphabet of some language?

If we show a neural network some examples from the Game of Life and expect it to master the rules of a cellular automaton, then aren't we asking too much from it? In some ways, this is analogous to expecting that if we show the neural network examples from the physical world, it will automatically derive Newton's three laws. Not every person observing the world around him can independently deduce Newton's laws from scratch, no matter how many examples he sees.

We already encountered a triangle with three right angles on the sphere

Nitpick, but the upper angle of triangle displayed is not right, it's 72 degree (1/5 of 360)

Hint is a text field that you fill in when creating a new cloud password. The hint text is generated based on password if you did not fill it yourself.

However, all template languages are inherently crippled: they can never achieve the same expressiveness and power as code. Quite simply, {{# each}}, ng-repeat and databind="foreach" are all poor replacements for something that is native and trivial in JavaScript: a for loop.

On the other hand, when using a template language I can put `foreach` loops and `if` conditions right into the template itself. And when using JSX I need to calculate the result of a `for` loop before the actual template and it looks much more complex and cumbersome.

Yesterday my Intel SSD 320 failed horrible and kills all data on my RAID system. I still do not understand how it happened. I was sure my storage system is very safe.

My system was configured as a mirror RAID with two 1TB HDDs. The RAID was created with Intel Rapid Storage technology (my motherboard is Asus p8z77v-pro). The failed SSD was used as a RAID cache configured with Intel Smart Response technology.

I was wary to use SSD directly as a main system drive, because I heard of "BAD_CTX 13F" error which happened with Intel 320 drives. My hope was that if SSD is used as just a cache, then in the case of SSD failure the data still be safe. Since this error reportedly occurs during power outage only, I set up an UPS. But all these precautions have not helped.

Yesterday I surfed web with Google Chrome, and my computer suddenly become unresponsive. At first only Chrome was unusually slow, and other open programs work normally, but in a few minutes the computer was totally freeze. I was forced to press reset, and upon restart Windows automatically entered into non-interruptible "recovery mode". After more then 24 hours the OS reported that "further recovery is impossible" and the RAID become unbootable. The SSD serial number was changed to "BAD_CTX 0000013F", the sign of famous "8mb bug". It is interesting that in my case this bug was not caused by any power outage except when I pressed "reset" button, but I don't think this is count as a power loss.

I take an HDD out of the RAID to connect it to other computer and save critical data, but without any success. At first sight all file system looks correct, and I even manage to copy all recent data files, but when I looked into those files it was total mess. Each file consist of some arbitrary chunks of unrelated files, mixed in random order - a bit of some executable file, several lines of my project source code, followed by chunk of some unknown xml configuration file, followed by random bytes, etc. A total mess.

I still don't understand the reason of such spectacular data corruption. I have three hypotheses: 1. SSD cache sent incorrect data to RAID on write (two month ago I switched SSD cache from "enhanced" to "maximized" mode, in which writes initially goes to SSD and only then to the RAID disks). 2. Intel RAID controller goes crazy due to a program error. 3. Windows corrupt data during non-interruptible "recovery" phase.

The moral is, even Intel SSD with UPS is not safe, and mirror RAID cannot not protect data from such errors.

The basic scenario where lost update is possible looks this way:

    session_1: BEGIN
    session_1: SELECT data from T1 where pk = 123

    session_2: BEGIN
    session_2: SELECT data from T1 where pk = 123
    session_2: UPDATE T1 set data = "value_2" where pk = 123
    session_2: COMMIT

    session_1: UPDATE T1 set data = "updated_value" where pk = 123
    session_1: COMMIT
In this scenario, session_1 does not see "value_2", so "updated_value" does not take "value2" into account, and update of session_2 becomes lost. There are two ways to prevent this problem:

1) Using SELECT FOR UPDATE instead of SELECT. This way session_1 take exclusive lock on the corresponding row, and session_2 execution is suspended until session_1 will commit.

2) Using more strict SERIALIZABLE isolation level. In this mode, session_1 in MySQL will take shared lock on each selected row, and each SELECT effectively becomes SELECT ... LOCK IN SHARE MODE. This way session_2 also blocks until session_1 will commit.

In PostgreSQL, SERIALIZED mode implemented the other way - SELECT works as usual (without locks), but during UPDATE session_1 will detect that the row was changed by concurrent transaction, and transaction of session_1 will be terminated with the exception "could not serialize access due to concurrent update".

MVCC in InnoDB and PostgreSQL are totally different. It is PostgreSQL and Oracle MVCC who have very similar, although not identical isolation behaviour (albeit very different implementation), but InnoDB MVCC stands apart from them.

At the beginning, I should point that the default isolation level in PostgreSQL and Oracle is READ COMMITTED, whereas MySQL uses REPEATABLE READ isolation level as a default. This means that in MySQL by default all normal (not FOR UPDATE) SELECT queries see database snapshot at the time when transaction started. In PostgreSQL and Oracle all SELECT queries by default see database snapshot at the time when the query (and not the transaction) started.

However, SELECT FOR UPDATE in MySQL sees database at the moment the query started, even with REPEATABLE READ isolation level. That is, it is possible that during the same transaction on default isolation level in MySQL SELECT and SELECT FOR UPDATE see totally different rows. In Oracle and PostgreSQL, SELECT and SELECT FOR UPDATE always see the same rows (at the moment of query start in READ COMMITTED mode, or at the moment of transaction start when in SERIALIZABLE mode). In other words, in Oracle and PostgreSQL SELECTs and UPDATEs see database at the same time in past, whereas in MySQL SELECTs see consistent snapshot at past, whereas SELECT FOR UPDATE and UPDATE see current data.

Also, as MySQL uses locks more aggressively, some "write skew" problems which are possible in PostgreSQL and Oracle, are not possible in MySQL.

I don't know what kind of MVCC is "better", but IMHO Oracle and PostgreSQL MVCC is much easier to understand than MySQL one. The difference between Oracle and PostgreSQL MVCC is the following: in READ COMMITTED mode, when long UPDATE query encounter some rows updated by concurrent transaction, it is blocked until concurrent transaction COMMIT or ROLLBACK. After ROLLBACK of concurrent transaction the query will continues execution, and this behavior is the same in Oracle and PostgreSQL. If concurrent transaction is committed, then PostgreSQL re-evaluates previously blocked row to determine whether it still matches the UPDATE criteria. But PostgreSQL does not re-check previously read rows which did not satisfy UPDATE criteria, whereas is is possible that these rows were also modified by concurrent transaction and now satisfy UPDATE criteria and should be updated. Oracle behaves in a different way: If long UPDATE query encounters some rows updated by concurrent transaction, and these rows no longer satisfy the UPDATE criteria, the query restarts and sees new database snapshot at the current time (remainder - I'm speaking about the default READ COMMITTED mode). This way Oracle doesn't miss rows which can be missed by PostgreSQL, but long UPDATE can potentially be restarted many times (that is, the query starvation is possible, although not likely). http://asktom.oracle.com/pls/asktom/f?p=100:11%3a%3a%3a%3a%3...

Actually, it is not always necessary to write a lot of boilerplate code when using a SQL database. For example, with Pony ORM the same SQL query could be written as following:

    select(p.name for p in pokemon
           if "Hydro Pump" in p.learn_moves.move.en_name
           ).order_by("p.sp_atk").first()
New Delicious 13 years ago

I cannot understand why Delicious team doesn't implemented collaborative filtering [1]. I think this is the main benefit that the user can get from a social bookmarking site - the ability to see the "bookmarks of others that are similar to your bookmarks". The implementation is not too complex and nicely covered in the book [2] with examples in Python. The book even has a special section "Building a del.icio.us Link Recommender".

If I'm not mistaken, there were prototypes for doing collaborative filtering of Delicious dataset written by standalone applications. But doing such filtering requires direct access to the database and cannot be done efficiently via HTTP. So, after the current team had bought Delicious from Yahoo, I thought that implementing collaborative filtering would be their main priority. Instead of that, they concentrate forces on worsening and cluttering UI experience along with breaking various Delicious bookmarking plugins for popular browsers.

[1] http://en.wikipedia.org/wiki/Collaborative_filtering [2] http://shop.oreilly.com/product/9780596529321.do

No, it is the other way, Python is translated to SQL, not vice versa ;)

According to SQL standard, aggregate functions must skip NULL values. The problem is with queries which return no rows. In this case, intuitive expected result of SUM aggregate function is zero, but, because of some quirk in SQL standard, SUM must return NULL if no rows were found.

Without coalesce function, result of SUM will be returned as None if no rows were found. Coalesce replaces this surprising None with zero.

I have not used Peewee, but at first sight the most visible difference is the query syntax - Peewee uses method chaining, whereas Pony ORM decompiles Python AST. I think that many queries can look simpler in Pony than in Pewee.

For example, if I want to retrieve all pairs of different users with the same name, in Pony this query would look as follow:

    select((u1, u2) for u1 in User for u2 in User if u1.name == u2.name and u1 != u2)
In Peewee, this query probably would look something like this (didn't test it):
    User2 = User.alias()
    User.select(User, User2).join(User2).where((User.name == User2.name) & (User.id != User2.id))
It’s probable the matter of taste, but I like Pony syntax better. Another example is taken from the Peewee doc:
    staff_users = User.select().where(is_staff=True)
    Tweet.select().where(~(Tweet.user << staff_users))
In Pony, this query would look as follow:
    select(t for t in Tweet if t.user not in (u for u in User if u.is_staff))
    
Also, this simple version is possible, don't know why it is not as simple in Peewee
    select(t for t in tweet if not t.user.is_staff)
Also, I don’t know if Peewee supports IdentityMap and optimistics transactions.

On there other side, it seems Peewee already has migration support, whereas in Pony migrations are not implemented yet, this is the next task.

One of the goals of Pony ORM is to reduce the necessity of using the specific syntax. For example, in Pony ORM you can write:

    (o for o in Order if o.date_shipped.year == 2012)
instead of something like:
    .filter(extract("year", Order.date_shipped) == 2012)
The goal is to use native Python syntax wherever possible

SQL Alchemy is fundamentally based on a relational model and translates queries to SQL. Pony ORM is based on an entity-relationship model, which is logically independent from SQL. Currently Pony translates queries to SQL only, but in the future noSQL support is also planned.

Hi, I answered about “pure Python” obsession in a separate comment ;)

> you're still littering Python with SQL keywords like order_by(), desc(), select(), and commit()

Well, I see them not as “SQL keywords” but as logical concepts

> The GROUP BY section of the documentation just reads like a rough translation of SQL into Python

Hmm, not sure I understand this. With PonyORM, GROUP BY section is generated automatically, and not present in Pony ORM queries. This eliminates the need to duplicate the same columns in SELECT and GROUP BY sections.

> Beyond a certain level of complexity, you need to know SQL in order to write complex queries.

I agree with this. But I hope PonyORM can lower entry barrier for new developers. Also, PonyORM allows developers to use pure SQL, if all they need from mapper is IdentityMap, caching, etc.

> By the way, why is there an order_by() method and a separate orderby() method?

Thanks for noticing this, it is a typo in documentation. The method was renamed from orderby() to order_by() recently.

Hi, I'm Alexander Kozlovsky, one of Pony ORM authors. I'll try to answer some questions

> why such an obsession with "pure" Python?

With this statement we are trying to say Pony is not just to offer some syntax sugar. The end goal of Pony ORM is to understand the semantic of each Python generator which can be translated to database query and then do translation accordingly.

The direct benefit of this is that generator query are more high-level then resulted SQL, and can be much easier to understand and refactor. The simplest example is an automatic converting of attribute path traversing such as grade.student.group.department.name to a minimal set of SQL joins. Pony also can use attribute path in reverse direction, from “one” to “many”, such as department.groups.students.gpa, which in this case resulted in a multiset of all ‘gpa’ values for this department. Also you can see examples of query optimization in the documentation, for example when the subquery is converted to LEFT JOIN with GROUP BY, where it can give performance benefit.

There are more distant benefits of the semantic translation. In future, we plan to add support of denormalized database schemas. Such denormalization will take place on a physical level (i.e. in the database) while on the logical level (in Python) all queries and logic will remain the same. But first of all we are going to add the migration support.

Another future plan is to add support of MongoDB. This is a distant plan, but I think Pony architecture will allow this. After that, migration of PonyORM-based project from relational databases to noSQL and vice versa will be possible.

> An ORM like Django's offers a higher level of abstraction

I disagree with this. I think, any Django query can be written in PonyORM much more concisely, and there are easy-to-understand PonyORM queries which cannot be expressed in Django easily.

Also, Pony ORM completely eliminates N+1 select problem.