issue_comments

15 rows where issue = 325958506 sorted by updated_at descending

View and edit SQL

Suggested facets: created_at (date), updated_at (date)

user

issue

  • Support cross-database joins · 15

author_association

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue
552140975 https://github.com/simonw/datasette/issues/283#issuecomment-552140975 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDU1MjE0MDk3NQ== simonw 9599 2019-11-09T21:51:41Z 2019-11-09T21:51:41Z OWNER

It may turn out that we have to recommend NOT exposing a Datasette instance to the public with dozens of database files that has multi-db queries enabled - will need to load test to understand if this recommendation is needed or not.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506
552140870 https://github.com/simonw/datasette/issues/283#issuecomment-552140870 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDU1MjE0MDg3MA== simonw 9599 2019-11-09T21:49:51Z 2019-11-09T21:49:51Z OWNER

Better idea: if you run Datasette in cross-database joining mode, all connections start out as memory connections and then have new databases attached to them on-demand.

All table view queries will be automatically rewritten to start SELECT db.table.one, db.table.two FROM db.table ...

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506
537716955 https://github.com/simonw/datasette/issues/283#issuecomment-537716955 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDUzNzcxNjk1NQ== simonw 9599 2019-10-02T23:02:15Z 2019-10-02T23:02:15Z OWNER

I've been thinking pretty hard about this as part of #569. My big concerns are:

  • If I'm caching and reusing connections I need to worry about the different combinations - if I have four databases do I cache separate connections for the ("one", "two") AND ("two", "three") AND ("one", "three") and so on pairs?
  • How does the API and interface deal with instances where you have a database connected as the primary and you want to ATTACH another database and talk to that as well?

I think the best way to do this is to say that cross-database joins will only be available against the :memory: database. Maybe with an optional mode you can run like datasette --crossdb which causes every database to be ATTACHd to that connection with an alias so you can start running queries.

If this proves to be a problem when hundreds of files are attached to a Datasette Library instance (#417) then maybe cross database joins are handled (in that case) by the authenticated user selecting which ones to ?_attach= and detaching them at the end of the request. Also perhaps limit to joining across a maximum of 3 databases at once in this case.

I can probably avoid the scariest negative consequences of cross-database joins by having them turned off by default for signed-out users. The datasette-on-my-laptop or authenticated Datasette Library cases can be opt-in and can be a little less locked down.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506
391768302 https://github.com/simonw/datasette/issues/283#issuecomment-391768302 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDM5MTc2ODMwMg== simonw 9599 2018-05-24T16:00:05Z 2018-05-24T16:00:05Z OWNER

I like /-/all-5de27e3 for this (with /-/all redirecting to the correct hash)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506
391756841 https://github.com/simonw/datasette/issues/283#issuecomment-391756841 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDM5MTc1Njg0MQ== simonw 9599 2018-05-24T15:27:42Z 2018-05-24T15:27:42Z OWNER

For an example query that pre-populates that textarea... maybe a UNION that pulls the first 10 rows from the first table of each of the first two databases?

select * from (select rowid, actors from fivethirtyeight.[love-actually/love_actually_adjacencies] limit 10)
   union all
select * from (select rowid, city from [google-trends].[20150430_UKDebate] limit 10)

https://datasette-cross-database-joins-prototype.now.sh/memory?sql=select++from+%28select+rowid%2C+actors+from+fivethirtyeight.%5Blove-actually%2Flove_actually_adjacencies%5D+limit+10%29%0D%0A+++union+all%0D%0Aselect++from+%28select+rowid%2C+city+from+%5Bgoogle-trends%5D.%5B20150430_UKDebate%5D+limit+10%29

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506
391755300 https://github.com/simonw/datasette/issues/283#issuecomment-391755300 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDM5MTc1NTMwMA== simonw 9599 2018-05-24T15:23:37Z 2018-05-24T15:23:37Z OWNER

On the /-all-5de27e3 page we can show the regular https://fivethirtyeight.datasettes.com/fivethirtyeight-5de27e3 interface but instead of the list of tables we can show a list of attached databases plus some help text showing how to construct a cross-database join.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506
391754506 https://github.com/simonw/datasette/issues/283#issuecomment-391754506 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDM5MTc1NDUwNg== simonw 9599 2018-05-24T15:21:37Z 2018-05-24T15:21:53Z OWNER

Giving it /all/ would be easier since that way the existing URL routes (including canned queries) would all work... but I would have to teach it NOT to expect a database content hash on that URL.

Or maybe it should still have a content hash (to enable far-future cache expiry headers on query results) but the hash should be constructed out of all of the other database hashes concatenated together.

That way the URLs would be /all-5de27e3 and /all-5de27e3/canned-query-name

Only downside: this would make it impossible to have a database file with the name all.db. I think that's probably an OK trade-off. You could turn the feature off with a config flag if you really want to use that filename (for whatever reason).

How about /-all-5de27e3/ instead to avoid collisions?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506
391752882 https://github.com/simonw/datasette/issues/283#issuecomment-391752882 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDM5MTc1Mjg4Mg== simonw 9599 2018-05-24T15:17:10Z 2018-05-24T15:17:10Z OWNER

Another option: give this the /-/all URL namespace.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506
391752629 https://github.com/simonw/datasette/issues/283#issuecomment-391752629 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDM5MTc1MjYyOQ== simonw 9599 2018-05-24T15:16:25Z 2018-05-24T15:16:25Z OWNER

Should this support canned queries too? I think it should, though that raises interesting questions regarding their URL structure.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506
391752425 https://github.com/simonw/datasette/issues/283#issuecomment-391752425 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDM5MTc1MjQyNQ== simonw 9599 2018-05-24T15:15:51Z 2018-05-24T15:15:51Z OWNER

This would make Datasett's SQL features a lot more instantly obvious to people who land on a homepage, which is probably a good thing.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506
391752218 https://github.com/simonw/datasette/issues/283#issuecomment-391752218 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDM5MTc1MjIxOA== simonw 9599 2018-05-24T15:15:19Z 2018-05-24T15:15:19Z OWNER

Most of the time Datasette is used with just a single database file. So maybe it makes sense for this option to be turned on by default and to ALWAYS be available on the Datasette instance homepage unless the user has explicitly disabled it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506
391584112 https://github.com/simonw/datasette/issues/283#issuecomment-391584112 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDM5MTU4NDExMg== simonw 9599 2018-05-24T04:26:29Z 2018-05-24T04:30:50Z OWNER

I built a very rough prototype of this to prove it could work. It's deployed here - and here's an example of a query that joins across two different databases:

https://datasette-cross-database-joins-prototype.now.sh/memory?sql=select+fivethirtyeight.%5Blove-actually%2Flove_actually_adjacencies%5D.rowid%2C%0D%0Afivethirtyeight.%5Blove-actually%2Flove_actually_adjacencies%5D.actors%2C%0D%0A%5Bgoogle-trends%5D.%5B20150430_UKDebate%5D.city%0D%0Afrom+fivethirtyeight.%5Blove-actually%2Flove_actually_adjacencies%5D%0D%0Ajoin+%5Bgoogle-trends%5D.%5B20150430_UKDebate%5D%0D%0A++on+%5Bgoogle-trends%5D.%5B20150430_UKDebate%5D.rowid+%3D+fivethirtyeight.%5Blove-actually%2Flove_actually_adjacencies%5D.rowid

select fivethirtyeight.[love-actually/love_actually_adjacencies].rowid,
fivethirtyeight.[love-actually/love_actually_adjacencies].actors,
[google-trends].[20150430_UKDebate].city
from fivethirtyeight.[love-actually/love_actually_adjacencies]
join [google-trends].[20150430_UKDebate]
  on [google-trends].[20150430_UKDebate].rowid = fivethirtyeight.[love-actually/love_actually_adjacencies].rowid

I deployed it like this:

datasette publish now --branch=cross-database-joins fivethirtyeight.db google-trends.db --name=datasette-cross-database-joins-prototype
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506
391584527 https://github.com/simonw/datasette/issues/283#issuecomment-391584527 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDM5MTU4NDUyNw== simonw 9599 2018-05-24T04:29:40Z 2018-05-24T04:29:40Z OWNER

Rather than stealing the /memory namespace for this it would be nicer if these cross-database joins could be executed at the very top-level URL of the Datasette instance - https://example.com/?sql=...

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506
391584366 https://github.com/simonw/datasette/issues/283#issuecomment-391584366 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDM5MTU4NDM2Ng== simonw 9599 2018-05-24T04:28:20Z 2018-05-24T04:28:20Z OWNER

I used some pretty ugly hacks, like faking an entire .inspect() block for the :memory: database just to get past the errors I was seeing. To ship this as a feature it will need quite a bit of code refactoring to make those hacks unnecessary.

https://github.com/simonw/datasette/blob/7a3040f5782375373b2b66e5969bc2c49b3a6f0e/datasette/views/database.py#L18-L26

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506
391583528 https://github.com/simonw/datasette/issues/283#issuecomment-391583528 https://api.github.com/repos/simonw/datasette/issues/283 MDEyOklzc3VlQ29tbWVudDM5MTU4MzUyOA== simonw 9599 2018-05-24T04:21:49Z 2018-05-24T04:21:49Z OWNER

The challenge here is which database should be the "default" database. The first database attached to SQLite is treated as the default - if no database is specified in a query, that's the database that queries will be executed against.

Currently, each database URL in Datasette (e.g. https://san-francisco.datasettes.com/sf-film-locations-84594a7 v.s. https://san-francisco.datasettes.com/sf-trees-ebc2ad9 ) gets its own independent connection, and all queries within that base URL run against that database.

If we're going to attach multiple databases to the same connection, how do we set which database gets to be the default?

The easiest thing to do here will be to have a special database (maybe which is turned off by default and can be enabled using datasette serve --enable-cross-database-joins or similar) which attaches to ALL the databases. Perhaps it starts as an in-memory database, maybe at /memory?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support cross-database joins 325958506

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);
Powered by Datasette · Query took 45.725ms · About: github-to-sqlite