issue_comments

131 rows where author_association = "CONTRIBUTOR" sorted by updated_at descending

View and edit SQL

Suggested facets: reactions, created_at (date), updated_at (date)

author_association

  • CONTRIBUTOR · 131
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue
653002499 https://github.com/simonw/datasette/issues/889#issuecomment-653002499 https://api.github.com/repos/simonw/datasette/issues/889 MDEyOklzc3VlQ29tbWVudDY1MzAwMjQ5OQ== amjith 49260 2020-07-02T13:22:13Z 2020-07-02T13:22:13Z CONTRIBUTOR

I was able to narrow this down to the fact that lifespan protocol is turned on.

I see the workaround you've used here: https://github.com/simonw/datasette-debug-asgi/commit/72d568d32a3159c763ce908c0b269736935c6987

If so, maybe it's time to update some of the asg_wrapper plugins.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
asgi_wrapper plugin hook is crashing at startup 649907676
652990131 https://github.com/simonw/datasette/issues/889#issuecomment-652990131 https://api.github.com/repos/simonw/datasette/issues/889 MDEyOklzc3VlQ29tbWVudDY1Mjk5MDEzMQ== amjith 49260 2020-07-02T12:58:11Z 2020-07-02T13:00:18Z CONTRIBUTOR

FWIW, this error does NOT happen in datasette 0.45a4.

It only started on 0.45a5

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
asgi_wrapper plugin hook is crashing at startup 649907676
652394742 https://github.com/simonw/datasette/pull/883#issuecomment-652394742 https://api.github.com/repos/simonw/datasette/issues/883 MDEyOklzc3VlQ29tbWVudDY1MjM5NDc0Mg== abdusco 3243482 2020-07-01T12:41:13Z 2020-07-01T12:41:13Z CONTRIBUTOR

Well tests need to be updated.

I need to get tests working on Windows.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Skip counting hidden tables 648749062
652297139 https://github.com/simonw/datasette/pull/883#issuecomment-652297139 https://api.github.com/repos/simonw/datasette/issues/883 MDEyOklzc3VlQ29tbWVudDY1MjI5NzEzOQ== abdusco 3243482 2020-07-01T09:11:29Z 2020-07-01T09:11:29Z CONTRIBUTOR

Turns out we should include hidden tables in the result dict, or we're breaking tests. I've committed a refactor https://github.com/simonw/datasette/pull/883/commits/4f06e1bf6fbe4b73be770b87f610bf7c0e6e3ea7

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Skip counting hidden tables 648749062
652255960 https://github.com/simonw/datasette/issues/877#issuecomment-652255960 https://api.github.com/repos/simonw/datasette/issues/877 MDEyOklzc3VlQ29tbWVudDY1MjI1NTk2MA== abdusco 3243482 2020-07-01T07:52:25Z 2020-07-01T08:10:00Z CONTRIBUTOR

I am calling the API from another origin, so injecting CSRF token into templates wouldn't work.

EDIT:

I'll try the new version, it sounds promising

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Consider dropping explicit CSRF protection entirely? 648421105
652261382 https://github.com/simonw/datasette/issues/877#issuecomment-652261382 https://api.github.com/repos/simonw/datasette/issues/877 MDEyOklzc3VlQ29tbWVudDY1MjI2MTM4Mg== abdusco 3243482 2020-07-01T08:03:17Z 2020-07-01T08:03:23Z CONTRIBUTOR

Bearer tokens sound interesting. Where do tokens come from? An auth provider of my choosing? How do they get verified?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Consider dropping explicit CSRF protection entirely? 648421105
652166115 https://github.com/simonw/datasette/issues/877#issuecomment-652166115 https://api.github.com/repos/simonw/datasette/issues/877 MDEyOklzc3VlQ29tbWVudDY1MjE2NjExNQ== abdusco 3243482 2020-07-01T03:28:07Z 2020-07-01T03:28:07Z CONTRIBUTOR

Does this mean custom routes get to expose endpoints accepting POST requests? I've tried earlier to add some POST endpoints, but requests were being rejected by Datasette due to CSRF

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Consider dropping explicit CSRF protection entirely? 648421105
652160909 https://github.com/simonw/datasette/issues/859#issuecomment-652160909 https://api.github.com/repos/simonw/datasette/issues/859 MDEyOklzc3VlQ29tbWVudDY1MjE2MDkwOQ== abdusco 3243482 2020-07-01T03:09:32Z 2020-07-01T03:10:21Z CONTRIBUTOR

I've just realized Datasette tries to count hidden tables too. There are 5 visible tables, 25 hidden tables, which I haven't realize earlier to consider their effect. I've turned off counting for hidden tables to see if it has any effect.

What's the point of counting FTS tables?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841
648669523 https://github.com/simonw/datasette/issues/859#issuecomment-648669523 https://api.github.com/repos/simonw/datasette/issues/859 MDEyOklzc3VlQ29tbWVudDY0ODY2OTUyMw== abdusco 3243482 2020-06-24T08:13:23Z 2020-06-24T10:30:36Z CONTRIBUTOR

I tried setting cache_size_kb=0 then cache_size_kb=100000, still getting this behavior. I even changed Database::table_counts and lowered time limit to 1

table_count = (
    await self.execute(
        "select count(*) from [{}]".format(table),
        custom_time_limit=1,
    )
).rows[0][0]
counts[table] = table_count

I feel like 10 seconds is a magic number, like a processing timeout and datasette gives up and returns the page.
Index page loads instantly, table page, query page, as well. But when I return to database page after some time, it loads in 10s.

EDIT:

It's always like 10 + 0.3s, like 10s wait and timeout then 300ms to render the page

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841
648232645 https://github.com/simonw/datasette/issues/859#issuecomment-648232645 https://api.github.com/repos/simonw/datasette/issues/859 MDEyOklzc3VlQ29tbWVudDY0ODIzMjY0NQ== abdusco 3243482 2020-06-23T15:19:53Z 2020-06-23T15:19:53Z CONTRIBUTOR

The issue seems to appear sporadically, like when I return to database page after a while, during which some records have been added to the database.

I've just visited database, page first visit took ~10s, consecutive visits took 0.3s.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841
647925594 https://github.com/simonw/datasette/issues/859#issuecomment-647925594 https://api.github.com/repos/simonw/datasette/issues/859 MDEyOklzc3VlQ29tbWVudDY0NzkyNTU5NA== abdusco 3243482 2020-06-23T05:55:21Z 2020-06-23T06:28:29Z CONTRIBUTOR

Hmm, not seeing the problem now.
I've removed the commented out sections in database.py and restarted the process. Database page now loads in <250ms.

I have couple of workers that check some pages regularly and scrape new content and save to the DB. Could it be that datasette tries to recount tables every time database size changes? Normally it keeps a count cache, but as DB gets updated so often (new content every 5 min or so) it's practically recounting every time I go to the database page?

EDIT:
It turns out it doesn't hold cache with mutable databases.

I'll update the issue with more findings and a better way to reproduce the problem if I encounter it again.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841
647936117 https://github.com/simonw/datasette/issues/859#issuecomment-647936117 https://api.github.com/repos/simonw/datasette/issues/859 MDEyOklzc3VlQ29tbWVudDY0NzkzNjExNw== abdusco 3243482 2020-06-23T06:25:17Z 2020-06-23T06:25:17Z CONTRIBUTOR

sqlite-generate many-cols.db --tables 2 --rows 200000 --columns 50

Looks like that will take 35 minutes to run (it's not a particularly fast tool).

Try chunking write operations into batches every 1000 records or so.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841
647935300 https://github.com/simonw/datasette/issues/859#issuecomment-647935300 https://api.github.com/repos/simonw/datasette/issues/859 MDEyOklzc3VlQ29tbWVudDY0NzkzNTMwMA== abdusco 3243482 2020-06-23T06:23:01Z 2020-06-23T06:23:01Z CONTRIBUTOR

You said "200k+, 50+ rows in a couple of tables" - does that mean 50+ columns? I'll try with larger numbers of columns and see what difference that makes.

Ah that was a typo, I meant 50k.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841
647923666 https://github.com/simonw/datasette/issues/859#issuecomment-647923666 https://api.github.com/repos/simonw/datasette/issues/859 MDEyOklzc3VlQ29tbWVudDY0NzkyMzY2Ng== abdusco 3243482 2020-06-23T05:49:31Z 2020-06-23T05:49:31Z CONTRIBUTOR

I think I should mention that having FTS on all tables mean I have 5 visible, 25 hidden (FTS) tables displayed on database page.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841
647922203 https://github.com/simonw/datasette/issues/859#issuecomment-647922203 https://api.github.com/repos/simonw/datasette/issues/859 MDEyOklzc3VlQ29tbWVudDY0NzkyMjIwMw== abdusco 3243482 2020-06-23T05:44:58Z 2020-06-23T05:44:58Z CONTRIBUTOR

I'm seeing the problem on database page. Index page and table page runs quite fast.

  • Tables have <10 columns (id, url, title, body_html, date, author, meta (for keeping unstructured json)). I've added index on date columns (using sqlite-utils) in addition to the index present on id columns.
  • All tables have FTS enabled on text and varchar columns (title, body_html vs) to speed up searching.
  • There are couple of tables related with foreign keys (think a thread in a forum and posts in that thread, related with thread_id)
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841
647194131 https://github.com/simonw/datasette/issues/859#issuecomment-647194131 https://api.github.com/repos/simonw/datasette/issues/859 MDEyOklzc3VlQ29tbWVudDY0NzE5NDEzMQ== abdusco 3243482 2020-06-21T23:15:54Z 2020-06-21T23:26:09Z CONTRIBUTOR

I'm not sure if table counts are to blame. There shouldn't be a ~3 orders of magnitude difference.

user@klein /a/w/scrapyard (master)> set sql "select count(*) from table_1; select count(*) from table_2; select count(*) from table_3;"
user@klein /a/w/scrapyard (master)> time sqlite3 scrapyard.db "$sql"
187489
46492
2229

________________________________________________________
Executed in   25.57 millis    fish           external
   usr time    3.55 millis    0.00 micros    3.55 millis
   sys time   22.42 millis  1123.00 micros   21.30 millis

but not letting datasette count the tables definitely helps.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841
647135713 https://github.com/simonw/datasette/issues/859#issuecomment-647135713 https://api.github.com/repos/simonw/datasette/issues/859 MDEyOklzc3VlQ29tbWVudDY0NzEzNTcxMw== abdusco 3243482 2020-06-21T14:30:02Z 2020-06-21T14:30:02Z CONTRIBUTOR

Oops, the same method is called from both index and database pages. But removing select count queries speed up the page load quite a bit.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841
645293374 https://github.com/simonw/datasette/issues/851#issuecomment-645293374 https://api.github.com/repos/simonw/datasette/issues/851 MDEyOklzc3VlQ29tbWVudDY0NTI5MzM3NA== abdusco 3243482 2020-06-17T10:32:02Z 2020-06-17T10:32:28Z CONTRIBUTOR

Welp, I'm an idiot.

Turns out I had a sneaky comma , after sql key:

... (:name, :url),

which tells sqlite to expect another values(...) list.

Correcting the SQL solved the issue.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Having trouble getting writable canned queries to work 640330278
643709037 https://github.com/simonw/datasette/issues/691#issuecomment-643709037 https://api.github.com/repos/simonw/datasette/issues/691 MDEyOklzc3VlQ29tbWVudDY0MzcwOTAzNw== amjith 49260 2020-06-14T02:35:16Z 2020-06-14T02:35:16Z CONTRIBUTOR

The server should reload in the config_dir mode.

Ref: #848

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
--reload sould reload server if code in --plugins-dir changes 574021194
632555800 https://github.com/simonw/datasette/issues/767#issuecomment-632555800 https://api.github.com/repos/simonw/datasette/issues/767 MDEyOklzc3VlQ29tbWVudDYzMjU1NTgwMA== rixx 2657547 2020-05-22T08:00:23Z 2020-05-22T08:00:23Z CONTRIBUTOR

That would be perfect!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Allow to specify a URL fragment for canned queries 620969465
623463200 https://github.com/simonw/datasette/pull/730#issuecomment-623463200 https://api.github.com/repos/simonw/datasette/issues/730 MDEyOklzc3VlQ29tbWVudDYyMzQ2MzIwMA== dependabot-preview[bot] 27856297 2020-05-04T13:27:22Z 2020-05-04T13:27:22Z CONTRIBUTOR

Superseded by #753.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Update pytest-asyncio requirement from ~=0.10.0 to >=0.10,<0.12 604001627
622599528 https://github.com/simonw/sqlite-utils/issues/103#issuecomment-622599528 https://api.github.com/repos/simonw/sqlite-utils/issues/103 MDEyOklzc3VlQ29tbWVudDYyMjU5OTUyOA== b0b5h4rp13 32605365 2020-05-01T22:49:12Z 2020-05-02T11:15:44Z CONTRIBUTOR

With SQLITE_MAX_VARS = 999, or even 899, This hits the problem with the batch rows causing a overflow (works fine if SQLITE_MAX_VARS = 799).

p.s. I have tried a few list of dicts to sqlite modules and this was the easiest to use/understand

------------- file begins ------------------
import sqlite_utils as su

data = [
{'tickerId': 913324382, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'CONSTELLATION B', 'symbol': 'STZ B', 'disSymbol': 'STZ-B', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'status': 'D', 'close': '163.13', 'change': '6.46', 'changeRatio': '0.0412', 'marketValue': '31180699895.63', 'volume': '417', 'turnoverRate': '0.0000'},
{'tickerId': 913323791, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Molina Health', 'symbol': 'MOH', 'disSymbol': 'MOH', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '173.25', 'change': '9.28', 'changeRatio': '0.0566', 'pPrice': '173.25', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '10520341695.50', 'volume': '1281557', 'turnoverRate': '0.0202'},
{'tickerId': 913257501, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Seattle Genetics', 'symbol': 'SGEN', 'disSymbol': 'SGEN', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '145.64', 'change': '8.41', 'changeRatio': '0.0613', 'pPrice': '146.45', 'pChange': '0.8100', 'pChRatio': '0.0056', 'marketValue': '25117961347.60', 'volume': '2791411', 'turnoverRate': '0.0162'},
{'tickerId': 925381971, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Bandwidth', 'symbol': 'BAND', 'disSymbol': 'BAND', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '89.22', 'change': '7.66', 'changeRatio': '0.0939', 'pPrice': '89.00', 'pChange': '-0.2200', 'pChRatio': '-0.0025', 'marketValue': '2100025474.98', 'volume': '1508629', 'turnoverRate': '0.0641'},
{'tickerId': 913323935, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Magellan Health', 'symbol': 'MGLN', 'disSymbol': 'MGLN', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '68.00', 'change': '7.27', 'changeRatio': '0.1197', 'pPrice': '68.00', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '1697894040.00', 'volume': '448919', 'turnoverRate': '0.0180'},
{'tickerId': 913254854, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'On Assignment', 'symbol': 'ASGN', 'disSymbol': 'ASGN', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '53.04', 'change': '6.59', 'changeRatio': '0.1419', 'pPrice': '53.04', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '2811120000.00', 'volume': '1339771', 'turnoverRate': '0.0253'},
{'tickerId': 913255732, 'exchangeId': 95, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Arcturus', 'symbol': 'ARCT', 'disSymbol': 'ARCT', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NMS', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '40.86', 'change': '6.36', 'changeRatio': '0.1843', 'pPrice': '42.60', 'pChange': '1.740', 'pChRatio': '0.0426', 'marketValue': '812021444.46', 'volume': '1577508', 'turnoverRate': '0.0794'},
{'tickerId': 913256616, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'DexCom', 'symbol': 'DXCM', 'disSymbol': 'DXCM', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '341.52', 'change': '6.32', 'changeRatio': '0.0189', 'pPrice': '340.00', 'pChange': '-1.5200', 'pChRatio': '-0.0045', 'marketValue': '31522296000.00', 'volume': '1008849', 'turnoverRate': '0.0109'},
{'tickerId': 913255108, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Clorox', 'symbol': 'CLX', 'disSymbol': 'CLX', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '192.71', 'change': '6.27', 'changeRatio': '0.0336', 'pPrice': '192.95', 'pChange': '0.2400', 'pChRatio': '0.0012', 'marketValue': '24185773318.28', 'volume': '4996414', 'turnoverRate': '0.0398'},
{'tickerId': 925314627, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'FRANCO NEVADA', 'symbol': 'FNV', 'disSymbol': 'FNV', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '137.85', 'change': '5.64', 'changeRatio': '0.0427', 'pPrice': '138.50', 'pChange': '0.6500', 'pChRatio': '0.0047', 'marketValue': '26110405326.30', 'volume': '1047688', 'turnoverRate': '0.0055'},
{'tickerId': 913254955, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Aon Plc', 'symbol': 'AON', 'disSymbol': 'AON', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '178.21', 'change': '5.54', 'changeRatio': '0.0321', 'pPrice': '178.21', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '41181209117.22', 'volume': '2026234', 'turnoverRate': '0.0088'},
{'tickerId': 913324105, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Willis Towers', 'symbol': 'WLTW', 'disSymbol': 'WLTW', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '183.34', 'change': '5.05', 'changeRatio': '0.0283', 'pPrice': '183.34', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '23597461124.96', 'volume': '968943', 'turnoverRate': '0.0075'},
{'tickerId': 913254759, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'TELADOC HEALTH', 'symbol': 'TDOC', 'disSymbol': 'TDOC', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '169.43', 'change': '4.84', 'changeRatio': '0.0294', 'pPrice': '168.88', 'pChange': '-0.5500', 'pChRatio': '-0.0032', 'marketValue': '12614616858.38', 'volume': '2628946', 'turnoverRate': '0.0353'},
{'tickerId': 913255222, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Emergent Bio', 'symbol': 'EBS', 'disSymbol': 'EBS', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '78.70', 'change': '4.75', 'changeRatio': '0.0642', 'pPrice': '78.40', 'pChange': '-0.3000', 'pChRatio': '-0.0038', 'marketValue': '4113368277.10', 'volume': '783804', 'turnoverRate': '0.0150'},
{'tickerId': 913323443, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Pool', 'symbol': 'POOL', 'disSymbol': 'POOL', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '216.02', 'change': '4.36', 'changeRatio': '0.0206', 'pPrice': '216.02', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '8696077573.82', 'volume': '310837', 'turnoverRate': '0.0077'},
{'tickerId': 913257075, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Masimo', 'symbol': 'MASI', 'disSymbol': 'MASI', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '218.00', 'change': '4.09', 'changeRatio': '0.0191', 'pPrice': '217.00', 'pChange': '-1.0000', 'pChRatio': '-0.0046', 'marketValue': '11797070000.00', 'volume': '542131', 'turnoverRate': '0.0100'},
{'tickerId': 913253761, 'exchangeId': 10, 'type': 2, 'secType': [62], 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Pope Resources', 'symbol': 'POPE', 'disSymbol': 'POPE', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NAS', 'listStatus': 1, 'template': 'stock', 'status': 'D', 'close': '101.05', 'change': '3.95', 'changeRatio': '0.0407', 'pPrice': '99.90', 'pChange': '2.800', 'pChRatio': '0.0288', 'marketValue': '447370075.75', 'volume': '33138', 'turnoverRate': '0.0075'},
{'tickerId': 913323560, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Seneca Foods', 'symbol': 'SENEB', 'disSymbol': 'SENEB', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'status': 'D', 'close': '40.04', 'change': '3.84', 'changeRatio': '0.1061', 'marketValue': '347950039.71', 'volume': '501'},
{'tickerId': 913324274, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Resmed', 'symbol': 'RMD', 'disSymbol': 'RMD', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '159.07', 'change': '3.75', 'changeRatio': '0.0241', 'pPrice': '159.07', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '23004217759.29', 'volume': '1267075', 'turnoverRate': '0.0088'},
{'tickerId': 913323736, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Vertex Pharms', 'symbol': 'VRTX', 'disSymbol': 'VRTX', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '254.90', 'change': '3.70', 'changeRatio': '0.0147', 'pPrice': '255.00', 'pChange': '0.1000', 'pChRatio': '0.0004', 'marketValue': '66062980780.10', 'volume': '1939843', 'turnoverRate': '0.0075'},
{'tickerId': 913323767, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'MCCORMICK VTG', 'symbol': 'MKC V', 'disSymbol': 'MKC-V', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'status': 'D', 'close': '159.99', 'change': '3.42', 'changeRatio': '0.0218', 'marketValue': '21262671000.00', 'volume': '432', 'turnoverRate': '0.0000'},
{'tickerId': 950118595, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'ZOOM VIDEO', 'symbol': 'ZM', 'disSymbol': 'ZM', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '138.56', 'change': '3.39', 'changeRatio': '0.0251', 'pPrice': '138.99', 'pChange': '0.4300', 'pChRatio': '0.0031', 'marketValue': '38620532420.16', 'volume': '13786017', 'turnoverRate': '0.0495'},
{'tickerId': 916040738, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'WHEATON PRECIOUS', 'symbol': 'WPM', 'disSymbol': 'WPM', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '41.10', 'change': '3.34', 'changeRatio': '0.0885', 'pPrice': '41.09', 'pChange': '-0.0100', 'pChRatio': '-0.0002', 'marketValue': '18404536146.30', 'volume': '5019137', 'turnoverRate': '0.0112'},
{'tickerId': 913257174, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Royal Gold', 'symbol': 'RGLD', 'disSymbol': 'RGLD', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '125.86', 'change': '3.33', 'changeRatio': '0.0272', 'pPrice': '125.86', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '8253015011.08', 'volume': '853473', 'turnoverRate': '0.0130'},
{'tickerId': 913254394, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Fortune Brand', 'symbol': 'FBHS', 'disSymbol': 'FBHS', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '51.50', 'change': '3.30', 'changeRatio': '0.0685', 'pPrice': '51.50', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '7194870278.50', 'volume': '3004021', 'turnoverRate': '0.0214'},
{'tickerId': 913323312, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Liberty Global', 'symbol': 'LBTYK', 'disSymbol': 'LBTYK', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '21.49', 'change': '3.18', 'changeRatio': '0.1737', 'pPrice': '21.48', 'pChange': '-0.0100', 'pChRatio': '-0.0005', 'marketValue': '13594662302.41', 'volume': '19980228', 'turnoverRate': '0.0315'},
{'tickerId': 913323882, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Preformed Line', 'symbol': 'PLPC', 'disSymbol': 'PLPC', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'status': 'D', 'close': '52.82', 'change': '3.14', 'changeRatio': '0.0632', 'pPrice': '52.10', 'pChange': '-0.7200', 'pChRatio': '-0.0136', 'marketValue': '264979981.20', 'volume': '9305', 'turnoverRate': '0.0018'},
{'tickerId': 913323248, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Discovery', 'symbol': 'DISCB', 'disSymbol': 'DISCB', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'status': 'A', 'close': '57.95', 'change': '23.63', 'changeRatio': '0.6884', 'pPrice': '54.26', 'pChange': '-3.6900', 'pChRatio': '-0.0637', 'marketValue': '29362894177.95', 'volume': '218305', 'turnoverRate': '0.0004'},
{'tickerId': 913323930, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'MercadoLibre', 'symbol': 'MELI', 'disSymbol': 'MELI', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '605.52', 'change': '22.01', 'changeRatio': '0.0377', 'pPrice': '603.69', 'pChange': '-1.8300', 'pChRatio': '-0.0030', 'marketValue': '30226598045.28', 'volume': '699008', 'turnoverRate': '0.0140'},
{'tickerId': 913257170, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Liberty Global', 'symbol': 'LBTYA', 'disSymbol': 'LBTYA', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '22.28', 'change': '2.86', 'changeRatio': '0.1473', 'pPrice': '22.29', 'pChange': '0.0100', 'pChRatio': '0.0004', 'marketValue': '14094419548.52', 'volume': '10534672', 'turnoverRate': '0.0167'},
{'tickerId': 913303991, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Liberty Brodband', 'symbol': 'LBRDK', 'disSymbol': 'LBRDK', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '125.44', 'change': '2.76', 'changeRatio': '0.0225', 'pPrice': '125.44', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '22817900904.96', 'volume': '926177', 'turnoverRate': '0.0042'},
{'tickerId': 913257082, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Helen of Troy', 'symbol': 'HELE', 'disSymbol': 'HELE', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '167.04', 'change': '2.76', 'changeRatio': '0.0168', 'pPrice': '167.04', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '4216707982.08', 'volume': '341465', 'turnoverRate': '0.0135'},
{'tickerId': 913256458, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Forrester', 'symbol': 'FORR', 'disSymbol': 'FORR', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '33.88', 'change': '2.58', 'changeRatio': '0.0824', 'marketValue': '635419400.00', 'volume': '85115', 'turnoverRate': '0.0045'},
{'tickerId': 950158952, 'exchangeId': 95, 'type': 2, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'LYRA THERAPEUTICS, INC.', 'symbol': 'LYRA', 'disSymbol': 'LYRA', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NMS', 'listStatus': 1, 'template': 'ipo', 'status': 'A', 'close': '18.56', 'change': '2.56', 'changeRatio': '0.1600', 'pPrice': '18.96', 'pChange': '0.4000', 'pChRatio': '0.0216', 'marketValue': '229705575.68', 'volume': '1738472', 'turnoverRate': '0.1405'},
{'tickerId': 913257570, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Bio-Techne', 'symbol': 'TECH', 'disSymbol': 'TECH', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '227.54', 'change': '2.54', 'changeRatio': '0.0113', 'pPrice': '227.54', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '8726538309.18', 'volume': '497006', 'turnoverRate': '0.0130'},
{'tickerId': 913323246, 'exchangeId': 96, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Bel Fuse', 'symbol': 'BELFB', 'disSymbol': 'BELFB', 'disExchangeCode': 'NASDAQ', 'exchangeCode': 'NSQ', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '9.99', 'change': '2.53', 'changeRatio': '0.3391', 'pPrice': '9.75', 'pChange': '-0.2400', 'pChRatio': '-0.0240', 'marketValue': '122562454.86', 'volume': '177634', 'turnoverRate': '0.0145'},
{'tickerId': 916040647, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Agnico Eagle', 'symbol': 'AEM', 'disSymbol': 'AEM', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '61.20', 'change': '2.52', 'changeRatio': '0.0429', 'pPrice': '61.10', 'pChange': '-0.1000', 'pChRatio': '-0.0016', 'marketValue': '14739911553.60', 'volume': '2820765', 'turnoverRate': '0.0117'},
{'tickerId': 913303768, 'exchangeId': 12, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'CHASE CORP', 'symbol': 'CCF', 'disSymbol': 'CCF', 'disExchangeCode': 'AMEX', 'exchangeCode': 'ASE', 'listStatus': 1, 'template': 'stock', 'status': 'D', 'close': '96.71', 'change': '2.45', 'changeRatio': '0.0260', 'marketValue': '916799598.60', 'volume': '29229', 'turnoverRate': '0.0031'},
{'tickerId': 913324557, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'Allergan', 'symbol': 'AGN', 'disSymbol': 'AGN', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'A', 'close': '189.74', 'change': '2.40', 'changeRatio': '0.0128', 'pPrice': '189.76', 'pChange': '0.0200', 'pChRatio': '0.0001', 'marketValue': '62424842326.10', 'volume': '5787032', 'turnoverRate': '0.0176'},
{'tickerId': 913324566, 'exchangeId': 11, 'type': 2, 'secType': 61, 'regionId': 6, 'regionCode': 'US', 'currencyId': 247, 'name': 'West Pharm Svc', 'symbol': 'WST', 'disSymbol': 'WST', 'disExchangeCode': 'NYSE', 'exchangeCode': 'NYSE', 'listStatus': 1, 'template': 'stock', 'derivativeSupport': 1, 'status': 'D', 'close': '191.64', 'change': '2.38', 'changeRatio': '0.0126', 'pPrice': '191.64', 'pChange': '0.0000', 'pChRatio': '0.0000', 'marketValue': '14078267117.08', 'volume': '352460', 'turnoverRate': '0.0042'}
]

db = su.Database(f"overnight hold.db" )
db['active'].insert_all(data)

--------------- file ends ----------------------

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
sqlite3.OperationalError: too many SQL variables in insert_all when using rows with varying numbers of columns 610517472
612216820 https://github.com/simonw/datasette/issues/236#issuecomment-612216820 https://api.github.com/repos/simonw/datasette/issues/236 MDEyOklzc3VlQ29tbWVudDYxMjIxNjgyMA== cldellow 193185 2020-04-10T21:03:38Z 2020-04-10T21:03:38Z CONTRIBUTOR

I made a repo at https://github.com/code402/datasette-lambda to demonstrate the idea, and scratch my personal itch for this.

The demo relies on some central authority having already published a public, reusable Lambda layer with Datasette & its dependencies. I think that differs from the other publish plugins which seem to mainly publish Dockerfiles that the host will interpret to install deps from a requirements.txt file.

I chose that approach because uvloop appears to be a dependency with native code that needs to be compiled for the target runtime environment. In this case, that's Amazon Linux 2. I'm not 100% clear on whether that's still required, because:

  • maybe uvloop is only needed for uvicorn, which the demo doesn't actually use since HTTP routing is handled by API Gateway
  • it seems like uvloop may be an optional, drop-in optimization for asyncio in any case (but I may be misreading this; I'm very much a Python noob)

If it's the case that uvloop is truly optional, then I think the publish plugin could do the packaging on the user's machine, regardless of what flavour of operating system they're on. That'd be a bit slower for the user, but would provide the most long-term flexibility in terms of supporting plugins.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette publish lambda plugin 317001500
608716819 https://github.com/simonw/datasette/issues/236#issuecomment-608716819 https://api.github.com/repos/simonw/datasette/issues/236 MDEyOklzc3VlQ29tbWVudDYwODcxNjgxOQ== cldellow 193185 2020-04-03T22:19:00Z 2020-04-03T22:19:00Z CONTRIBUTOR

Hi Simon,

I'm thinking of attempting this. Can you clarify some questions I have?

1) I assume the goal is to have a CORS-friendly HTTPS endpoint that hosts the datasette service + user's db.

2) If that's the goal, I think Lambda alone is insufficient. Lambda provides the compute fabric, but not the HTTP routing. You'd also need to add Application Load Balancer or API Gateway to provide an HTTP endpoint that routes to the lambda function.

Do you have a preference between ALB or API GW? ALB has better economics at scale, but has a minimum monthly cost. API GW has worse per-request economics, but scales to zero when no requests are happening.

3) Does Datasette have any native components, or is it all pure python? If it has native bits, they'll likely need to be recompiled to work on Amazon Linux 2.

4) There are a few disparate services that need to be wired together to expose a Python service securely to the web. If I was doing this outside of the datasette publish system, I'd use an AWS CloudFormation template. Even within datasette, I think it still makes sense to use a CloudFormation template and just have the publish plugin invoke it (via the standard aws cli) with user-specified parameters. Does that sound reasonable to you?

Thanks for your help!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette publish lambda plugin 317001500
604328163 https://github.com/simonw/datasette/issues/573#issuecomment-604328163 https://api.github.com/repos/simonw/datasette/issues/573 MDEyOklzc3VlQ29tbWVudDYwNDMyODE2Mw== psychemedia 82988 2020-03-26T09:41:30Z 2020-03-26T09:41:30Z CONTRIBUTOR

Fixed by @simonw; example here: https://github.com/simonw/jupyterserverproxy-datasette-demo

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Exposing Datasette via Jupyter-server-proxy 492153532
590022164 https://github.com/simonw/datasette/pull/666#issuecomment-590022164 https://api.github.com/repos/simonw/datasette/issues/666 MDEyOklzc3VlQ29tbWVudDU5MDAyMjE2NA== kevindkeogh 13896256 2020-02-23T03:26:00Z 2020-02-23T03:26:00Z CONTRIBUTOR

It was very helpful for me, using it for a 15M row table. Added a test, happy to amend though!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use inspect-file, if possible, for total row count 562085508
586599424 https://github.com/simonw/datasette/issues/417#issuecomment-586599424 https://api.github.com/repos/simonw/datasette/issues/417 MDEyOklzc3VlQ29tbWVudDU4NjU5OTQyNA== psychemedia 82988 2020-02-15T15:12:19Z 2020-02-15T15:12:33Z CONTRIBUTOR

So could the polling support also allow you to call sqlite_utils to update a database with csv files? (Though I'm guessing you would only want to handle changed files? Do your scrapers check and cache csv datestamps/hashes?)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Datasette Library 421546944
582106085 https://github.com/simonw/datasette/pull/653#issuecomment-582106085 https://api.github.com/repos/simonw/datasette/issues/653 MDEyOklzc3VlQ29tbWVudDU4MjEwNjA4NQ== jaywgraves 418191 2020-02-04T20:43:43Z 2020-02-04T20:43:43Z CONTRIBUTOR

but this also doesn't have to land at all if it doesn't match your use case.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
allow leading comments in SQL input field 541331755
582105810 https://github.com/simonw/datasette/pull/653#issuecomment-582105810 https://api.github.com/repos/simonw/datasette/issues/653 MDEyOklzc3VlQ29tbWVudDU4MjEwNTgxMA== jaywgraves 418191 2020-02-04T20:43:01Z 2020-02-04T20:43:01Z CONTRIBUTOR

I think the existing code will be OK even if I strip the lines in the middle of a new line delimited string.

It's only used for the validation, SQLite handles the -- just fine and the whole SQL textarea still gets sent once it passes validation.

I can add your test case to my branch later this evening though.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
allow leading comments in SQL input field 541331755
573389669 https://github.com/simonw/sqlite-utils/issues/74#issuecomment-573389669 https://api.github.com/repos/simonw/sqlite-utils/issues/74 MDEyOklzc3VlQ29tbWVudDU3MzM4OTY2OQ== jayvdb 15092 2020-01-12T07:21:17Z 2020-01-12T07:21:17Z CONTRIBUTOR

I guess there is some extra flag for CliRunner.invoke to check exitcode and raise the exception, or that should be an extra assert added.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Test failures on openSUSE 15.1: AssertionError: Explicit other_table and other_column 546073980
573388052 https://github.com/simonw/sqlite-utils/issues/74#issuecomment-573388052 https://api.github.com/repos/simonw/sqlite-utils/issues/74 MDEyOklzc3VlQ29tbWVudDU3MzM4ODA1Mg== jayvdb 15092 2020-01-12T06:51:30Z 2020-01-12T06:51:30Z CONTRIBUTOR

Thanks. That showed me that there was a click cli runner error, and setting export LANG=en_US.UTF-8 fixed it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Test failures on openSUSE 15.1: AssertionError: Explicit other_table and other_column 546073980
565755208 https://github.com/simonw/datasette/pull/644#issuecomment-565755208 https://api.github.com/repos/simonw/datasette/issues/644 MDEyOklzc3VlQ29tbWVudDU2NTc1NTIwOA== chris48s 6025893 2019-12-14T21:33:31Z 2019-12-14T21:33:31Z CONTRIBUTOR

Hi @simonw

Have you had a chance to look at this at all?

I'm going to have a chunk of time free next week so if there is additional work needed on this, that would be a particularly convenient time for me to revisit this.

Cheers

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Validate metadata json on startup 530513784
559632608 https://github.com/simonw/datasette/issues/573#issuecomment-559632608 https://api.github.com/repos/simonw/datasette/issues/573 MDEyOklzc3VlQ29tbWVudDU1OTYzMjYwOA== psychemedia 82988 2019-11-29T01:43:38Z 2019-11-29T01:43:38Z CONTRIBUTOR

In passing, it looks like a start was made on a datasette Jupyter server extension in https://github.com/lucasdurand/jupyter-datasette although the build fails in MyBinder.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Exposing Datasette via Jupyter-server-proxy 492153532
559207224 https://github.com/simonw/datasette/issues/642#issuecomment-559207224 https://api.github.com/repos/simonw/datasette/issues/642 MDEyOklzc3VlQ29tbWVudDU1OTIwNzIyNA== psychemedia 82988 2019-11-27T18:40:57Z 2019-11-27T18:41:07Z CONTRIBUTOR

Would cookie cutter approaches also work for creating various flavours of customised templates?

I need to try to create a couple of sites for myself to get a feel for what sorts of thing are easily doable, and what cribbable cookie cutter items might be. I'm guessing https://simonwillison.net/2019/Nov/25/niche-museums/ is a good place to start from?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Provide a cookiecutter template for creating new plugins 529429214
558687342 https://github.com/simonw/datasette/issues/639#issuecomment-558687342 https://api.github.com/repos/simonw/datasette/issues/639 MDEyOklzc3VlQ29tbWVudDU1ODY4NzM0Mg== jacobian 21148 2019-11-26T15:40:00Z 2019-11-26T15:40:00Z CONTRIBUTOR

A bit of background: the reason heroku git:clone brings down an empty directory is because datasette publish heroku uses the builds API, rather than a git push, to release the app. I originally did this because it seemed like a lower bar than having a working git, but the downside is, as you found out, that tweaking the created app is hard.

So there's one option -- change datasette publish heroku to use git push instead of heroku builds:create.

@pkoppstein - what you suggested seems like it ought to work (you don't need maintenance mode, though). I'm not sure why it doesn't.

You could also look into using the slugs API to download the slug, change metadata.json, re-pack and re-upload the slug.

Ultimately though I think I think @simonw's idea of reading metadata.json from an external source might be better (#357). Reading from an alternate URL would be fine, or you could also just stuff the whole metadata.json into a Heroku config var, and write a plugin to read it from there.

Hope this helps a bit!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
updating metadata.json without recreating the app 527670799
549246007 https://github.com/simonw/datasette/pull/602#issuecomment-549246007 https://api.github.com/repos/simonw/datasette/issues/602 MDEyOklzc3VlQ29tbWVudDU0OTI0NjAwNw== rixx 2657547 2019-11-04T07:29:33Z 2019-11-04T07:29:33Z CONTRIBUTOR

Not sure – I'm always a bit weirded out when elements that I clicked disappear on me.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Offer to format readonly SQL 509535510
544214418 https://github.com/simonw/datasette/pull/601#issuecomment-544214418 https://api.github.com/repos/simonw/datasette/issues/601 MDEyOklzc3VlQ29tbWVudDU0NDIxNDQxOA== rixx 2657547 2019-10-20T02:29:49Z 2019-10-20T02:29:49Z CONTRIBUTOR

Submitted in #602!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Don't auto-format SQL on page load 509340359
544008944 https://github.com/simonw/datasette/pull/601#issuecomment-544008944 https://api.github.com/repos/simonw/datasette/issues/601 MDEyOklzc3VlQ29tbWVudDU0NDAwODk0NA== rixx 2657547 2019-10-18T23:40:48Z 2019-10-18T23:40:48Z CONTRIBUTOR

The only negative impact that comes to mind is that now you have no way to get the read-only query to be formatted nicely, I think, so maybe a second PR adding the formatting functionality even to the read-only page would be good?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Don't auto-format SQL on page load 509340359
544008463 https://github.com/simonw/datasette/pull/601#issuecomment-544008463 https://api.github.com/repos/simonw/datasette/issues/601 MDEyOklzc3VlQ29tbWVudDU0NDAwODQ2Mw== rixx 2657547 2019-10-18T23:39:21Z 2019-10-18T23:39:21Z CONTRIBUTOR

That looks right, and I completely agree with the intent.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Don't auto-format SQL on page load 509340359
541587823 https://github.com/simonw/datasette/pull/590#issuecomment-541587823 https://api.github.com/repos/simonw/datasette/issues/590 MDEyOklzc3VlQ29tbWVudDU0MTU4NzgyMw== rixx 2657547 2019-10-14T09:58:23Z 2019-10-14T09:58:23Z CONTRIBUTOR

Added tests.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spaces in DB names 505818256
541562581 https://github.com/simonw/datasette/pull/590#issuecomment-541562581 https://api.github.com/repos/simonw/datasette/issues/590 MDEyOklzc3VlQ29tbWVudDU0MTU2MjU4MQ== rixx 2657547 2019-10-14T08:57:46Z 2019-10-14T08:57:46Z CONTRIBUTOR

Ah, thank you – I saw the need for unit tests but wasn't sure what the best way to add one would be.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spaces in DB names 505818256
541119038 https://github.com/simonw/datasette/issues/512#issuecomment-541119038 https://api.github.com/repos/simonw/datasette/issues/512 MDEyOklzc3VlQ29tbWVudDU0MTExOTAzOA== rixx 2657547 2019-10-11T15:49:13Z 2019-10-11T15:49:13Z CONTRIBUTOR

How open are you to changing the config variable names (with appropriate deprecation, of course)? "about_url_text", "license_url_text" etc might be better suited to convey that these are just meant as basically URL titles.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"about" parameter in metadata does not appear when alone 457147936
541118904 https://github.com/simonw/datasette/issues/507#issuecomment-541118904 https://api.github.com/repos/simonw/datasette/issues/507 MDEyOklzc3VlQ29tbWVudDU0MTExODkwNA== rixx 2657547 2019-10-11T15:48:49Z 2019-10-11T15:48:49Z CONTRIBUTOR

Headless Chrome and Firefox via Selenium are a solid choice in my experience. You may be interested in how pretix and pretalx solve this problem: They use pytest to create those screenshots on release to make sure they are up to date. See this writeup and this repo.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Every datasette plugin on the ecosystem page should have a screenshot 455852801
541052329 https://github.com/simonw/datasette/issues/585#issuecomment-541052329 https://api.github.com/repos/simonw/datasette/issues/585 MDEyOklzc3VlQ29tbWVudDU0MTA1MjMyOQ== rixx 2657547 2019-10-11T12:53:51Z 2019-10-11T12:53:51Z CONTRIBUTOR

I think this would be good, yeah – currently, databases are explicitly sorted by name in the IndexView, we could just remove that part (and use an OrderedDict for consistency, I suppose)?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Databases on index page should display in order they were passed to "datasette serve"? 503217375
533818697 https://github.com/simonw/sqlite-utils/issues/61#issuecomment-533818697 https://api.github.com/repos/simonw/sqlite-utils/issues/61 MDEyOklzc3VlQ29tbWVudDUzMzgxODY5Nw== amjith 49260 2019-09-21T18:09:01Z 2019-09-21T18:09:28Z CONTRIBUTOR

@witeshadow The library version doesn't have helpers around CSV (at least not from what I can see in the code).

But here's a snippet that makes it easy to insert from CSV using the library.

import csv
from sqlite_utils import Database

# CSV Reader

csv_file = open("filename.csv")   # open the csv file.
reader = csv.reader(csv_file)  # Create a CSV reader
headers = next(reader)   # First line is the header
docs = (dict(zip(headers, row)) for row in reader)

# Now you can use the `sqlite_utils` library. 

db = Database("my_database.db")
db["table_name"].insert_all(docs)

This snippet is adapted from reading the CLI source code on how it implements the csv option.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
importing CSV to SQLite as library 491219910
527211047 https://github.com/simonw/sqlite-utils/pull/57#issuecomment-527211047 https://api.github.com/repos/simonw/sqlite-utils/issues/57 MDEyOklzc3VlQ29tbWVudDUyNzIxMTA0Nw== amjith 49260 2019-09-02T17:30:43Z 2019-09-02T17:30:43Z CONTRIBUTOR

I have merged the other PR (#56) into this one.

I have incorporated your suggestions. Cheers!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add triggers while enabling FTS 487987958
527209840 https://github.com/simonw/sqlite-utils/pull/56#issuecomment-527209840 https://api.github.com/repos/simonw/sqlite-utils/issues/56 MDEyOklzc3VlQ29tbWVudDUyNzIwOTg0MA== amjith 49260 2019-09-02T17:23:21Z 2019-09-02T17:23:21Z CONTRIBUTOR

I have updated the other PR with the changes from this one and added tests. I have also changed the escaping from double quotes to brackets.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Escape the table name in populate_fts and search. 487847945
510730200 https://github.com/simonw/datasette/issues/511#issuecomment-510730200 https://api.github.com/repos/simonw/datasette/issues/511 MDEyOklzc3VlQ29tbWVudDUxMDczMDIwMA== abdusco 3243482 2019-07-12T03:23:22Z 2019-07-12T03:23:22Z CONTRIBUTOR

@simonw yes it works fine on Windows, but test suite doesn't run properly, for that I had to use WSL

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Get Datasette working on Windows, including CI 456578474
509629331 https://github.com/simonw/datasette/pull/554#issuecomment-509629331 https://api.github.com/repos/simonw/datasette/issues/554 MDEyOklzc3VlQ29tbWVudDUwOTYyOTMzMQ== abdusco 3243482 2019-07-09T12:51:35Z 2019-07-09T12:51:35Z CONTRIBUTOR

I wanted to add a test for it too, but I've realized it's impossible to test a server process as we cannot get its exit code.

# tests/test_cli.py
def test_static_mounts_on_windows():
    if sys.platform != "win32":
        return
    runner = CliRunner()
    result = runner.invoke(
        cli, ["serve", "--static", r"s:C:\\"]
    )
    assert result.exit_code == 0
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Fix static mounts using relative paths and prevent traversal exploits 465728430
509618339 https://github.com/simonw/datasette/pull/554#issuecomment-509618339 https://api.github.com/repos/simonw/datasette/issues/554 MDEyOklzc3VlQ29tbWVudDUwOTYxODMzOQ== abdusco 3243482 2019-07-09T12:16:32Z 2019-07-09T12:16:32Z CONTRIBUTOR

I've also added another fix for using static mounts with absolute paths on Windows.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Fix static mounts using relative paths and prevent traversal exploits 465728430
509013413 https://github.com/simonw/datasette/issues/507#issuecomment-509013413 https://api.github.com/repos/simonw/datasette/issues/507 MDEyOklzc3VlQ29tbWVudDUwOTAxMzQxMw== psychemedia 82988 2019-07-07T16:31:57Z 2019-07-07T16:31:57Z CONTRIBUTOR

Chrome and Firefox both support headless screengrabs from command line, but I don't know how parameterised they can be?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Every datasette plugin on the ecosystem page should have a screenshot 455852801
504809397 https://github.com/simonw/datasette/issues/523#issuecomment-504809397 https://api.github.com/repos/simonw/datasette/issues/523 MDEyOklzc3VlQ29tbWVudDUwNDgwOTM5Nw== rixx 2657547 2019-06-24T01:38:14Z 2019-06-24T01:38:14Z CONTRIBUTOR

Ah, apologies – I had found and read those issues, but I was under the impression that they refered only to the filtered row count, not the unfiltered total row count.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Show total/unfiltered row count when filtering 459627549
504690927 https://github.com/simonw/datasette/issues/514#issuecomment-504690927 https://api.github.com/repos/simonw/datasette/issues/514 MDEyOklzc3VlQ29tbWVudDUwNDY5MDkyNw== russss 45057 2019-06-22T19:06:07Z 2019-06-22T19:06:07Z CONTRIBUTOR

I'd rather not turn this into a systemd support thread, but you're trying to execute the package directory there. Your datasette executable is probably at /home/chris/Env/datasette/bin/datasette.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Documentation with recommendations on running Datasette in production without using Docker 459397625
504684831 https://github.com/simonw/datasette/issues/514#issuecomment-504684831 https://api.github.com/repos/simonw/datasette/issues/514 MDEyOklzc3VlQ29tbWVudDUwNDY4NDgzMQ== russss 45057 2019-06-22T17:38:23Z 2019-06-22T17:38:23Z CONTRIBUTOR

WorkingDirectory=/path/to/data

@russss, Which directory does this represent?

It's the working directory (cwd) of the spawned process. In this case if you set it to the directory your data is in, you can use relative paths to the db (and metadata/templates/etc) in the ExecStart command.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Documentation with recommendations on running Datasette in production without using Docker 459397625
504663766 https://github.com/simonw/datasette/issues/514#issuecomment-504663766 https://api.github.com/repos/simonw/datasette/issues/514 MDEyOklzc3VlQ29tbWVudDUwNDY2Mzc2Ng== russss 45057 2019-06-22T12:57:59Z 2019-06-22T12:57:59Z CONTRIBUTOR

This example is useful to - I like how it has a Makefile that knows how to set up systemd: https://github.com/pikesley/Queube

I wasn't even aware it was possible to add a systemd service at an arbitrary path, but it seems a little messy to me.

Maybe worth noting that systemd does support per-user services which don't require root access. Cool but probably overkill for most people (especially when you're going to need root to listen on port 80 anyway, directly or via a reverse proxy).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Documentation with recommendations on running Datasette in production without using Docker 459397625
504662904 https://github.com/simonw/datasette/issues/514#issuecomment-504662904 https://api.github.com/repos/simonw/datasette/issues/514 MDEyOklzc3VlQ29tbWVudDUwNDY2MjkwNA== russss 45057 2019-06-22T12:45:21Z 2019-06-22T12:45:39Z CONTRIBUTOR

On most modern Linux distros, systemd is the easiest answer.

Example systemd unit file (save to /etc/systemd/system/datasette.service):

[Unit]
Description=Datasette
After=network.target

[Service]
Type=simple
User=<username>
WorkingDirectory=/path/to/data
ExecStart=/path/to/datasette serve -h 0.0.0.0 ./my.db
Restart=on-failure

[Install]
WantedBy=multi-user.target

Activate it with:

$ sudo systemctl daemon-reload
$ sudo systemctl enable datasette
$ sudo systemctl start datasette

Logs are best viewed using journalctl -u datasette -f.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Documentation with recommendations on running Datasette in production without using Docker 459397625
499923145 https://github.com/simonw/datasette/issues/394#issuecomment-499923145 https://api.github.com/repos/simonw/datasette/issues/394 MDEyOklzc3VlQ29tbWVudDQ5OTkyMzE0NQ== kevindkeogh 13896256 2019-06-07T15:10:57Z 2019-06-07T15:11:07Z CONTRIBUTOR

Putting this here in case anyone else encounters the same issue with nginx, I was able to resolve it by passing the header in the nginx proxy config (i.e., proxy_set_header Host $host).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
base_url configuration setting 396212021
499320973 https://github.com/simonw/datasette/issues/394#issuecomment-499320973 https://api.github.com/repos/simonw/datasette/issues/394 MDEyOklzc3VlQ29tbWVudDQ5OTMyMDk3Mw== kevindkeogh 13896256 2019-06-06T02:07:59Z 2019-06-06T02:07:59Z CONTRIBUTOR

Hey was this ever merged? Trying to run this behind nginx, and encountering this issue.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
base_url configuration setting 396212021
489342728 https://github.com/simonw/datasette/pull/450#issuecomment-489342728 https://api.github.com/repos/simonw/datasette/issues/450 MDEyOklzc3VlQ29tbWVudDQ4OTM0MjcyOA== russss 45057 2019-05-04T16:37:35Z 2019-05-04T16:37:35Z CONTRIBUTOR

For a bit more context: this fixes a crash with unsupported operand type(s) for +: 'int' and 'NoneType' on the index page for me.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Coalesce hidden table count to 0 440304714
489222223 https://github.com/simonw/datasette/issues/446#issuecomment-489222223 https://api.github.com/repos/simonw/datasette/issues/446 MDEyOklzc3VlQ29tbWVudDQ4OTIyMjIyMw== russss 45057 2019-05-03T20:01:19Z 2019-05-03T20:01:29Z CONTRIBUTOR

Also I have a slight preference against (ab)using __slots__ to enforce fields, although I have done it myself in the past. It would be possible to do this with __setattr__ instead, although that's an implementation detail and I'm not too fussed about it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Define mechanism for plugins to return structured data 440134714
489221481 https://github.com/simonw/datasette/issues/446#issuecomment-489221481 https://api.github.com/repos/simonw/datasette/issues/446 MDEyOklzc3VlQ29tbWVudDQ4OTIyMTQ4MQ== russss 45057 2019-05-03T19:58:31Z 2019-05-03T19:58:31Z CONTRIBUTOR

In this particular case I don't think there's an issue making all those required. However, I suspect we might have to allow optional values at some point - my preferred solution to russss/datasette-geo#2 would need one.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Define mechanism for plugins to return structured data 440134714
489163939 https://github.com/simonw/datasette/pull/434#issuecomment-489163939 https://api.github.com/repos/simonw/datasette/issues/434 MDEyOklzc3VlQ29tbWVudDQ4OTE2MzkzOQ== rprimet 10352819 2019-05-03T16:49:45Z 2019-05-03T16:50:03Z CONTRIBUTOR

The second time I ran the command I got an error:

ERROR: (gcloud.beta.run.deploy) Deployment endpoint was not found. Perhaps the
provided region was invalid. Set the run/region property to a valid region and
retry. Ex: gcloud config set run/region us-central1

Yes, I was able to reproduce this; I used to get prompted for a run region interactively by the gcloud tool before, but maybe this is changing? (the documentation now assumes run/region is set).

Not sure which course of action is best: making datasette ensure that run/region is set beforehand or wait a bit until the gcloud CLI stabilizes?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"datasette publish cloudrun" command to publish to Google Cloud Run 434321685
489060765 https://github.com/simonw/datasette/issues/419#issuecomment-489060765 https://api.github.com/repos/simonw/datasette/issues/419 MDEyOklzc3VlQ29tbWVudDQ4OTA2MDc2NQ== russss 45057 2019-05-03T11:07:42Z 2019-05-03T11:07:42Z CONTRIBUTOR

Are you planning on removing inspect entirely?

I didn't spot this work before I started on datasette-geo, but ironically I think it has a use case which really needs the inspect functionality (or some replacement).

Datasette-geo uses it to store the bounding box of all the geographic features in the table. This is needed when rendering the map because it avoids having to send loads of tile requests for areas which are empty.

Even with relatively small datasets, calculating the bounding box seems to take around 5 seconds, so I don't think it's really feasible to do this on page load.

One possible fix would be to do this on startup, and then in a thread which watches the database for changes.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Default to opening files in mutable mode, special option for immutable files 421551434
488595724 https://github.com/simonw/datasette/pull/432#issuecomment-488595724 https://api.github.com/repos/simonw/datasette/issues/432 MDEyOklzc3VlQ29tbWVudDQ4ODU5NTcyNA== russss 45057 2019-05-02T08:50:53Z 2019-05-02T08:50:53Z CONTRIBUTOR

Can I pull those needs out of the Facet class somehow?

I was thinking that it might be handy for datasette to have a request object which wraps the Sanic Request. This could include the datasette-specific querystring decoding and the special_args parsing from TableView.data.

This would mean that we could expose the request object to plugin hooks without coupling them to Sanic.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Refactor facets to a class and new plugin, refs #427 432893491
488247617 https://github.com/simonw/datasette/pull/441#issuecomment-488247617 https://api.github.com/repos/simonw/datasette/issues/441 MDEyOklzc3VlQ29tbWVudDQ4ODI0NzYxNw== russss 45057 2019-05-01T09:57:50Z 2019-05-01T09:57:50Z CONTRIBUTOR

Just for the record, this PR is now finished and ready to merge from my perspective.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add register_output_renderer hook 438437973
487859345 https://github.com/simonw/datasette/pull/439#issuecomment-487859345 https://api.github.com/repos/simonw/datasette/issues/439 MDEyOklzc3VlQ29tbWVudDQ4Nzg1OTM0NQ== russss 45057 2019-04-30T08:21:19Z 2019-04-30T08:21:19Z CONTRIBUTOR

I think the best approach to this is to pass through the view_name parameter I added in #441. It's then simple enough for me to add .geojson to the URL in JS - I don't need the pkey.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
[WIP] Add primary key to the extra_body_script hook arguments 438240541
487748271 https://github.com/simonw/datasette/pull/441#issuecomment-487748271 https://api.github.com/repos/simonw/datasette/issues/441 MDEyOklzc3VlQ29tbWVudDQ4Nzc0ODI3MQ== russss 45057 2019-04-29T21:20:17Z 2019-04-29T21:20:17Z CONTRIBUTOR

Also I just pushed a change to add registered output renderers to the templates:

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add register_output_renderer hook 438437973
487735247 https://github.com/simonw/datasette/pull/441#issuecomment-487735247 https://api.github.com/repos/simonw/datasette/issues/441 MDEyOklzc3VlQ29tbWVudDQ4NzczNTI0Nw== russss 45057 2019-04-29T20:39:43Z 2019-04-29T20:39:43Z CONTRIBUTOR

I updated the hook to pass the datasette object through now.

You can see the working GeoJSON render function here - the hook function is here.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add register_output_renderer hook 438437973
487724539 https://github.com/simonw/datasette/pull/441#issuecomment-487724539 https://api.github.com/repos/simonw/datasette/issues/441 MDEyOklzc3VlQ29tbWVudDQ4NzcyNDUzOQ== russss 45057 2019-04-29T20:08:32Z 2019-04-29T20:08:32Z CONTRIBUTOR

I also just realised that I should be passing the datasette object into the hook function...as I just found I need it. So hold off merging until I've fixed that.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add register_output_renderer hook 438437973
487723476 https://github.com/simonw/datasette/pull/441#issuecomment-487723476 https://api.github.com/repos/simonw/datasette/issues/441 MDEyOklzc3VlQ29tbWVudDQ4NzcyMzQ3Ng== russss 45057 2019-04-29T20:05:23Z 2019-04-29T20:05:23Z CONTRIBUTOR

This is the minimal example (I also included it in the docs):

from datasette import hookimpl

def render_test(args, data, view_name):
    return {
       'body': 'Hello World',
       'content_type': 'text/plain'
    }

@hookimpl
def register_output_renderer():
    return {
        'extension': 'test',
        'callback': render_test
    }

I'm working on the GeoJSON one now and it should be ready soon. (I forgot I was going to run into the same problem as before - that Spatialite's stupid binary format isn't WKB and I have no way of altering the query to change that - but I've just managed to write some code to rearrange the bytes from Spatialite blob-geometry into WKB...)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add register_output_renderer hook 438437973
487692377 https://github.com/simonw/datasette/pull/424#issuecomment-487692377 https://api.github.com/repos/simonw/datasette/issues/424 MDEyOklzc3VlQ29tbWVudDQ4NzY5MjM3Nw== russss 45057 2019-04-29T18:30:46Z 2019-04-29T18:30:46Z CONTRIBUTOR

Actually no, I ended up not using the inspected column types in my plugin, and the binary column issue can be solved a lot more simply, so I'll close this.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Column types in inspected metadata 427429265
487689477 https://github.com/simonw/datasette/pull/424#issuecomment-487689477 https://api.github.com/repos/simonw/datasette/issues/424 MDEyOklzc3VlQ29tbWVudDQ4NzY4OTQ3Nw== russss 45057 2019-04-29T18:22:40Z 2019-04-29T18:22:40Z CONTRIBUTOR

This is pretty conflicty because I forgot how to use git fetch. If you're interested in merging this I'll rewrite it against an actual modern checkout...

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Column types in inspected metadata 427429265
487686655 https://github.com/simonw/datasette/pull/441#issuecomment-487686655 https://api.github.com/repos/simonw/datasette/issues/441 MDEyOklzc3VlQ29tbWVudDQ4NzY4NjY1NQ== russss 45057 2019-04-29T18:14:25Z 2019-04-29T18:14:25Z CONTRIBUTOR

Subsidiary note which I forgot in the commit message:

I've decided to give each view a short string name to aid in differentiating which view a hook is being called from. Since hooks are functions and not subclasses, and can get called from different places in the URL hierarchy, it's sometimes difficult to distinguish what data you're actually operating on. I think this will come in handy for other hooks as well.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add register_output_renderer hook 438437973
487542486 https://github.com/simonw/datasette/pull/439#issuecomment-487542486 https://api.github.com/repos/simonw/datasette/issues/439 MDEyOklzc3VlQ29tbWVudDQ4NzU0MjQ4Ng== russss 45057 2019-04-29T11:20:30Z 2019-04-29T11:20:30Z CONTRIBUTOR

Actually I think this is not the whole story because of the rowid issue. I'm going to think about this one a bit more.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
[WIP] Add primary key to the extra_body_script hook arguments 438240541
487537452 https://github.com/simonw/datasette/pull/437#issuecomment-487537452 https://api.github.com/repos/simonw/datasette/issues/437 MDEyOklzc3VlQ29tbWVudDQ4NzUzNzQ1Mg== russss 45057 2019-04-29T10:58:49Z 2019-04-29T10:58:49Z CONTRIBUTOR

I've just spotted that this implements #215.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add inspect and prepare_sanic hooks 438048318
483202658 https://github.com/simonw/datasette/issues/429#issuecomment-483202658 https://api.github.com/repos/simonw/datasette/issues/429 MDEyOklzc3VlQ29tbWVudDQ4MzIwMjY1OA== psychemedia 82988 2019-04-15T10:48:01Z 2019-04-15T10:48:01Z CONTRIBUTOR

Minor UI observation:

_where= renders a [remove] link whereas _facet= gets a cross to remove it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_where=sql-fragment parameter for table views 432636432
483017176 https://github.com/simonw/datasette/issues/431#issuecomment-483017176 https://api.github.com/repos/simonw/datasette/issues/431 MDEyOklzc3VlQ29tbWVudDQ4MzAxNzE3Ng== psychemedia 82988 2019-04-14T16:58:37Z 2019-04-14T16:58:37Z CONTRIBUTOR

Hmm... nope... I see an updated timestamp from ls -al on the db but no reload?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Datasette doesn't reload when database file changes 432870248
474282321 https://github.com/simonw/datasette/issues/412#issuecomment-474282321 https://api.github.com/repos/simonw/datasette/issues/412 MDEyOklzc3VlQ29tbWVudDQ3NDI4MjMyMQ== psychemedia 82988 2019-03-19T10:09:46Z 2019-03-19T10:09:46Z CONTRIBUTOR

Does this also relate to https://github.com/simonw/datasette/issues/283 and the ability to ATTACH DATABASE?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Linked Data(sette) 411257981
474280581 https://github.com/simonw/datasette/issues/417#issuecomment-474280581 https://api.github.com/repos/simonw/datasette/issues/417 MDEyOklzc3VlQ29tbWVudDQ3NDI4MDU4MQ== psychemedia 82988 2019-03-19T10:06:42Z 2019-03-19T10:06:42Z CONTRIBUTOR

This would be really interesting but several possibilities in use arise, I think?

For example:

  • I put a new CSV file into the import dir and a new table is created therefrom
  • I put a CSV file into the import dir that replaces a previous file / table of the same name as a pre-existing table (eg files that contain monthly data in year to date). The data may also patch previous months, so a full replace / DROP on the original table may well be in order.
  • I put a CSV file into the import dir that updates a table of the same name as a pre-existing table (eg files that contain last month's data)

CSV files may also have messy names compared to the table you want. Or for an update CSV, may have the form MYTABLENAME-February2019.csv etc

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Datasette Library 421546944
459915995 https://github.com/simonw/datasette/issues/160#issuecomment-459915995 https://api.github.com/repos/simonw/datasette/issues/160 MDEyOklzc3VlQ29tbWVudDQ1OTkxNTk5NQ== psychemedia 82988 2019-02-02T00:43:16Z 2019-02-02T00:58:20Z CONTRIBUTOR

Do you have any simple working examples of how to use --static? Inspection of default served files suggests locations such as http://example.com/-/static/app.css?0e06ee.

If datasette is being proxied to http://example.com/foo/datasette, what form should arguments to --static take so that static files are correctly referenced?

Use case is here: https://github.com/psychemedia/jupyterserverproxy-datasette-demo Trying to do a really simple datasette demo in MyBinder using jupyter-server-proxy.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability to bundle and serve additional static files 278208011
436042445 https://github.com/simonw/datasette/issues/370#issuecomment-436042445 https://api.github.com/repos/simonw/datasette/issues/370 MDEyOklzc3VlQ29tbWVudDQzNjA0MjQ0NQ== psychemedia 82988 2018-11-05T21:30:42Z 2018-11-05T21:31:48Z CONTRIBUTOR

Another route would be something like creating a datasette IPython magic for notebooks to take a dataframe and easily render it as a datasette. You'd need to run the app in the background rather than block execution in the notebook. Related to that, or to publishing a dataframe in notebook cell for use in other cells in a non-blocking way, there may be cribs in something like https://github.com/micahscopes/nbmultitask .

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Integration with JupyterLab 377155320
436037692 https://github.com/simonw/datasette/issues/370#issuecomment-436037692 https://api.github.com/repos/simonw/datasette/issues/370 MDEyOklzc3VlQ29tbWVudDQzNjAzNzY5Mg== psychemedia 82988 2018-11-05T21:15:47Z 2018-11-05T21:18:37Z CONTRIBUTOR

In terms of integration with pandas, I was pondering two different ways datasette/csvs_to_sqlite integration may work:

  • like pandasql, to provide a SQL query layer either by a direct connection to the sqlite db or via datasette API;
  • as an improvement of pandas.to_sql(), which is a bit ropey (e.g. pandas.to_sql_from_csvs(), routing the dataframe to sqlite via csvs_tosqlite rather than the dodgy mapping that pandas supports).

The pandas.publish_* idea could be quite interesting though... Would it be useful/fruitful to think about publish_ as a complement to pandas.to_?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Integration with JupyterLab 377155320
435862009 https://github.com/simonw/datasette/issues/371#issuecomment-435862009 https://api.github.com/repos/simonw/datasette/issues/371 MDEyOklzc3VlQ29tbWVudDQzNTg2MjAwOQ== psychemedia 82988 2018-11-05T12:48:35Z 2018-11-05T12:48:35Z CONTRIBUTOR

I think you need to register a domain name you own separately in order to get a non-IP address address? https://www.digitalocean.com/docs/networking/dns/

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette publish digitalocean plugin 377156339
422915450 https://github.com/simonw/datasette/issues/329#issuecomment-422915450 https://api.github.com/repos/simonw/datasette/issues/329 MDEyOklzc3VlQ29tbWVudDQyMjkxNTQ1MA== jaywgraves 418191 2018-09-19T18:45:02Z 2018-09-20T10:50:50Z CONTRIBUTOR

That works for me. Was able to pull the public image and no errors on my canned query. (~although a small rendering bug. I'll create an issue and if I have time today, a PR to fix~ this turned out to be my error.)
Thanks for the quick response!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Travis should push tagged images to Docker Hub for each release 336465018
422821483 https://github.com/simonw/datasette/issues/329#issuecomment-422821483 https://api.github.com/repos/simonw/datasette/issues/329 MDEyOklzc3VlQ29tbWVudDQyMjgyMTQ4Mw== jaywgraves 418191 2018-09-19T14:17:42Z 2018-09-19T14:17:42Z CONTRIBUTOR

I'm using the docker image (0.23.2) and notice some differences/bugs between the docs and the published version with canned queries. (submitted a tiny doc fix also)

I was able to build the docker container locally using master and I'm using that for now.
Would it be possible to manually push 0.24 to DockerHub until the TravisCI stuff is fixed?

I would like to run this in our Kubernetes cluster but don't want to publish a version in our internal registry if I don't have to.
Thanks!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Travis should push tagged images to Docker Hub for each release 336465018
405026800 https://github.com/simonw/datasette/issues/294#issuecomment-405026800 https://api.github.com/repos/simonw/datasette/issues/294 MDEyOklzc3VlQ29tbWVudDQwNTAyNjgwMA== russss 45057 2018-07-14T14:24:31Z 2018-07-14T14:24:31Z CONTRIBUTOR

I had a quick look at this in relation to #343 and I feel like it might be worth modelling the inspected table metadata internally as an object rather than a dict. (We'd still have to serialise it back to JSON.)

There are a few places where we rely on the structure of this metadata dict for various reasons, including in templates (and potentially also in user templates). It would be nice to have a reasonably well defined API for accessing metadata internally so that it's clearer what we're breaking.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
inspect should record column types 327365110
405026441 https://github.com/simonw/datasette/issues/343#issuecomment-405026441 https://api.github.com/repos/simonw/datasette/issues/343 MDEyOklzc3VlQ29tbWVudDQwNTAyNjQ0MQ== russss 45057 2018-07-14T14:17:14Z 2018-07-14T14:17:14Z CONTRIBUTOR

This probably depends on #294.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Render boolean fields better by default 341228846
405022335 https://github.com/simonw/datasette/issues/344#issuecomment-405022335 https://api.github.com/repos/simonw/datasette/issues/344 MDEyOklzc3VlQ29tbWVudDQwNTAyMjMzNQ== russss 45057 2018-07-14T13:00:48Z 2018-07-14T13:00:48Z CONTRIBUTOR

Looks like this was a red herring actually, and heroku had a blip when I was testing it...

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette publish heroku fails without name provided 341229113
401312981 https://github.com/simonw/datasette/issues/276#issuecomment-401312981 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDQwMTMxMjk4MQ== russss 45057 2018-06-29T10:14:54Z 2018-06-29T10:14:54Z CONTRIBUTOR

@RusSs Different map projections can presumably be handled on the client side using a leaflet plugin to transform the geometry (eg kartena/Proj4Leaflet) although the leaflet side would need to detect or be informed of the original projection?

Well, as @simonw mentioned, GeoJSON only supports WGS84, and GeoJSON (and/or TopoJSON) is the standard we probably want to aim for. On-the-fly reprojection in spatialite is not an issue anyway, and in general I think you want to be serving stuff to web maps in WGS84 or Web Mercator.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838
401310732 https://github.com/simonw/datasette/issues/276#issuecomment-401310732 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDQwMTMxMDczMg== psychemedia 82988 2018-06-29T10:05:04Z 2018-06-29T10:07:25Z CONTRIBUTOR

@russs Different map projections can presumably be handled on the client side using a leaflet plugin to transform the geometry (eg kartena/Proj4Leaflet) although the leaflet side would need to detect or be informed of the original projection?

Another possibility would be to provide an easy way/guidance for users to create an FK'd table containing the WGS84 projection of a non-WGS84 geometry in the original/principle table? This could then as a proxy for serving GeoJSON to the leaflet map?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838
393106520 https://github.com/simonw/datasette/issues/276#issuecomment-393106520 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDM5MzEwNjUyMA== russss 45057 2018-05-30T10:09:25Z 2018-05-30T10:09:25Z CONTRIBUTOR

I don't think it's unreasonable to only support spatialite geometries in a coordinate reference system which is at least transformable to WGS84. It would be nice to support different CRSes in the database so conversion to spatialite from the source data is lossless.

I think the working CRS for datasette should be WGS84 though (leaflet requires it, for example) - it's just a case of calling ST_Transform(geom, 4326) on the column while we're loading the data.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838
392825746 https://github.com/simonw/datasette/issues/276#issuecomment-392825746 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDM5MjgyNTc0Ng== russss 45057 2018-05-29T15:42:53Z 2018-05-29T15:42:53Z CONTRIBUTOR

I haven't had time to look further into this, but if doing this as a plugin results in useful hooks then I think we should do it that way. We could always require the plugin as a standard dependency.

I think this is going to result in quite a bit of refactoring anyway so it's a good time to add hooks regardless.

On the other hand, if we have to add lots of specialist hooks for it then maybe it's worth integrating into the core.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838
391505930 https://github.com/simonw/datasette/issues/276#issuecomment-391505930 https://api.github.com/repos/simonw/datasette/issues/276 MDEyOklzc3VlQ29tbWVudDM5MTUwNTkzMA== russss 45057 2018-05-23T21:41:37Z 2018-05-23T21:41:37Z CONTRIBUTOR

I'm not keen on anything that modifies the SQLite file itself on startup

Ah I didn't mean that - I meant altering the SELECT query to fetch the data so that it ran a spatialite function to transform that specific column.

I think that's less useful as a general-purpose plugin hook though, and it's not that hard to parse the WKB in Python (my default approach would be to use shapely, which is great, but geomet looks like an interesting pure-python alternative).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle spatialite geometry columns better 324835838
391355030 https://github.com/simonw/datasette/pull/280#issuecomment-391355030 https://api.github.com/repos/simonw/datasette/issues/280 MDEyOklzc3VlQ29tbWVudDM5MTM1NTAzMA== r4vi 565628 2018-05-23T13:53:27Z 2018-05-23T15:22:45Z CONTRIBUTOR

No objections;
It's good to go @simonw

On Wed, 23 May 2018, 14:51 Simon Willison, notifications@github.com wrote:

@r4vi https://github.com/r4vi any objections to me merging this?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/simonw/datasette/pull/280#issuecomment-391354237, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAihfM_2DN5WR2mkO-VK6ozDmkUQ4IMjks5t1WlcgaJpZM4UI_2m
.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Build Dockerfile with recent Sqlite + Spatialite 325373747
391290271 https://github.com/simonw/datasette/pull/280#issuecomment-391290271 https://api.github.com/repos/simonw/datasette/issues/280 MDEyOklzc3VlQ29tbWVudDM5MTI5MDI3MQ== r4vi 565628 2018-05-23T09:53:38Z 2018-05-23T09:53:38Z CONTRIBUTOR

Running:

docker run -p 8001:8001 -v `pwd`:/mnt datasette \
    datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db \
        --load-extension=/usr/local/lib/mod_spatialite.so

is now returning FTS5 enabled in the versions output:

{
    "datasette": {
        "version": "0.22"
    },
    "python": {
        "full": "3.6.5 (default, May  5 2018, 03:07:21) \n[GCC 6.3.0 20170516]",
        "version": "3.6.5"
    },
    "sqlite": {
        "extensions": {
            "json1": null,
            "spatialite": "4.4.0-RC0"
        },
        "fts_versions": [
            "FTS5",
            "FTS4",
            "FTS3"
        ],
        "version": "3.23.1"
    }
}

The old query didn't work because specifying (t TEXT) caused an error

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Build Dockerfile with recent Sqlite + Spatialite 325373747
391141391 https://github.com/simonw/datasette/pull/280#issuecomment-391141391 https://api.github.com/repos/simonw/datasette/issues/280 MDEyOklzc3VlQ29tbWVudDM5MTE0MTM5MQ== r4vi 565628 2018-05-22T21:08:39Z 2018-05-22T21:08:39Z CONTRIBUTOR

I'm going to clean this up for consistency tomorrow morning so hold off
merging until then please

On Tue, May 22, 2018 at 6:34 PM, Simon Willison notifications@github.com
wrote:

Yeah let's try this without pysqlite3 and see if we still get the correct
version.


You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/simonw/datasette/pull/280#issuecomment-391076458, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAihfMI-H6CBt-Py0xdBbH2xDK0KsjT2ks5t1EwYgaJpZM4UI_2m
.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Build Dockerfile with recent Sqlite + Spatialite 325373747
391077700 https://github.com/simonw/datasette/pull/279#issuecomment-391077700 https://api.github.com/repos/simonw/datasette/issues/279 MDEyOklzc3VlQ29tbWVudDM5MTA3NzcwMA== rgieseke 198537 2018-05-22T17:38:17Z 2018-05-22T17:38:17Z CONTRIBUTOR

Alright, that should work now -- let me know if you would prefer any different behaviour.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add version number support with Versioneer 325352370
391073267 https://github.com/simonw/datasette/pull/279#issuecomment-391073267 https://api.github.com/repos/simonw/datasette/issues/279 MDEyOklzc3VlQ29tbWVudDM5MTA3MzI2Nw== rgieseke 198537 2018-05-22T17:24:16Z 2018-05-22T17:24:16Z CONTRIBUTOR

Sorry, just realised you rely on version being a module ...

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add version number support with Versioneer 325352370
391073009 https://github.com/simonw/datasette/pull/279#issuecomment-391073009 https://api.github.com/repos/simonw/datasette/issues/279 MDEyOklzc3VlQ29tbWVudDM5MTA3MzAwOQ== rgieseke 198537 2018-05-22T17:23:26Z 2018-05-22T17:23:26Z CONTRIBUTOR

I think I prefer the aesthetics of just "0.22" for the version string if it's a tagged release with no additional changes - does that work?

Yes! That's the default versioneer behaviour.

I'd like to continue to provide a tuple that can be imported from the version.py module as well, as seen here:

Should work now, it can be a two (for a tagged version), three or four items tuple.

In [2]: datasette.__version__
Out[2]: '0.12+292.ga70c2a8.dirty'

In [3]: datasette.__version_info__
Out[3]: ('0', '12+292', 'ga70c2a8', 'dirty')
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add version number support with Versioneer 325352370
391059008 https://github.com/simonw/datasette/pull/280#issuecomment-391059008 https://api.github.com/repos/simonw/datasette/issues/280 MDEyOklzc3VlQ29tbWVudDM5MTA1OTAwOA== r4vi 565628 2018-05-22T16:40:27Z 2018-05-22T16:40:27Z CONTRIBUTOR
>>> import sqlite3
>>> sqlite3.sqlite_version
'3.23.1'
>>> 

running the above in the container seems to show 3.23.1 too so maybe we don't need pysqlite3 at all?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Build Dockerfile with recent Sqlite + Spatialite 325373747

Next page

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);
Powered by Datasette · Query took 302.883ms · About: github-to-sqlite