{"rowid": 338523957, "body": "I also need to solve for weird primary keys. If it\u2019s a single integer or a single char field that\u2019s easy. But what if it is a compound key with more than one chat field? What delimiter can I use that will definitely be safe?\r\n\r\nLet\u2019s say I use hyphen. Now I need to find a durable encoding for any hyphens that might exist in the key fields themselves.\r\n\r\nHow about I use URLencoding for every non-alpha-numeric character? That will turn hyphens into (I think) %2D. It should also solve for unicode characters, but it means the vast majority of keys (integers) will display neatly, including a compound key of eg 5678-345\r\n\r\n", "issue_comments_fts": 16393, "rank": null}
{"rowid": 338524454, "body": "Table rendering logic needs to detect the primary key field and turn it into a hyperlink. If there is a compound primary key it should add an extra column at the start of the table which displays the compound key as a link", "issue_comments_fts": 16393, "rank": null}
{"rowid": 338524857, "body": "https://stackoverflow.com/a/14468878/6083\r\n\r\nLooks like I should order by compound primary key and implement cursor-based pagination.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 338526148, "body": "https://github.com/ahupp/python-magic/blob/master/README.md", "issue_comments_fts": 16393, "rank": null}
{"rowid": 338530389, "body": "This means I need a good solution for these compile time options while running in development mode \r\n ", "issue_comments_fts": 16393, "rank": null}
{"rowid": 338530480, "body": " How about when the service starts up it checks for a compile.json file and, if it is missing, creates it using the same code we run at compile time normally ", "issue_comments_fts": 16393, "rank": null}
{"rowid": 338530704, "body": "Needed by https://github.com/simonw/stateless-datasets/issues/4#issuecomment-338530389", "issue_comments_fts": 16393, "rank": null}
{"rowid": 338531827, "body": "Many of the applications I want to implement with this would benefit from having permanent real URLs.\r\n\r\nSo let\u2019s have both. The sha1 urls will serve far future cache headers (and an etag derived from their path). The non sha1 URLs will serve 302 uncached redirects to the sha1 locations.\r\n\r\nWe will have a setting that lets people opt out of this behavior.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 338697223, "body": "Now returning this:\r\n\r\n {\r\n \"error\": \"attempt to write a readonly database\",\r\n \"ok\": false\r\n }\r\n", "issue_comments_fts": 16393, "rank": null}
{"rowid": 338768860, "body": "I could use the table-reflow mechanism demonstrated here: http://demos.jquerymobile.com/1.4.3/table-reflow/", "issue_comments_fts": 16393, "rank": null}
{"rowid": 338769538, "body": "Maybe this should be handled by views instead?\r\n\r\nhttps://stateless-datasets-wreplxalgu.now.sh/ lists some views\r\n\r\nhttps://stateless-datasets-wreplxalgu.now.sh/?sql=select%20*%20from%20%22Order%20Subtotals%22 is an example showing the content of a view.\r\n\r\nWhat would the URL to views be? I don't think a view can share a name with a table, so the same URL scheme could work for both.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 338789734, "body": "URL design:\r\n\r\n /database/table.json - redirects to /database-6753f4a/table.json\r\n\r\nSo we always redirect to the version with the truncated hash in the URL.\r\n", "issue_comments_fts": 16393, "rank": null}
{"rowid": 338797522, "body": "https://stackoverflow.com/a/18134919/6083 is a good answer about how many characters of the hash are needed to be unique. I say we default to 7 characters, like git does - but allow extras to be configured.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 338799438, "body": "Can I take advantage of HTTP/2 so even if you get redirected I start serving you the correct resource straight away?", "issue_comments_fts": 16393, "rank": null}
{"rowid": 338804173, "body": "Looks like the easiest way to implement HTTP/2 server push today is to run behind Cloudflare and use this:\r\n\r\n Link: ; rel=preload; as=script\r\n\r\nhttps://blog.cloudflare.com/announcing-support-for-http-2-server-push-2/\r\n\r\nHere's the W3C draft: https://w3c.github.io/preload/\r\n\r\nFrom https://w3c.github.io/preload/#as-attribute it looks like I should use `as=fetch` if the content is intended for consumption by fetch() or XMLHTTPRequest.\r\n\r\nUnclear if I should throw `as=fetch crossorigin` in there. Need to experiment on that.\r\n", "issue_comments_fts": 16393, "rank": null}
{"rowid": 338806718, "body": "Here's what the homepage of cloudflare.com does (with newlines added within the link header for clarity):\r\n\r\n $ curl -i 'https://www.cloudflare.com/' \r\n HTTP/1.1 200 OK\r\n Date: Mon, 23 Oct 2017 21:45:58 GMT\r\n Content-Type: text/html; charset=utf-8\r\n Transfer-Encoding: chunked\r\n Connection: keep-alive\r\n link:\r\n ; rel=preload; as=style,\r\n ; rel=preload; as=style,\r\n ; rel=preload,\r\n ; rel=preload,\r\n ; rel=preload; as=video,\r\n ; rel=preload; as=video,\r\n ; rel=preload; as=video,\r\n ; rel=preload; as=video,\r\n ; rel=preload; as=video,\r\n ; rel=preload; as=image\r\n\r\nThe original header looked like this:\r\n\r\n link: ; rel=preload; as=style, ; rel=preload; as=style, ; rel=preload, ; rel=preload, ; rel=preload; as=video, ; rel=preload; as=video, ; rel=preload; as=video, ; rel=preload; as=video, ; rel=preload; as=video, ; rel=preload; as=image\r\n\r\n", "issue_comments_fts": 16393, "rank": null}
{"rowid": 338834213, "body": "If I can\u2019t setect a primary key, I won\u2019t provide a URL for those records", "issue_comments_fts": 16393, "rank": null}
{"rowid": 338852971, "body": "I'm not going to bother with this.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 338853083, "body": "Fixed in 9d219140694551453bfa528e0624919eb065f9d6", "issue_comments_fts": 16393, "rank": null}
{"rowid": 338854988, "body": " /database-name/table-name?name__contains=simon&sort=id+desc\r\n\r\nNote that if there's a column called \"sort\" you can still do sort__exact=blah\r\n\r\n", "issue_comments_fts": 16393, "rank": null}
{"rowid": 338857568, "body": "I can find the primary keys using:\r\n\r\n PRAGMA table_info(myTable)\r\n", "issue_comments_fts": 16393, "rank": null}
{"rowid": 338859620, "body": "I\u2019m going to implement everything in https://docs.djangoproject.com/en/1.11/ref/models/querysets/#field-lookups with the exception of range and the various date ones.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 338859709, "body": "I\u2019m going to need to write unit tests for this, is this depends on #9", "issue_comments_fts": 16393, "rank": null}
{"rowid": 338861511, "body": "Some tables won't have primary keys, in which case I won't generate pages for individual records.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 338863155, "body": "I\u2019m going to use py.test and start with all tests in a single tests.py module", "issue_comments_fts": 16393, "rank": null}
{"rowid": 338872286, "body": "I'm going to use `,` as the separator between elements of a compound primary key. If those elements themselves include a comma I will use `%2C` in its place.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 338882110, "body": "Well, I've started it at least.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 338882207, "body": "Next step: generate links to these.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 339003850, "body": "As of b46e370ee6126aa2fa85cf789a31da38aed98496 this is done.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 339019873, "body": "Here's what I've got now:\r\n\r\n![\"localhost_8006_northwind-40d049b_categories_json\"](\"https://user-images.githubusercontent.com/9599/31950748-1bc85f3e-b891-11e7-8205-d89da6129f4b.png\")
\r\n", "issue_comments_fts": 16393, "rank": null}
{"rowid": 339027711, "body": "I have code to detect primary keys on tables... but what should I do for tables that lack primary keys? How should I even sort them?", "issue_comments_fts": 16393, "rank": null}
{"rowid": 339028979, "body": "Looks like I can use the SQLite specific \u201crowid\u201d in that case. It isn\u2019t guaranteed to stay consistent across a VACUUM but that\u2019s ok because we are immutable anyway.\r\n\r\n https://www.sqlite.org/lang_createtable.html#rowid", "issue_comments_fts": 16393, "rank": null}
{"rowid": 339138809, "body": "May as well support most of https://sqlite.org/lang_expr.html", "issue_comments_fts": 16393, "rank": null}
{"rowid": 339186887, "body": "Still to do:\r\n\r\n- [x] `gt`, `gte`, `lt`, `lte`\r\n- [x] `like`\r\n- [x] `glob`\r\n", "issue_comments_fts": 16393, "rank": null}
{"rowid": 339210353, "body": "I'm going to call this one done for the moment. The date filters can go in a stretch goal.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 339366612, "body": "I had to manually set the content disposition header:\r\n\r\n return await response.file_stream(\r\n filepath, headers={\r\n 'Content-Disposition': 'attachment; filename=\"{}\"'.format(ilepath)\r\n }\r\n )\r\n\r\nIn the next release of Sanic I can just use the filename= argument instead:\r\n\r\nhttps://github.com/channelcat/sanic/commit/07e95dba4f5983afc1e673df14bdd278817288aa", "issue_comments_fts": 16393, "rank": null}
{"rowid": 339382054, "body": "Could this be as simple as using the iterative JSON encoder and adding a yield statement in between each chunk?", "issue_comments_fts": 16393, "rank": null}
{"rowid": 339388215, "body": "First experiment: hook up an iterative CSV dump (just because that\u2019s a tiny bit easier to get started with than iterative a JSON). Have it execute a big select statement and then iterate through the result set 100 rows at a time using sqite fetchmany() - also have it async sleep for a second in between each batch of 100.\r\n\r\nCan this work without needing python threads? ", "issue_comments_fts": 16393, "rank": null}
{"rowid": 339388771, "body": "If this does work, I need to figure it what to do about the HTML view. ASsuming I can iteratively produce JSON and CSV, what to do about HTML? One option: render the first 500 rows as HTML, then hand off to an infinite scroll experience that iteratively loads more rows as JSON.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 339389105, "body": "The gold standard here is to be able to serve up increasingly large datasets without blocking the event loop and while using a sustainable amount of RAM", "issue_comments_fts": 16393, "rank": null}
{"rowid": 339389328, "body": "Ideally we can get some serious gains from the fact that our database file is opened with the immutable option.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 339395551, "body": "Simplest implementation will be to create a temporary directory somewhere, copy in a Dockerfile and the databases and run \u201cnow\u201d in it.\r\n\r\nIdeally I can use symlinks rather than copying potentially large database files around.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 339406634, "body": "It certainly looks like some of the stuff in https://sqlite.org/pragma.html could be used to screw around with things. Example: `PRAGMA case_sensitive_like = 1` - would that affect future queries?", "issue_comments_fts": 16393, "rank": null}
{"rowid": 339413825, "body": "Could I use https://sqlparse.readthedocs.io/en/latest/ to parse incoming statements and ensure they are pure SELECTs? Would that prevent people from using a compound SELECT statement to trigger an evil PRAGMA of some sort?", "issue_comments_fts": 16393, "rank": null}
{"rowid": 339420462, "body": "https://sitesforprofit.com/responsive-table-plugins-and-patterns has some useful links.\r\n\r\nI really like the pattern from https://css-tricks.com/responsive-data-tables/\r\n\r\n /* \r\n Max width before this PARTICULAR table gets nasty\r\n This query will take effect for any screen smaller than 760px\r\n and also iPads specifically.\r\n */\r\n @media \r\n only screen and (max-width: 760px),\r\n (min-device-width: 768px) and (max-device-width: 1024px) {\r\n\r\n /* Force table to not be like tables anymore */\r\n table, thead, tbody, th, td, tr { \r\n display: block; \r\n }\r\n \r\n /* Hide table headers (but not display: none;, for accessibility) */\r\n thead tr { \r\n position: absolute;\r\n top: -9999px;\r\n left: -9999px;\r\n }\r\n \r\n tr { border: 1px solid #ccc; }\r\n \r\n td { \r\n /* Behave like a \"row\" */\r\n border: none;\r\n border-bottom: 1px solid #eee; \r\n position: relative;\r\n padding-left: 50%; \r\n }\r\n \r\n td:before { \r\n /* Now like a table header */\r\n position: absolute;\r\n /* Top/left values mimic padding */\r\n top: 6px;\r\n left: 6px;\r\n width: 45%; \r\n padding-right: 10px; \r\n white-space: nowrap;\r\n }\r\n \r\n /*\r\n Label the data\r\n */\r\n td:nth-of-type(1):before { content: \"First Name\"; }\r\n td:nth-of-type(2):before { content: \"Last Name\"; }\r\n td:nth-of-type(3):before { content: \"Job Title\"; }\r\n td:nth-of-type(4):before { content: \"Favorite Color\"; }\r\n td:nth-of-type(5):before { content: \"Wars of Trek?\"; }\r\n td:nth-of-type(6):before { content: \"Porn Name\"; }\r\n td:nth-of-type(7):before { content: \"Date of Birth\"; }\r\n td:nth-of-type(8):before { content: \"Dream Vacation City\"; }\r\n td:nth-of-type(9):before { content: \"GPA\"; }\r\n td:nth-of-type(10):before { content: \"Arbitrary Data\"; }\r\n }", "issue_comments_fts": 16393, "rank": null}
{"rowid": 339510770, "body": "It looks like I should double quote my columns and ensure they are correctly escaped https://blog.christosoft.de/2012/10/sqlite-escaping-table-acolumn-names/ - hopefully using ? placeholders for column names will work. I should use ? for tables too.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 339514819, "body": "I\u2019m going to have a single command-line app that does everything.\r\n\r\nName to be decided - options include dataset, stateless, datasite (I quite like that - it reflects SQLite and the fact that you create a website)", "issue_comments_fts": 16393, "rank": null}
{"rowid": 339515822, "body": "datasite .\r\n\r\n- starts web app in current directory, serving all DB files\r\n\r\ndatasite . -p 8001\r\n\r\n- serves on custom port\r\n\r\ndatasite blah.db blah2.db\r\n\r\n- serves specified files\r\n\r\nYou can\u2019t specify more than one directory. You can specify as many files as you like. If you specify two files with different oaths but the same name then they must be accessed by hash.\r\n\r\ndatasite publish .\r\n\r\n- publishes current directory to the internet! Uses now by default, if it detects it on your path. Other publishers will be eventually added as plugins.\r\n\r\ndatasite publish http://path-to-db.db\r\n\r\n- publishes a DB available at a URL. Works by constructing the Dockerfile with wget calls in it.\r\n\r\ndatasite blah.db -m metadata.json\r\n\r\nIf you specify a directory it looks for metadata.json in that directory. Otherwise you can pass an explicit metadata file oath with -m or \u2014metadata", "issue_comments_fts": 16393, "rank": null}
{"rowid": 339516032, "body": "Another potential name: datapi ", "issue_comments_fts": 16393, "rank": null}
{"rowid": 339517846, "body": "I\u2019m going to use Click for this\r\n\r\nhttp://nvie.com/posts/writing-a-cli-in-python-in-under-60-seconds/\r\nhttps://kushaldas.in/posts/building-command-line-tools-in-python-with-click.html", "issue_comments_fts": 16393, "rank": null}
{"rowid": 339724700, "body": "Here\u2019s how to make the \u201cserve\u201d subcommand the default if it is called with no arguments:\r\n\r\n @click.group(invoke_without_command=True)\r\n def serve():\r\n # ...", "issue_comments_fts": 16393, "rank": null}
{"rowid": 339866724, "body": "![\"databases\"](\"https://user-images.githubusercontent.com/9599/32087550-4e10b8a4-ba91-11e7-992b-e628d77114d0.png\")
\r\n", "issue_comments_fts": 16393, "rank": null}
{"rowid": 339891755, "body": "Deploys to Now aren't working at the moment - they aren't showing the uploaded databases, because I've broken the path handling somehow. I need to do a bit more work here.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 340561577, "body": "http://the-hitchhikers-guide-to-packaging.readthedocs.io/en/latest/quickstart.html describes how to package this for PyPI", "issue_comments_fts": 16393, "rank": null}
{"rowid": 340787868, "body": "Here\u2019s how I can (I think) provide safe execution of arbitrary SQL while blocking PRAGMA calls: let people use names parameters in their SQL and apply strict filtering to the SQL query but not to the parameter values.\r\n\r\n cur.execute(\r\n \"select * from people where name_last=:who and age=:age\", {\r\n \"who\": who,\r\n \"age\": age\r\n })\r\n\r\nIn URL form:\r\n\r\n ?sql=select...&who=Terry&age=34\r\n\r\nNow we can apply strict, dumb validation rules to the SQL part while allowing anything in the named queries - so people can execute a search for PRAGMA without being able to execute a PRAGMA statement.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 341938424, "body": "Done: https://github.com/simonw/stateless-datasets/commit/edaa10587e60946e0c1935333f6b79553db33798", "issue_comments_fts": 16393, "rank": null}
{"rowid": 341945420, "body": "To simplify things a bit, I'm going to require that every database is explicitly listed in the command line. I won't support \"serve everything in this directory\" for the moment.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 342030075, "body": "... I tried that, I don't like it. I'm going to bring back \"directory serving\" by allowing you to pass a directory as an argument to `datasite` (including `datasite .`). I may even make `.` the default if you don't provide anything at all.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 342032943, "body": "Default look with Bootstrap 4 looks like this:\r\n![\"flights\"](\"https://user-images.githubusercontent.com/9599/32423242-1b4458ce-c25a-11e7-910f-2dc1de909b8f.png\")
\r\n", "issue_comments_fts": 16393, "rank": null}
{"rowid": 342484889, "body": "I\u2019m going to call this feature \u201ccount values\u201d", "issue_comments_fts": 16393, "rank": null}
{"rowid": 342521344, "body": "GDS Registries could be fun too: https://registers.cloudapps.digital/", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343164111, "body": "Implemented in 31b21f5c5e15fc3acab7fabb170c1da71dc3c98c", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343168796, "body": "Won't fix: ujson is not compatible with the custom JSON encoder I'm using here: https://github.com/simonw/immutabase/blob/b2dee11fcd989d9e2a7bf4de1e23dbc320c05013/immutabase/app.py#L401-L416", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343237982, "body": "More terms:\r\n\r\n* publish\r\n* share\r\n* docker\r\n* host\r\n* stateless\r\n\r\nI want to capture the idea of publishing an immutable database in a stateless container.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343238262, "body": "The name should ideally be available on PyPI and should make sense as both a command line application and a library.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343239062, "body": "This looks promising: https://github.com/esnme/ultrajson/issues/124#issuecomment-323882878", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343266326, "body": "http://sanic.readthedocs.io/en/latest/sanic/testing.html", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343281876, "body": "How about datasette?", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343551356, "body": "I'm going with datasette.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343557070, "body": "https://file.io/ looks like it could be good for this. It's been around since 2015, and lets you upload a temporary file which can be downloaded once.\r\n\r\n $ curl -s -F \"file=@database.db\" \"https://file.io/?expires=1d\"\r\n {\"success\":true,\"key\":\"ySrl1j\",\"link\":\"https://file.io/ySrl1j\",\"expiry\":\"1 day\"}\r\n\r\nDownloading from that URL serves up the data with a `Content-disposition` header containing the filename:\r\n\r\n simonw$ curl -vv https://file.io/ySrl1j | more\r\n % Total % Received % Xferd Average Speed Time Time Time Current\r\n Dload Upload Total Spent Left Speed\r\n 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 34.232.1.167...\r\n * Connected to file.io (34.232.1.167) port 443 (#0)\r\n * TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256\r\n * Server certificate: file.io\r\n * Server certificate: Amazon\r\n * Server certificate: Amazon Root CA 1\r\n * Server certificate: Starfield Services Root Certificate Authority - G2\r\n > GET /ySrl1j HTTP/1.1\r\n > Host: file.io\r\n > User-Agent: curl/7.43.0\r\n > Accept: */*\r\n > \r\n < HTTP/1.1 200 OK\r\n < Date: Fri, 10 Nov 2017 18:14:38 GMT\r\n < Content-Type: undefined\r\n < Transfer-Encoding: chunked\r\n < Connection: keep-alive\r\n < X-Powered-By: Express\r\n < X-RateLimit-Limit: 5\r\n < X-RateLimit-Remaining: 4\r\n < Access-Control-Allow-Origin: *\r\n < Access-Control-Allow-Headers: Cache-Control,X-reqed-With,x-requested-with\r\n < Content-disposition: attachment; filename=database.db\r\n ...\r\n", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343581130, "body": "I'm going to handle this a different way. I'm going to support a local history of your own queries stored in localStorage, but if you want to share a query you have to do it with a URL.\r\n\r\nIf people really want canned query support, they can do that using custom templates - see #12 - or by adding views to their database before they publish it.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343581332, "body": "I'm not going to use Sanic's mechanism for this. I'll use arguments passed to my cli instead.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343643332, "body": "Here's what a table looks like now at a smaller screen size:\r\n\r\n![\"parlgov-development__info_data_source\"](\"https://user-images.githubusercontent.com/9599/32686818-7ffa691e-c662-11e7-964d-201fce0694e4.png\")
\r\n", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343644891, "body": "I can detect something is a view like this:\r\n\r\n SELECT name from sqlite_master WHERE type ='view';\r\n", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343644976, "body": "Simplest version of this:\r\n\r\n1. Create a temporary directory\r\n2. Write a Dockerfile into it that pulls an image and pip installs datasette\r\n3. Add symlinks to the DBs they listed (so we don't have to copy them)\r\n4. Shell out to \"now\"\r\n5. Done!\r\n", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343645249, "body": "Doing this works:\r\n\r\n import os\r\n os.link('/tmp/databases/northwind.db', '/tmp/tmp-blah/northwind.db')\r\n\r\nThat creates a link in tmp-blah - and then when I delete that entire directory like so:\r\n\r\n import shutil\r\n shutil.rmtree('/tmp/tmp-blah')\r\n\r\nThe original database is not deleted, just the link.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343645327, "body": "I can create the temporary directory like so:\r\n\r\n import tempfile\r\n t = tempfile.TemporaryDirectory()\r\n t\r\n \r\n t.name\r\n '/var/folders/w9/0xm39tk94ng9h52g06z4b54c0000gp/T/tmpkym70wlp'\r\n\r\nAnd then to delete it all:\r\n\r\n t.cleanup()\r\n", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343646740, "body": "I'm happy with this now that I've implemented the publish command in #26 ", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343647102, "body": "http://2016.padjo.org/tutorials/data-primer-census-acs1-demographics/ has a sqlite database: http://2016.padjo.org/files/data/starterpack/census-acs-1year/acs-1-year-2015.sqlite\r\n\r\nI tested this by deploying it here: https://datasette-fewuggrvwr.now.sh/", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343647300, "body": "Still needed:\r\n\r\n- [ ] A link to the homepage from some kind of navigation bar in the header\r\n- [ ] link to github.com/simonw/datasette in the footer\r\n- [ ] Slightly better titles (maybe ditch the visited link colours for titles only? should keep those for primary key links)\r\n- [ ] Links to the .json and .jsono versions of every view", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343675165, "body": "The plugin system can also allow alternative providers for the `publish` command - e.g. maybe hook up hyper.sh as an option for publishing containers.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343676574, "body": "See also #14", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343683566, "body": "I\u2019m going to solve this by making it an optional argument you can pass to the serve command. Then the Dockerfile can still build and use it but it won\u2019t interfere with tests or dev.\r\n\r\nIf argument is not passed, we will calculate hashes on startup and calculate table row counts on demand.\r\n", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343690060, "body": " \"parlgov-development.db\": {\r\n \"url\": \"http://www.parlgov.org/\"\r\n },\r\n \"nhsadmin.sqlite\": {\r\n \"url\": \"https://github.com/psychemedia/openHealthDataDoodles\"\r\n }", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343691342, "body": "Closing this, opening a fresh ticket for the navigation stuff.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343697291, "body": "I'm going to bundle sql and sql_params together into a query nested object like this:\r\n\r\n {\r\n \"query\": {\r\n \"sql\": \"select ...\",\r\n \"params\": {\r\n \"p0\": \"blah\"\r\n }\r\n }\r\n }", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343698214, "body": "I'm closing #50 - more tests will be added in the future, but the framework is neatly in place for them now.\r\n", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343699115, "body": "This needs to incorporate a sensible way of presenting custom SQL query results too. And let's get a textarea in there for executing SQL while we're at it.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343705966, "body": "https://github.com/fivethirtyeight/data has a ton of CSVs", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343707624, "body": "Split the SQL thing out into #65 ", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343707676, "body": "Here's the new design:\r\n\r\n![\"parlgov-development\"](\"https://user-images.githubusercontent.com/9599/32695161-82821226-c708-11e7-835c-b3d91850b2e0.png\")
\r\n\r\nAlso lists views at the bottom (refs #54):\r\n\r\n![\"parlgov-development\"](\"https://user-images.githubusercontent.com/9599/32695164-99efa7de-c708-11e7-8272-bc5f5b870b84.png\")
\r\n", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343708447, "body": "I ditched the metadata file concept.", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343709217, "body": "![\"nhsadmin\"](\"https://user-images.githubusercontent.com/9599/32695392-3ea12612-c70f-11e7-873b-9e6ad2c869e7.png\")
\r\n", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343715915, "body": " con = sqlite3.connect('existing_db.db')\r\n with open('dump.sql', 'w') as f:\r\n for line in con.iterdump():\r\n f.write('%s\\n' % line)\r\n", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343752404, "body": "Re-opening this - I've decided to bring back this concept, see #68 ", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343752579, "body": "By default I'll allow LIMIT and OFFSET up to a maximum of X (where X is let's say 50,000 to start with, but can be custom configured to a larger number or set to None for no limit).", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343752683, "body": "Maybe SQL views should have their own Sanic view class (`ViewView` is kinda funny), subclassed from `TableView`?", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343753999, "body": "For initial launch, I could just support this as some optional command line arguments you pass to the publish command:\r\n\r\n datasette publish data.db --title=\"Title\" --source=\"url\"", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343754058, "body": "I\u2019m going to store this stuff in a file called metadata.json and move the existing automatically generated metadata to a file called build.json", "issue_comments_fts": 16393, "rank": null}
{"rowid": 343769692, "body": "I have created a Docker Hub public repository for this: https://hub.docker.com/r/simonwillison/datasette/", "issue_comments_fts": 16393, "rank": null}