HN user
Rust is a "new" language and many packages from other languages get reimplemented in it. This is similar to Julia. Unfortunately, I had the experience that many Julia packages are not of high quality, not maintained, or do not run any more on the newest version.
How is this in Rust?
What car do you have that comes with a LiDAR? Is it already used for a useful driving assistant?
I read or saw a similar paper. I guess that they used (multiple) 3D cameras instead of a LiDAR to get the point cloud. But otherwise it was similar. They used a octree or similar data structure for speed up. What did you use?
Very interesting. What LiDAR are you using? Does it happen if the sun shines directly into the LiDAR from the front?
LiDAR is finally cheap.
I repeat my comment on LiDAR that I gave a few days ago. The gist is that LiDAR is cheap and you will be able to buy a LiDAR with sufficient resolution for in the next 1-2 years because it will be integrated in normal passenger cars for L2/L3 assistants. These cars are coming out now or in the next year.
LiDAR is finally getting cheap. OEMs (like VW) are very price sensitive. It is estimated the sensors from Valeo cost about 500 dollars. The fact that you see more and more normal passenger cars with higher resolution LiDARs means that LiDARs are getting cheaper.
The Audi A8 used Valeo's (with Ibeo) first generation low resolution LiDAR Scala 1 from the automotive supplier Valeo. Mercedes new models will be using Valeo's second (or third) generation LiDAR. All these are used for L2/L3 assistants. Valeo is a traditional large automotive supplier.
Luminar, a public company from the US, cooperates with Volvo. Some models will come with a LiDAR in the base configuration. These are "new LiDARs" with high resolution.
Innoviz, a 'startup' from Isreal, will supplies LiDARs to VW. Its angular resolution is (in its focus area) about 0.1 (or 0.2) degrees, which is sufficient for higher levels of autonomy and surpasses/equals the resolution of the expensive Velodyne sensors of the past. They will probably be in the same price range. Due to the limited FOV due to the technology, you will need multiply LiDARs.
Many new models from Chinese car brands will also ve equipped with a LiDAR. Most of them with Chinese LiDAR manufacturers like RoboSense or Hesai. Some are equipped by European manufactures like Ibeo/ZF. For example, there is the automotive sensor AT128 by Hesai. It targets normals vehicles (see price range above) and claims a similar performance (except for FOV, so you need multiple) like the Velodyne Ultra Puck (~$50000).
So costs of LiDARs are a not the very expensive obstacle they were in the past. The only problem could be that the new LiDAR manufactures cannot scale up series production. For example, Ibeo just filed for insolvency because they could not close another round after aggressively increasing spending in the past years.
LiDAR is finally getting cheap. OEMs (like VW) are very price sensitive. It is estimated the sensors from Valeo cost about 500 dollars. The fact that you see more and more normal passenger cars with higher resolution LiDARs means that LiDARs are getting cheaper.
The Audi A8 used Valeo's (with Ibeo) first generation low resolution LiDAR Scala 1 from the automotive supplier Valeo. Mercedes new models will be using Valeo's second (or third) generation LiDAR. All these are used for L2/L3 assistants. Valeo is a traditional large automotive supplier.
Luminar, a public company from the US, cooperates with Volvo. Some models will come with a LiDAR in the base configuration. These are "new LiDARs" with high resolution.
Innoviz, a 'startup' from Isreal, will supplies LiDARs to VW. Its angular resolution is (in its focus area) about 0.1 (or 0.2) degrees, which is sufficient for higher levels of autonomy and surpasses/equals the resolution of the expensive Velodyne sensors of the past. They will probably be in the same price range. Due to the limited FOV due to the technology, you will need multiply LiDARs.
Many new models from Chinese car brands will also ve equipped with a LiDAR. Most of them with Chinese LiDAR manufacturers like RoboSense or Hesai. Some are equipped by European manufactures like Ibeo/ZF. For example, there is the automotive sensor AT128 by Hesai. It targets normals vehicles (see price range above) and claims a similar performance (except for FOV, so you need multiple) like the Velodyne Ultra Puck (~$50000).
So costs of LiDARs are a not the very expensive obstacle they were in the past. The only problem could be that the new LiDAR manufactures cannot scale up series production. For example, Ibeo just filed for insolvency because they could not close another round after aggressively increasing spending in the past years.
DVC has had the following problems, when I tested it (half a year ago):
I gets super slow (waiting minutes) when there are a few thousand files tracked. Thousands files have to be tracked, if you have e.g. a 10GB file per day and region and artifacts generated from it.
You are encouraged (it only can track artifacts) if you model your pipeline in DVC (think like make). However, it cannot run tasks it parallel. So it takes a lot of time to run a pipeline while you are on a beefy machine and only one core is used. Obviously, you cannot run other tools (e.g. snakemake) to distribute/parallelize on multiple machines. Running one (part of a) stage has also some overhead, because it does commit/checks after/before running the executable of the task.
Sometimes you get merge conflicts, if you run a (partial parmaretized) stage on one machine and the other part on the other machine manually. These are cumbersome to fix.
Currently, I think they are more focused on ML features like experiment tracking (I prefer other mature tools here) instead of performance and data safety.
There is an alternative implementation from a single developer (I cannot find it right now) that fixes some problems. However, I do not use this because it propably will not have the same development progress and testing as DVC.
This sounds negative but I think it is currently the one of the best tools in this space.
I used Julia in a robotics project doing statistics/estimation/easy optimization but not deep learning. I also do ML/DL:
Julia vs. Python
- PyTorch is standard and it is hard to convince other people to switch
- long compile time on startup during deployment (not so good for a robot) but also for plotting; other people really hated this
Julia vs. C++
- Julia has a JIT and is MUCH faster than Python if you cannot write it as a sequence of numpy operations, e.g. if you have loops and if-blocks in the main loop; C++ obviously also shines here
- however, similar to Python you can only detect problems of the code when running it - the linters etc. are not good enough; hence, I also fear changing only a few lines; programming in C++ is much easier and you have much more confidence that the code is correct if it compiles
After learning JAX in Python, which compiles numeric code JIT, I have almost no reason using Julia anymore. Of course, DifferentialEquations.jl and many optimization libraries are top notch.
I guess the text was extracted using two different methods. One results in 0.8TB and the other in 0.5TB text.
1) I assume 1TB (not TiB) of uncompressed (?) text
2) I assume one character is one byte
3) I assume 5 (actually it seems to be 4.7 in English) characters per word
So 1TB/1B/5 = 1.0E12/5 ~= 2.0E11 = 0.2T = 200B words.
Your article mentioned that Chinchilla is trained on 1.4T tokens. So there is quite some difference.
The article also mentions different mysterious book data sets with 27B tokens, 560B tokens, or 390B tokens.
The latter datasets were made by Google. So you are still behind Google massive book dataset even if you use probably the largest book dataset "available" to people or instituions outside of Google.
EDIT: I thought I made a mistake, but T stands for trillion or tera which are both 1E12.
This sounds like a very interesting area! I guess you are the "Statistical Process Monitoring"/"Control charts"/"Shewhart charts" [0] for images. Very cool!
Is this correct or is your solution totally different? In what aspect is it most similar and most different from "Control charts"?
Are there any keywords for interested hackernews readers to research this further and play with this concept? Is it correct that you do "just" outlier detection on the embeddings of the images? I guess it works something like this:
1) Image --CNN--> Embedding: maybe enforce (properties) of distribution on the embedding (something like VAE)
2) Approximate this distribution and call a (sequence of) images an outlier if its likelihood is small. Alternatively, compare the empirical distribution of a few collected images to a distribution of "good images", e.g. via embedding into RKHS.
What type of anomalies can be detected? Does in evaluate each image separately (i.e. it cannot differentiate between objects going from left to right) or does it "understand" short sequences of images? The latter sound even more interesting. Could you provide some keywords for it.
On the production line, there are already cameras and computer vision products, e.g. Halcon. These can be used to "drag/drop" a computer vision pipeline together. Could your software be integrated into it such that the output can be further processed in Halcon etc. ?
Thank's a lot. This document really makes it clear how it works. Awesome idea!
I would be interesting if you could describe how it exactly works. Can you give us a little more color on that?
1. You mentioned that this only a marketplace between lenders and startups. Does a startup create a kind of brochure describing their idea trying to lure in lenders (like kickstarter or other crowdfunding/lending sites)? Or does the startup just fills out a form and then you decide about the loan? In this case, how do the lenders decide if the want to give this startup money?
2. Who decides if a startup gets money?
3. What exactly is your part in this process? Do you quantify the risk, so lenders can only give money to startups in a certain risk category?
4. What happens if the startup fails? Who has lost money? You or the third-party lenders?
That sounds really interesting! Can you give a few more details how you use Graphviz? How does it give such a great advantage "to do in a couple of hours what typically would take client staff months."
Did I understand it correctly that you use it to discover patterns? Are these patterns discovered by just using the layout engines? Arent' other tools, e.g. networkx in Python or cytoscape in javascript easier to use interactively in a REPL? What is the typical workflow (maybe plot, find interesting pattern, change query/data in a loop)?
I'm really interested in how Graphviz can be so great. I am currently working with the other mentioned tools for visualization purposes.
Pretty exciting stuff! This looks really interesting. I have a similar with setup hosts in the cloud, in the office, at home, and on UAVs with cellular internet. It was important that the hosts 'see' each other and it works on IP level in any direction.
When I set it up, I have chosen ZeroTier instead of WireGuard. It does the following: (a) Hosts discovery and initiating handshake between clients with the help of a server from ZeroTier, (b) NAT hole punching, (c) pushing centrally managed routes to hosts, (d) network ACL rules. I primarily have chosen it because it is easy to setup by anyone (e) and I do not have to manage a server (f).
- Can you tell a little bit how Graviti compares to it. I guess WireGuard itself does not have the features (a) to (f). I guess the Netmake server replaces the ZeroTier servers and provides some of these features.
- Are you inclined to install Netmaker client on any host or use one node in a LAN as a router?
- Is this more geared to servers/professional managed hosts or also for laptops?
For my usecase with ZeroTier I found the following currenlty missing features useful:
- Easy setup of a node as a router (or virtual switch) to connect a local network to the virtual one without installing it on all devices (hardware like GPS receiver do not allow to install new software). Of course, you can do it with the normal Linux tools.
- Installing it only inside a Docker container and not on the host. But I guess that will not be possible because it has to live in the kernel.
That is a very interesting idea. As far as I understand, you generate the information purely read only.
Column statistics: Sample data randomly and calculate estimator from that.
Data Lineage/usage: From the queries (and 'similar' stuff like materialized views. It seems like that currenltly have only lineage on table level, i.e. which table depends on which tables. Do you plan to implement data lineage on column level, i.e. which column depends on which column? This sounds challenging and very interesting.
I tried to watch the demo video on your website, but it was too small and maximizing the video is disabled (Firefox). So I had to click through to youtube. I'm not sure if this is intended.